A fault-tolerant multimodal medical feature reconstruction system and method
By using a dynamic modal state detector and a multi-directional conditional feature reconstruction engine, combined with dynamic mask matrix weight adjustment, the fault tolerance problem of multimodal AI models when modalities are missing is solved, improving the accuracy and robustness of survival prediction, and making it suitable for prognostic assessment of complex malignant tumors.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHERN MEDICAL UNIVERSITY
- Filing Date
- 2026-04-20
- Publication Date
- 2026-06-26
AI Technical Summary
Existing multimodal AI models cannot effectively trigger forward inference when faced with modality loss, lacking fault tolerance mechanisms, which leads to serious misleading of survival prediction risk stratification. Especially in the prognostic assessment of complex malignant tumors, existing completion algorithms are decoupled from medical logic, failing to reflect the biological relationship between macroscopic images and microscopic tissue structures, and the introduction of synthetic noise causes unacceptable prediction bias.
A dynamic modal state detector is used to perceive the existence of modalities in real time. Multi-source medical data features are extracted through parallel feature encoding branches. When a modality is missing, a multi-dimensional conditional feature reconstruction engine is used for cross-scale inference compensation. The feature fusion weights are adjusted by combining a dynamic mask matrix, and a confidence penalty factor is introduced to reduce the weights of reconstructed features. A dual-path consistency-guided training strategy is used to optimize the model.
It achieves adaptive fault tolerance for multimodal missing scenarios, significantly improves the robustness and accuracy of survival prediction, and provides reliable support for intelligent auxiliary decision-making in complex clinical scenarios.
Smart Images
Figure CN122067756B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical technology, and in particular to a fault-tolerant multimodal medical feature reconstruction system and a fault-tolerant multimodal medical feature reconstruction method. Background Technology
[0002] In the prognostic assessment of complex malignancies such as colorectal cancer (LARC), clinicians need to integrate macroscopic morphological phenotypes from mp-MRI, microscopic heterogeneity from histopathology (WSI), and clinical semantic information. Most existing multimodal AI models are based on the assumption of "multi-source input alignment," but in real-world clinical scenarios, due to insufficient biopsy sample size, image motion artifacts, or fragmented medical records, the modality loss rate can be as high as 20%-30%. When modality loss occurs, existing technical solutions face the following technical problems: On the one hand, traditional models such as simple splicing or static distillation cannot trigger forward inference when the input dimension is missing, lacking an effective fault tolerance mechanism; on the other hand, existing completion algorithms are decoupled from medical logic, and mean filling or feature alignment based on contrastive learning only approaches each other in terms of vector distance, failing to reflect the biological relationship between macroscopic images and microscopic tissue structures. Furthermore, the synthetic noise introduced after completion by such methods is prone to produce a large weight bias when entering the attention mechanism, ultimately causing serious misleading of survival prediction risk stratification based on overall survival or disease-free survival, resulting in unacceptable prediction bias when the model faces key clinical decisions (such as the determination of sphincter-preserving surgery and the evaluation of neoadjuvant efficacy). Summary of the Invention
[0003] In view of the above problems, the present invention is proposed to provide a fault-tolerant multimodal medical feature reconstruction system and a corresponding fault-tolerant multimodal medical feature reconstruction method that overcomes or at least partially solves the above problems.
[0004] This invention discloses a fault-tolerant multimodal medical feature reconstruction system, the system comprising:
[0005] A dynamic modal state detector is used to receive raw multi-source medical data and output a state vector representing the presence or absence of each modality.
[0006] Parallel feature encoding branches are used to extract features from the macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical texts in the original multi-source medical data, respectively, and map them to a unified-dimensional latent feature space.
[0007] A multi-dimensional conditional feature reconstruction engine is used to reconstruct and compensate the features of the missing modality based on the features of the remaining modality when the state vector output by the dynamic modality state detector indicates that there is a lack of modality, based on cross-scale semantic generation logic, so as to realize cross-scale inference compensation between macroscopic image features and microscopic pathological features.
[0008] The fault-tolerant fusion decision layer based on dynamic mask is used to construct a dynamic mask matrix according to the state vector, and to fuse the features extracted by the parallel feature encoding branch and the features reconstructed by the multi-directional conditional feature reconstruction engine based on the dynamic mask matrix. The dynamic mask matrix is used to adjust the decision weights of features from different sources during fusion.
[0009] The global survival risk assessment output module is used to predict and output a survival risk score based on the fusion results of the fault-tolerant fusion decision layer based on dynamic mask.
[0010] Optionally, the parallel feature encoding branch includes:
[0011] The macroscopic image feature encoding module is used to extract morphological features of T2WI, blood supply enhancement features of T1WIC and diffusion restriction features of ADC from macroscopic morphological mp-MRI images using a multi-scale feature extraction method, and generate image feature vectors characterizing the macroscopic invasiveness of tumors.
[0012] The micropathological feature encoding module is used to extract micropathological features focused on areas of dense tumor cells and areas of stromal infiltration from micro-heterogeneous pathological WSI images using a gated attention mechanism based on the UNI basic model.
[0013] The clinical text encoding module is used to extract semantic features from unstructured text using BERT, and simultaneously extract the embedding features of structured indicators using linear layers. After fusing the semantic features of unstructured text and the embedding features of structured indicators, global clinical features are generated.
[0014] Optionally, the multi-directional conditional feature reconstruction engine includes:
[0015] The conditional variational inference branch is used to learn the posterior probability distribution of pathological features by using at least one residual modality feature as a conditional variable when pathological modality is detected as missing, and to generate reconstructed pseudo-pathological features by using reparameterized sampling.
[0016] The semantic prototype retrieval branch is used to retrieve the most similar visual prototype from a pre-built semantic prototype library when a missing text modality is detected, using the existing visual features as the query vector, and then retrieving the corresponding text prototype vector to fill in the missing text. The visual features include image features and pathological features.
[0017] Optionally, the semantic prototype retrieval branch is further used for:
[0018] By using a linear correction layer, the retrieved text prototype vector is fine-tuned in conjunction with the patient's basic statistical characteristics to accommodate individual differences.
[0019] Optionally, the fault-tolerant fusion decision layer based on dynamic masking is further used for:
[0020] Depending on whether the feature source is original acquisition or reconstruction compensation, a corresponding confidence penalty factor is set in the dynamic mask matrix, and the confidence penalty factor is used to correct the attention score matrix in the multi-head self-attention calculation to reduce the weight of the reconstructed features in the fusion; the magnitude of the confidence penalty factor is dynamically adjusted according to the state vector.
[0021] Optionally, the system further includes:
[0022] The dual-path consistency-guided training module is used to randomly mask some modalities in the input during training to force the activation of the reconstruction engine, and to optimize the model by approximating the risk distribution of the full modal input through constrained completion path output risk distribution.
[0023] This invention also discloses a fault-tolerant multimodal medical feature reconstruction method, the method comprising:
[0024] It receives raw multi-source medical data and outputs state vectors representing the presence or absence of each modality;
[0025] Feature extraction was performed on the macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical texts in the original multi-source medical data, and mapped to a unified potential feature space.
[0026] When the state vector indicates the presence of a missing modality, the features of the missing modality are reconstructed and compensated based on the features of the remaining modality, based on the cross-scale semantic generation logic, so as to achieve cross-scale inference compensation between macroscopic imaging features and microscopic pathological features.
[0027] A dynamic mask matrix is constructed based on the state vector, and the extracted features and reconstructed features are fused based on the dynamic mask matrix. The dynamic mask matrix is used to adjust the decision weights of features from different sources during fusion.
[0028] Based on the fusion results, a prediction is made and a survival risk score is output.
[0029] This invention has the following advantages:
[0030] The fault-tolerant multimodal medical feature reconstruction system of this invention uses a dynamic modality state detector to perceive the existence state of each modality in real time. When a missing modality is detected, a multi-directional conditional feature reconstruction engine is activated: for missing pathological modalities, pseudo-pathological features are generated through a conditional variational autoencoder using image features as conditions, achieving cross-scale reasoning from macro to micro; for missing textual modalities, matching textual prototypes are retrieved from a semantic prototype library using visual features as queries for filling. Based on this, a fault-tolerant fusion decision layer based on dynamic masks constructs a dynamic mask matrix according to the feature source, introducing a confidence penalty factor in multi-head attention calculation to reduce the decision weight of reconstructed features, ensuring that the fusion process is biased towards the actual collected data. The system also performs joint optimization through random modality masking and prediction logic consistency constraints, making the completed features task-oriented. This invention achieves adaptive fault tolerance for multimodal missing scenarios, significantly improving the robustness and accuracy of survival prediction, and providing reliable support for intelligent auxiliary decision-making in complex clinical scenarios. Attached Figure Description
[0031] Figure 1 This is a structural block diagram of the fault-tolerant multimodal medical feature reconstruction system provided in this embodiment of the invention;
[0032] Figure 2 This is a diagram illustrating the mechanism of multi-directional feature reconstruction and semantic alignment provided in this embodiment of the invention. Detailed Implementation
[0033] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The patient data used in the present invention was obtained with the patient's informed consent and in accordance with ethical principles.
[0034] This invention proposes a dual-track compensation mechanism based on Conditional Variational Inference and Semantic Prototype Retrieval. When the system detects a missing modality through the state vector, it automatically activates the generation branch and uses the semantic penetration capability of the remaining modalities to reconstruct the distribution of the missing modalities. At the same time, a dynamic mask matrix is introduced to automatically adjust the decision weight of features in global fusion based on their authenticity.
[0035] Reference Figure 1 The diagram shows a structural block diagram of a fault-tolerant multimodal medical feature reconstruction system provided in an embodiment of the present invention. The specific operation process of the system is as follows:
[0036] 1. Standardized joint characterization of heterogeneous multimodal features
[0037] The system proposed in this invention first maps macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical text to a unified-dimensional latent feature space using a parallel heterogeneous encoder. The purpose of this step is to eliminate scale differences and modal barriers between data collected by different physical devices.
[0038] (1) Macroscopic imaging feature coding (mp-MRI branch):
[0039] For multi-sequence MRI data, the system employs an improved ResNet-50 as the backbone network. Unlike traditional single-sequence processing, this invention extracts morphological features from T2WI, blood supply enhancement features from T1WIC, and diffusion-restricted features from ADC. Through multi-scale feature extraction, skip connections are made in the Res3 and Res4 layers of the encoder to capture the spiculation of tumor margins and capsule integrity. The image features are finally dimensionality-reduced using global average pooling (GAP) to generate a vector characterizing the macroscopic invasiveness of the tumor. .
[0040] (2) Microscopic pathological feature coding (WSI branch):
[0041] Whole-slice images (WSI) of pathology are huge in size and highly heterogeneous. This embodiment employs a gated attention mechanism based on the UNI base model. First, the WSI is segmented into 256×256 image patches, and then the large model is used to extract patch features. The contribution weight of each block to survival prognosis is then calculated using a learnable attention module:
[0042]
[0043] This mechanism automatically focuses on areas of dense tumor cell infiltration and stromal invasion, ignoring meaningless fat and background areas, and aggregates them into microscopic features. .
[0044] (3) Clinical text encoding branch: The clinical text encoding branch uses BERT to extract semantic features of unstructured text, and simultaneously uses linear layers to extract the embedding features of structured indicators. Then, the two features are concatenated into vectors and deep fusion and dimensionality reduction mapping are achieved through fully connected layers to generate global clinical features. .
[0045]
[0046] in, This indicates that the concatenated vectors have undergone a dimensionality reduction mapping.
[0047] (4) Semantic Alignment across modal latent space:
[0048] To ensure reconfigurability when modalities are missing, the system introduces a symmetric topological alignment loss during the training phase. This requires the same patient... Minimize the distance in Euclidean space:
[0049]
[0050] The physical significance of this step is that it enables the image feature encoder to perceive the corresponding pathological patterns while extracting morphological information, thereby providing sufficient semantic prior when pathological modalities are missing.
[0051] 2. Missing Feature Reconstruction Engine Based on Conditional Variational Inference (CVAE)
[0052] This engine is the core of achieving "system fault tolerance". When the system identifies the modal state vector... When the number of pathological sites is 0, the reconstruction engine is automatically triggered. (See reference...) Figure 2 .
[0053] (1) Generative Reconstruction Mechanism:
[0054] This invention employs a conditional variational autoencoder (CVAE) structure. It utilizes at least one residual modal feature as a condition variable, such as an image... By inferring the posterior probability distribution of pathological features through a network. .
[0055] (2) Reparameterization techniques and sampling:
[0056] To ensure the differentiability of the network, random noise is introduced. Calculate latent variables through reparameterization Then, the decoder receives... With conditional features Reconstructing pseudo-pathological features :
[0057]
[0058] (3) Loss function constraint (ELBO):
[0059] The reconstruction process is constrained by the variational lower bound loss to ensure that the generated It not only conforms to the true distribution of pathological features but also logically aligns with the current imaging manifestations of patients. This generative ability of "inferring the micro from the macroscopic" constitutes the essential difference between this invention and pure retrieval methods.
[0060] 3. Detailed Explanation of the Operation of the Enhanced Semantic Prototype Bank
[0061] For scenarios where clinical texts (such as medical records and laboratory indicators) are missing, this invention designs a semantic prototype retrieval mechanism based on knowledge graph logic. (Refer to...) Figure 2 .
[0062] (1) Construction of the global prototype library:
[0063] The system utilizes pre-collected multicenter rectal cancer data from 1996 cases to decouple multimodal features along the prognostic risk dimension. It then clusters the samples using a graph attention network (GAT) to extract... A representative "prototype" vector Each prototype represents a specific pathological evolution pattern (e.g., CEA high expression accompanied by vascular invasion).
[0064] (2) Adaptive search completion:
[0065] When text data is detected When missing, the system utilizes existing visual features. As a query vector, it retrieves the most similar visual prototype from the library:
[0066]
[0067] Then retrieve the corresponding text prototype vector. Fill in the blanks.
[0068] (3) Semantic smoothing correction:
[0069] The retrieved features are processed through a linear correction layer, which is then fine-tuned based on the patient's baseline statistical characteristics (such as age and gender) to ensure that the completed features can adapt to subtle individual differences.
[0070] 4. Fault-tolerant fusion logic based on dynamic mask (Fault-tolerant Mask)
[0071] In the feature fusion stage, to prevent noise carried by "completing features" from interfering with the final prediction, this invention proposes a confidence-sensitive fault-tolerant fusion architecture.
[0072] (1) Construction of state-aware mask:
[0073] The system defines a dynamic mask matrix. When a modality is missing and completion is triggered, the system will identify the source of the feature.
[0074] (2) Mask self-attention calculation:
[0075] In multi-head attention calculation, the relevant score matrix is corrected:
[0076]
[0077] in, It is a confidence penalty factor. For generated pseudo-features, the system will automatically reduce their energy in weight allocation, making the decision more biased towards the "truly collected modalities".
[0078] (3) Self-healing prediction:
[0079] This architecture ensures that even in extreme missing combinations (such as with only a single image), the system will not cause gradient explosion or collapse due to outliers in the input vector, thus outputting a robust survival risk score.
[0080] 5. Dual-path consistency-guided training strategy
[0081] To enable the model to handle missing data, this invention employs a teacher-student distillation training framework.
[0082] (1) Random modal masking (Masked Pre-training):
[0083] During training, the system randomly masks any one or two modalities in the input with a certain probability (e.g., 30%), forcing the model to activate the reconstruction engine and retrieval logic.
[0084] (2) Predictive logic consistency constraint:
[0085] Introducing KL divergence loss to constrain the risk distribution of the completed path output The risk distribution must approximate the full modal input. :
[0086]
[0087] (3) Multi-task joint optimization:
[0088] The system simultaneously optimizes survival prediction accuracy (Cox loss) and feature completion accuracy (reconstruction loss). This training strategy ensures that the completion capability learned by the system is "task-oriented," meaning that the completed features must be discriminative for survival prognosis (OS / DFS), rather than just pixel-level filling.
[0089] The following are two verification examples of this invention:
[0090] Example 1: Robustness performance under extreme missing scenarios
[0091] In a test set covering 752 rectal cancer patients, the clinically common “image + text” input was simulated (pathology was missing due to biopsy failure).
[0092] (1) Control group A (zero filling): OS C-index dropped to 0.62.
[0093] (2) Control group B (comparison alignment method): OS C-index is 0.69.
[0094] (3) The system of this invention: The correlation between the virtual pathological features reconstructed by CVAE and the original features is 0.88, and the predicted OS C-index is 0.782. This proves that generative reconstruction has a stronger information recovery capability than simple feature merging.
[0095] Example 2: Semantic retrieval utility in cross-institutional validation
[0096] At external validation set center B, due to different medical record standards, 50% of patients had severely missing clinical indicators.
[0097] Implementation process: The system identifies missing text and automatically retrieves the global semantic prototype library through image phenotype.
[0098] Technical effect: The significance of the HR (hazard ratio) of the completed model on the external validation set improved from p=0.03 to p<0.001. This demonstrates the significant role of the prototype retrieval mechanism in recovering lost prognostic semantics.
[0099] The fault-tolerant multimodal medical feature reconstruction method provided in this embodiment of the invention may specifically include the following steps:
[0100] It receives raw multi-source medical data and outputs state vectors representing the presence or absence of each modality;
[0101] Feature extraction was performed on the macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical texts in the original multi-source medical data, and mapped to a unified potential feature space.
[0102] When the state vector indicates the presence of a missing modality, the features of the missing modality are reconstructed and compensated based on the features of the remaining modality, based on the cross-scale semantic generation logic, so as to achieve cross-scale inference compensation between macroscopic imaging features and microscopic pathological features.
[0103] A dynamic mask matrix is constructed based on the state vector, and the extracted features and reconstructed features are fused based on the dynamic mask matrix. The dynamic mask matrix is used to adjust the decision weights of features from different sources during fusion.
[0104] Based on the fusion results, a prediction is made and a survival risk score is output.
[0105] In an optional embodiment of the present invention, feature extraction is performed on the macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical texts in the original multi-source medical data, including:
[0106] A multi-scale feature extraction method was used to extract morphological features of T2WI, blood supply enhancement features of T1WIC, and diffusion restriction features of ADC from macroscopic morphological mp-MRI images, generating image feature vectors characterizing the macroscopic invasiveness of tumors.
[0107] We employed a gated attention mechanism based on the UNI model to extract micropathological features focused on areas of dense tumor cells and areas of stromal invasion from microscopic heterogeneous pathological WSI images.
[0108] BERT is used to extract semantic features from unstructured text, and a linear layer is used to extract embedding features of structured indicators. The semantic features of unstructured text and the embedding features of structured indicators are then fused to generate global clinical features.
[0109] In an optional embodiment of the present invention, when the state vector indicates the presence of a missing modality, based on cross-scale semantic generation logic, the features of the missing modality are reconstructed and compensated based on the features of the remaining modality to achieve cross-scale inference compensation between macroscopic imaging features and microscopic pathological features, including:
[0110] When pathological modality loss is detected, at least one residual modality feature is used as a conditional variable. The posterior probability distribution of the pathological feature is learned through a conditional variational autoencoder, and reparameterized sampling is used to generate reconstructed pseudo-pathological features.
[0111] When a missing text modality is detected, the existing visual features are used as query vectors to retrieve the most similar visual prototype from a pre-built semantic prototype library, and the corresponding text prototype vector is retrieved to fill the missing text; the visual features include image features and pathological features.
[0112] In an optional embodiment of the present invention, when a missing text modality is detected, the most similar visual prototype is retrieved from a pre-built semantic prototype library using the currently existing visual features as the query vector, and the corresponding text prototype vector is retrieved to fill the missing modality, including:
[0113] By using a linear correction layer, the retrieved text prototype vector is fine-tuned in conjunction with the patient's basic statistical characteristics to accommodate individual differences.
[0114] In an optional embodiment of the present invention, the extracted features and reconstructed features are fused based on the dynamic mask matrix, including:
[0115] Depending on whether the feature source is original acquisition or reconstruction compensation, a corresponding confidence penalty factor is set in the dynamic mask matrix, and the confidence penalty factor is used to correct the attention score matrix in the multi-head self-attention calculation to reduce the weight of the reconstructed features in the fusion; the magnitude of the confidence penalty factor is dynamically adjusted according to the state vector.
[0116] In an optional embodiment of the present invention, the method further includes:
[0117] During training, some modalities in the input are randomly masked to force the activation of the reconstruction compensation step, and the risk distribution of the completed path output is approximated to the risk distribution of the full modal input to optimize the model.
[0118] For the method embodiments, since they are basically similar to the system embodiments, the description is relatively simple, and relevant details can be found in the description of the system embodiments. It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions; however, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0119] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0120] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0121] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.
Claims
1. A fault-tolerant multimodal medical feature reconstruction system, characterized in that, The system includes: A dynamic modal state detector is used to receive raw multi-source medical data and output a state vector representing the presence or absence of each modality. Parallel feature encoding branches are used to extract features from the macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical texts in the original multi-source medical data, respectively, and map them to a unified-dimensional latent feature space. A multi-directional conditional feature reconstruction engine is used to reconstruct and compensate for the features of the missing modality based on cross-scale semantic generation logic and the features of the remaining modality when the state vector output by the dynamic modality state detector indicates that there is a lack of modality, so as to realize cross-scale inference compensation between macroscopic image features and microscopic pathological features. The fault-tolerant fusion decision layer based on dynamic mask is used to construct a dynamic mask matrix according to the state vector, and to fuse the features extracted by the parallel feature encoding branch and the features reconstructed by the multi-directional conditional feature reconstruction engine based on the dynamic mask matrix. The dynamic mask matrix is used to adjust the decision weights of features from different sources during fusion. The global survival risk assessment output module is used to predict and output a survival risk score based on the fusion results of the fault-tolerant fusion decision layer based on dynamic mask. The multi-directional conditional feature reconstruction engine includes: The conditional variational inference branch is used to learn the posterior probability distribution of pathological features by using at least one residual modality feature as a conditional variable when pathological modality is detected as missing, and to generate reconstructed pseudo-pathological features by using reparameterized sampling. The semantic prototype retrieval branch is used to retrieve the most similar visual prototype from the pre-built semantic prototype library when a missing text modality is detected, using the existing visual features as the query vector, and then retrieving the corresponding text prototype vector to fill in the missing text. The visual features include image features and pathological features. The fault-tolerant fusion decision layer based on dynamic masks is also used for: Depending on whether the feature source is original acquisition or reconstruction compensation, a corresponding confidence penalty factor is set in the dynamic mask matrix, and the confidence penalty factor is used to correct the attention score matrix in the multi-head self-attention calculation to reduce the weight of the reconstructed features in the fusion; the magnitude of the confidence penalty factor is dynamically adjusted according to the state vector.
2. The system according to claim 1, characterized in that, The parallel feature encoding branch includes: The macroscopic image feature encoding module is used to extract morphological features of T2WI, blood supply enhancement features of T1WIC and diffusion restriction features of ADC from macroscopic morphological mp-MRI images using a multi-scale feature extraction method, and generate image feature vectors characterizing the macroscopic invasiveness of tumors. The micropathological feature encoding module is used to extract micropathological features focused on areas of dense tumor cells and areas of stromal infiltration from micro-heterogeneous pathological WSI images using a gated attention mechanism based on the UNI basic model. The clinical text encoding module is used to extract semantic features from unstructured text using BERT, and simultaneously extract the embedding features of structured indicators using linear layers. After fusing the semantic features of unstructured text and the embedding features of structured indicators, global clinical features are generated.
3. The system according to claim 1, characterized in that, The semantic prototype retrieval branch is also used for: By using a linear correction layer, the retrieved text prototype vector is fine-tuned in conjunction with the patient's basic statistical characteristics to accommodate individual differences.
4. The system according to claim 1, characterized in that, The system also includes: The dual-path consistency-guided training module is used to randomly mask some modalities in the input during training to force the activation of the reconstruction engine, and to optimize the model by approximating the risk distribution of the full modal input through constrained completion path output risk distribution.
5. A fault-tolerant multimodal medical feature reconstruction method, characterized in that, The method includes: It receives raw multi-source medical data and outputs state vectors representing the presence or absence of each modality; Feature extraction was performed on the macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical texts in the original multi-source medical data, and mapped to a unified potential feature space. When the state vector indicates the presence of a missing modality, the features of the missing modality are reconstructed and compensated based on the cross-scale semantic generation logic and the features of the remaining modality, so as to achieve cross-scale inference compensation between macroscopic imaging features and microscopic pathological features. A dynamic mask matrix is constructed based on the state vector, and the extracted features and reconstructed features are fused based on the dynamic mask matrix. The dynamic mask matrix is used to adjust the decision weights of features from different sources during fusion. Based on the fusion results, a prediction is made and a survival risk score is output. When the state vector indicates the presence of a missing modality, based on cross-scale semantic generation logic, the features of the missing modality are reconstructed and compensated based on the features of the remaining modality to achieve cross-scale inference compensation between macroscopic imaging features and microscopic pathological features, including: When pathological modality loss is detected, at least one residual modality feature is used as a conditional variable. The posterior probability distribution of the pathological feature is learned through a conditional variational autoencoder, and reparameterized sampling is used to generate reconstructed pseudo-pathological features. When a missing text modality is detected, the existing visual features are used as query vectors to retrieve the most similar visual prototype from a pre-built semantic prototype library and then the corresponding text prototype vector is retrieved to fill the missing text; the visual features include image features and pathological features. The extracted features and reconstructed features are fused based on the dynamic mask matrix, including: Depending on whether the feature source is original acquisition or reconstruction compensation, a corresponding confidence penalty factor is set in the dynamic mask matrix, and the confidence penalty factor is used to correct the attention score matrix in the multi-head self-attention calculation to reduce the weight of the reconstructed features in the fusion; the magnitude of the confidence penalty factor is dynamically adjusted according to the state vector.
6. The method according to claim 5, characterized in that, Feature extraction was performed on the macroscopic morphological mp-MRI images, microscopic heterogeneous pathological WSI images, and clinical texts from the original multi-source medical data, including: A multi-scale feature extraction method was used to extract morphological features of T2WI, blood supply enhancement features of T1WIC, and diffusion restriction features of ADC from macroscopic morphological mp-MRI images, generating image feature vectors characterizing the macroscopic invasiveness of tumors. We employed a gated attention mechanism based on the UNI model to extract micropathological features focused on areas of dense tumor cells and areas of stromal invasion from microscopic heterogeneous pathological WSI images. BERT is used to extract semantic features from unstructured text, and a linear layer is used to extract embedding features of structured indicators. The semantic features of unstructured text and the embedding features of structured indicators are then fused to generate global clinical features.
7. The method according to claim 5, characterized in that, When a missing text modality is detected, the existing visual features are used as the query vector to retrieve the most similar visual modality from a pre-built semantic modality library, and the corresponding text modality vector is retrieved to fill the missing modality, including: By using a linear correction layer, the retrieved text prototype vector is fine-tuned in conjunction with the patient's basic statistical characteristics to accommodate individual differences.