Multi-modal data fusion-based idiopathic membranous nephropathy prognosis prediction method and system

Abstract

The invention belongs to the technical field of intelligent medical treatment and clinical decision, and particularly relates to an idiopathic membranous nephropathy prognosis prediction method and system based on deep learning and a cross-modal attention mechanism. In order to realize accurate prognosis evaluation of IMN, depth features of a kidney tissue pathological image are automatically extracted through a deep learning technology and are innovatively and deeply fused with clinical data of a patient, and finally a high-precision and explainable intelligent prediction model is constructed to guide clinical early intervention and individualized treatment decision.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence-assisted medical and clinical decision-making technology, specifically relating to a method and system for predicting the prognosis of idiopathic membranous nephropathy (IMN) based on deep learning and cross-modal attention mechanisms. This invention aims to achieve quantitative, high-precision, and interpretable prediction of the long-term hard endpoint prognostic risk of renal diseases in IMN patients by deeply integrating the features of patients' clinical laboratory indicators and whole-slice images (WSI). Background Technology

[0002] Idiopathic membranous nephropathy (IMN) is the most common pathological type of nephrotic syndrome in adults, and its clinical course is highly heterogeneous and uncertain. Although approximately 30% of patients achieve spontaneous remission, a staggering 30%–40% experience disease progression, and of these, about 30% may irreversibly progress to end-stage renal disease (ESRD) within 10–15 years. This makes accurate prognostic risk stratification of patients a critical challenge in clinical practice, particularly in selecting treatment options (such as the use of immunosuppressants).

[0003] Current technical solutions for IMN prognostic assessment have the following core limitations:

[0004] 1. Inherent bottlenecks of clinical models: Existing clinical models (such as the Toronto Risk Score) rely only on functional indicators (such as urine protein and creatinine) and lack integration of a large number of nonlinear and high-dimensional morphological features contained in the renal biopsy tissue pathology images, so it is difficult to break through the bottleneck in prediction accuracy.

[0005] 2. Subjectivity and inefficiency of traditional pathological assessment: Currently, pathological diagnosis mainly relies on doctors' empirical staging or descriptive assessment (such as the degree of thickening of the nail process and basement membrane) under microscopic observation. This method is highly subjective, difficult to quantify, and takes a long time to process a whole slice image (WSI), with poor repeatability, and cannot meet the needs of large-scale, standardized prediction.

[0006] 3. Limitations of single-modal AI analysis: Existing deep learning research mostly focuses on the analysis of pathological images themselves, such as glomerular identification and lesion classification. It has failed to effectively solve the problem of heterogeneous data fusion between high-dimensional, unstructured pathological image features and low-dimensional clinical indicators, and lacks deep interaction and mutual verification, resulting in models that are not comprehensive and robust enough.

[0007] 4. Lack of prediction of hard endpoints: Most existing studies focus on surrogate endpoints such as "relief of proteinuria", lacking the ability to directly predict hard endpoint events related to patient survival, such as "doubling of serum creatinine" and "ESRD".

[0008] In summary, existing technologies lack an IMN prognostic prediction scheme that can objectively and quantitatively integrate clinical functional data and pathological morphological data in depth, and provide interpretable evidence. Summary of the Invention

[0009] This invention addresses the aforementioned problems by providing a prognostic classification method and system for IMN based on multimodal data fusion. This method utilizes deep learning technology to automatically extract deep features from renal tissue pathological images and clinical indicators, and achieves dynamic and deep feature fusion through a cross-modal attention mechanism. Ultimately, it constructs a high-precision, interpretable intelligent prediction model, enabling accurate assessment of IMN prognosis.

[0010] To achieve the above objectives, the present invention employs the following technical solution:

[0011] In a first aspect, the present invention provides an IMN prognostic prediction method based on multimodal data fusion, comprising the following steps:

[0012] Step 1: Acquire multimodal data of IMN patients, including clinical data and whole-slice images (WSI) of the kidney, and preprocess them separately;

[0013] Step 2: Extract single-modal deep features from the preprocessed data and construct clinical feature vectors and pathological image feature vectors based on multi-instance learning aggregation, respectively.

[0014] Step 3: Construct a cross-modal attention fusion module, using clinical feature vectors as queries to dynamically weight and guide the pathological image feature vectors to generate multimodal fusion features;

[0015] Step 4: Input the multimodal fusion features into the classifier and output the patient's adverse prognostic risk probability within a set time period.

[0016] Furthermore, step 1 specifically includes:

[0017] Collect clinical data from IMN patients, including but not limited to: demographic information (age, sex, body mass index, smoking history at biopsy), comorbidities, blood pressure, renal function-related indicators (24-hour urine protein quantification, serum albumin, serum creatinine, eGFR), immunological indicators (anti-PLA2R antibody titer), liver function, lipid metabolism and coagulation function parameters, and standardize continuous clinical indicators.

[0018] Obtain matching kidney tissue slices (preferably PASM stained), digitize them into whole slice images (WSI), and use a pre-trained semantic segmentation Mask R-CNN model to automatically identify and crop all glomerular region image patches from the whole slice images to ensure the targeting of feature extraction;

[0019] Prognostic labeling is defined as follows: a poor prognostic label is defined as the patient reaching any of the following hard endpoints within a certain period of time after diagnosis: 1) doubling of serum creatinine; 2) progression to end-stage renal disease; 3) sustained decrease in eGFR ≥50%; 4) death.

[0020] Furthermore, step 2 specifically includes:

[0021] The standardized clinical indicators are concatenated into a one-dimensional vector. Clinical feature vectors after dimension mapping are obtained through a multilayer perceptron. ;

[0022] We used a pre-trained deep convolutional neural network, DenseNet-121, as a feature extractor to extract each glomerular image patch. Instance features .

[0023] An attention-based multiple instance learning (MIL) mechanism is used to aggregate features from all instances. First, the attention weight for each instance is calculated. :

[0024] ;

[0025] in, and For a learnable parameter matrix, Indicates transpose, h j Let N represent the j-th instance feature, and N represent the number of instance features.

[0026] According to the weight For instance features By performing weighted summation, the global pathological image feature vector is obtained. :

[0027] .

[0028] Furthermore, step 3 involves constructing a cross-modal attention fusion module to dynamically guide the extraction of pathological features using clinical features. Specifically:

[0029] clinical feature vector Mapped to query vector , pathological image feature vector Mapped to key vectors respectively Sum vector V:

[0030]

[0031] K

[0032] V

[0033] FC stands for Fully Connected Layer Operation.

[0034] Calculate cross-modal attention weights: Calculate the query vector With key vector The interaction correlation is used to obtain the cross-modal attention weight matrix. :

[0035]

[0036] in, Let be the dimension of the key vector. Scaling factor This indicates the matrix transpose.

[0037] Using the weight matrix Weighting the value vector V generates clinically guided image features. and compared it with clinical feature vectors and image feature vector Concatenate the features to obtain multimodal fusion features. :

[0038]

[0039]

[0040] Concat means concatenation.

[0041] Furthermore, step 4 specifically involves:

[0042] The fusion features The input prognostic classification neural network, through fully connected layers and activation functions, outputs the probability of a patient progressing to a poor prognosis within a set period. ;

[0043] Based on risk probability Patients can be intelligently divided into two layers: "good prognosis" and "poor prognosis," or into three layers: low risk, medium risk, and high risk.

[0044] Secondly, the present invention provides a prognostic prediction system for idiopathic membranous nephropathy based on multimodal data fusion, comprising:

[0045] Data input module: used to receive multimodal data;

[0046] Data preprocessing module: used to perform Z-score normalization of clinical data and ROI segmentation of pathological images;

[0047] Feature extraction module: used to extract single-modal features using deep neural networks and the Attention-MIL formula mentioned above;

[0048] Cross-modal fusion module: used to execute the above cross-modal attention calculation formula to generate clinically guided fusion features;

[0049] Prediction output module: used to calculate risk probabilities based on fused features and generate a visual heatmap.

[0050] Compared with the prior art, the present invention has the following significant and quantifiable advantages:

[0051] 1. Significantly improved prediction accuracy: By integrating clinical functional indicators and pathological morphological characteristics, the model is significantly better than a single model in predicting long-term adverse hard endpoints of IMN (such as ESRD), solving the problem of insufficient prediction accuracy in existing technologies.

[0052] 2. The integration mechanism is scientific and reasonable: it adopts... From clinical, The attention mechanism designed from images simulates the diagnostic thinking of a nephrologist when reviewing images in conjunction with lab reports, ensuring the logic and effectiveness of feature interactions.

[0053] 3. Provide clinical decision support: the risk probability output by the model. It directly targets hard endpoints, providing clinicians with objective and quantitative evidence when developing key plans such as immunosuppressant therapy or observation and waiting in the early stages of patient diagnosis.

[0054] 4. Prospective risk stratification: Effectively identify occult high-risk patients whose clinical indicators seem normal (e.g., low proteinuria, acceptable eGFR) but whose pathological damage is severe (e.g., significant basement membrane thickening), thus avoiding delays in the best treatment time.

[0055] 5. High interpretability: This invention can be interpreted based on attention weights. and Generating heatmaps visually demonstrates which clinical indicators (such as antibody titers) and pathological regions (such as severe glomerulosclerosis) contribute most to high risk, enhancing the transparency and clinical credibility of model results. Attached Figure Description

[0056] Figure 1 The flowchart provided for the embodiments of the present invention illustrates the overall architecture from multimodal data input, feature extraction, cross-modal fusion to prognostic risk output.

[0057] Figure 2This is a schematic diagram of the cross-modal attention fusion module in an embodiment of the present invention, illustrating the interaction of query vector Q, key vector K, and value vector V, and the generation process of fusion features.

[0058] Figure 3 The system structure diagram provided for the embodiments of the present invention illustrates the connection relationship of each functional module. Detailed Implementation

[0059] To further illustrate the technical solution of the present invention, the present invention will be further described below through embodiments.

[0060] Example 1

[0061] This embodiment presents a prognostic prediction method for idiopathic membranous nephropathy based on multimodal data fusion. This method achieves deep fusion of pathological images and clinical data (e.g., [missing information]) through a multimodal deep neural network. Figure 1 (As shown), including the following steps:

[0062] Step 1: Prepare multimodal data and preprocess it;

[0063] (1) Multimodal data preparation:

[0064] This embodiment uses the IMN patient dataset from the Department of Nephrology of Shanxi Provincial People's Hospital, which includes a total of 728 patient records.

[0065] Inclusion criteria: 1) confirmed by renal biopsy pathology of IMN; 2) age ≥18 years; 3) complete baseline clinical data and high-quality PASM stained slides.

[0066] Clinical data: 48 clinical data points were collected from patients at the time of diagnosis, including:

[0067] Demographic data: age, sex, body mass index, and smoking history at the time of biopsy;

[0068] Comorbidities: Diabetes mellitus, hypertension;

[0069] Renal function indicators: serum creatinine (SCr), estimated glomerular filtration rate (eGFR);

[0070] Proteinuria indicators: 24-hour urine protein quantification;

[0071] Serum marker: serum albumin (Alb);

[0072] Immunological marker: serum anti-PLA2R antibody titer;

[0073] Other indicators: liver function, lipid metabolism, and coagulation function parameters.

[0074] Prognostic label (hard endpoint definition): Poor prognosis is defined as the occurrence of any of the following events within 12 months of diagnosis:

[0075] Serum creatinine levels doubled from baseline;

[0076] eGFR continues to decline by ≥50%;

[0077] Progression to end-stage renal disease (ESRD, defined as requiring dialysis or kidney transplantation);

[0078] He died from complications related to kidney disease.

[0079] (2) Data preprocessing:

[0080] (2.1) Handling missing values:

[0081] Clinical features with a missing rate >30% were removed, and the remaining missing features were filled using a multiple linear imputation algorithm.

[0082] (2.2) Standardization of clinical indicators:

[0083] Forty-eight clinical indicators were selected as input features. For continuous variables (such as 24-hour urinary protein quantification, anti-PLA2R antibody titer, eGFR, etc.), Z-score standardization was used.

[0084] ;

[0085] in, The mean of this metric is used for the training set. The standard deviation is given. The processed data is concatenated into a one-dimensional vector. .

[0086] (2.3) Pathological image processing:

[0087] The pathological images were digitized and an image pyramid was constructed.

[0088] Digital scanning: PASM stained sections are scanned at 40X magnification using a digital pathology scanner to generate whole-slide image (WSI) files in .svs or .ndpi format.

[0089] Image pyramid construction: A 20x magnification is selected, and the Otsu thresholding method is used to remove background blank areas, generating a tissue mask. A sliding window technique is then employed to... Using pixels as the size and 50% as the overlap rate, the tissue region is cropped into a sequence of image patches.

[0090] (2.4) Glomerular localization and noise reduction:

[0091] A Mask R-CNN model, pre-trained on the COCO dataset and fine-tuned on the kidney pathology dataset, was loaded. Artifacts such as background, bubbles, and staining spots were removed, retaining only image patches with a confidence score > 0.8 and containing intact glomerular structures. Finally, one image patch containing [database structure details missing] was obtained for each patient. A collection of effective glomerular image patches ,in For variable length sequences ( ).

[0092] Step 2: Extract single-modal depth features from the preprocessed data;

[0093] (1) Extraction of depth features from pathological images

[0094] (1.1) Instance Feature Encoding: Using the DenseNet-121 network, its original fully connected classification layers are removed, leaving only the last convolutional block. Each element of size... Image blocks The input is fed into the network, passes through a global average pooling layer, and the output dimension is... Instance feature vectors :

[0095] ;

[0096] (1.2) Attention-MIL (Multi-Instance Learning) Convergence: In order to solve To address the issue of variable values, the following aggregation layer is designed: two learnable weight matrices are set. and and vectors ,in 8.

[0097] Calculate the contribution weight of each instance feature vector to the overall prognostic risk. , all instance feature vectors The weighted aggregation is used to form a fixed-length image feature vector. :

[0098] ;

[0099] .

[0100] (2) Construction of clinical feature vectors

[0101] Construct a simple multilayer perceptron (MLP) containing two fully connected layers (FC): FC1: input 48 → output 128 (ReLU activation); FC2: input 128 → output 256, ultimately obtaining the dimension-mapped clinical feature vector. .

[0102] Step 3, as follows Figure 2 As shown, the extracted features are fused through cross-modal attention to obtain multimodal fused features;

[0103] This step dynamically guides the interpretation of pathological image features based on clinical information, employing an asymmetric attention structure, including:

[0104] (1) Feature transformation: and Dimension matching and transformation are performed using fully connected layers (FC) to obtain the values ​​used for attention computation. vector:

[0105] ,in ;

[0106] ,in ;

[0107] ,in ;

[0108] (2) Cross-modal attention computation: computation and The dot product similarity was calculated and normalized using the softmax function to obtain the cross-modal interaction weights. :

[0109] =softmax( );

[0110] in, , Let K be the transpose of matrix K.

[0111] (3) Generate guiding features: Weights Acting on Generate clinically guided image features :

[0112] ;

[0113] (4) Final feature fusion: In order to maximize the robustness of the model, the final fused features Using ternary stitching, i.e., clinically guided image features Clinical features after dimensional mapping and the value vector of image features To splice:

[0114] ;

[0115] Step 4: Input the multimodal fusion features into the classifier and output the probability of adverse prognosis risk for the patient within a set time period;

[0116] The fusion features The input prognostic classification neural network, through fully connected layers and activation functions, outputs the probability of a patient progressing to a poor prognosis within 12 months. ;

[0117] Based on risk probability Patients can be intelligently divided into two layers: "good prognosis" and "poor prognosis," or into three layers: low risk, medium risk, and high risk.

[0118] (1) Prognostic classification neural network: fusing features The input is fed into a classifier consisting of multiple layers (including Dropout layers and a Softmax function) of a fully connected neural network. Fully connected layer 1: Input 768 Output 128 (ReLU + BatchNorm). Dropout layer: dropout rate set to 0.5 to prevent overfitting.

[0119] (2) Output layer: Fully connected layer 2: Input 128 Output 1. Sigmoid activation: Output The probability values ​​between.

[0120] (3) Risk assessment: Set the threshold to 0.5, and output the probability If the risk is high (poor prognosis), it is considered high risk; otherwise, it is considered low risk.

[0121] .

[0122] (3) Visual interpretation of results:

[0123] The interpretability of the model is achieved by generating pathological heatmaps: First, the attention weights assigned to each glomerular image patch by the attention-based MIL layer are extracted. Subsequently, these weights were... Interval normalization was performed, and Jet color mapping was used to map high-weighted areas (close to 1) to red and low-weighted areas (close to 0) to blue. Finally, the color layer was upsampled to the original WSI resolution and displayed overlaid on the original image with a transparency of 0.4. This heatmap visually demonstrates the microscopic pathological areas that the model focuses on for prognostic assessment, such as the degree of basement membrane thickening or glomerular sclerosis, thus assisting clinicians in validating results.

[0124] Model training strategy and parameter settings

[0125] In optimizing the model training configuration, considering the potential imbalance between positive and negative samples, a weighted binary cross-entropy loss function (Weighted BCE Loss) is used for end-to-end optimization:

[0126]

[0127] in, The algorithm is dynamically calculated based on the proportion of positive samples in the training set to balance the samples by increasing the penalty weight for the minority class.

[0128] The AdamW optimizer was selected, and the weight decay was set to... Regarding the learning rate strategy, the initial learning rate is set to... It employs a cosine annealing strategy and sets the minimum learning rate to [value missing]. The data augmentation process is only performed on the pathological image patches in the training set. Specifically, it includes random horizontal / vertical flipping, random rotation of 90° / 180° / 270°, and color jittering operations with brightness, contrast, and saturation fluctuating by 0.1 each, thereby enhancing the robustness of the model.

[0129] To verify the effectiveness of this invention, ablation experiments were conducted on a test set (n=145) for comparison. The results are shown in Table 1:

[0130] Table 1 Ablation Experiment Results

[0131] Model Method Modal input AUC Accuracy Sensitivity Specificity Random Forest Clinical only 0.712 0.685 0.620 0.741 ResNet-50 Image only 0.745 0.703 0.685 0.714 DenseNet-MIL Image only 0.781 0.732 0.710 0.755 Method of the present invention Clinical + Image 0.865 0.821 0.805 0.834

[0132] Example 2

[0133] like Figure 3 As shown, this embodiment of the idiopathic membranous nephropathy prognostic prediction system based on multimodal data fusion is used to perform the aforementioned corresponding steps, including:

[0134] The data input module is used to receive multimodal data;

[0135] The data preprocessing module is used to perform Z-score normalization of clinical data and ROI segmentation of pathological images;

[0136] The feature extraction module is used to extract single-modal features using deep neural networks and the Attention-MIL formula mentioned above;

[0137] The cross-modal fusion module is used to execute the cross-modal attention calculation formula mentioned above to generate clinically guided fusion features;

[0138] The prediction output module is used to calculate the risk probability based on the fused features and generate a visual heatmap.

[0139] The foregoing has shown and described the main features and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

[0140] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A prognosis prediction method for idiopathic membranous nephropathy based on multi-modal data fusion, characterized in that, The method comprises the following steps: Step 1, obtaining multi-modal data of idiopathic membranous nephropathy patients, including clinical data and kidney pathological whole section images, and respectively pre-processing; Step 2, extracting single-modal deep features from the pre-processed data, and respectively constructing a clinical feature vector and a pathological image feature vector based on multi-instance learning aggregation; Step 3, constructing a cross-modal attention fusion module, using the clinical feature vector as a query to dynamically weight and guide the pathological image feature vector, and generating multi-modal fusion features; Step 4, inputting the multi-modal fusion features into a classifier to output the risk probability of adverse prognosis of the patient within a certain time period.

2. The method according to claim 1, wherein, The step 1 is specifically: Collecting clinical data of idiopathic membranous nephropathy patients, and performing standardization processing on continuous clinical data; Obtaining matched kidney tissue sections, digitizing them into whole section images, and using a pre-trained semantic segmentation Mask R-CNN model to automatically identify and crop all glomerular region image blocks in the whole section images; Defining and labeling prognosis labels, and defining an adverse prognosis label as reaching any of the following hard endpoints within a certain time after diagnosis: 1) doubling of serum creatinine; 2) entering end-stage renal disease; 3) eGFR continuously decreasing by ≥50%; 4) death.

3. The method according to claim 2, wherein, The step 2 is specifically: Concatenate the standardized clinical indicators into a one-dimensional vector , obtain the dimension-mapped clinical feature vector through a multi-layer perception machine ; Using a pre-trained deep convolutional neural network DenseNet-121 as a feature extractor, extract the features of each glomerular image block Examples of features of the instance ; An attention-based multi-instance learning mechanism is adopted, which aggregates all instance features and calculates the attention weight of each instance : ； wherein, and are learnable parameter matrices, denotes a transpose, h j denotes the jth instance feature, and N denotes the number of instance features; According to the weights On the instance features Weighted sum to get the global pathological image feature vector : 。 4. The idiopathic membranous nephropathy prognosis classification method based on multi-modal data fusion according to claim 3, characterized in that, The step 3 is specifically: mapping the clinical feature vector to a query vector mapping the pathology image feature vector to a key vector and a value vector V: ； K ； V ； Wherein, FC represents a full connection layer operation; Compute cross-modal attention weights: Compute query vector interaction with key vectors to get cross-modal attention weight matrix : ； wherein, is the dimension of the key vector, is a scaling factor, denotes matrix transposition; Utilizing cross-modal attention weight matrices The value vector V is weighted to generate a clinically guided image feature , and is spliced with the clinical feature vector and the value vector to obtain a multi-modal fusion feature : ； ； Wherein, Concat represents splicing.

5. The method of claim 4, wherein the method is characterized by, The step 4 is specifically: The fusion features are combined The input prognosis classification neural network outputs, through a fully connected layer and an activation function, a probability of risk of the patient progressing to a poor prognosis within a set period ; According to the risk probability The patient intelligence is divided into two layers of "good prognosis" and "poor prognosis", or into three layers of low risk, medium risk and high risk.

6. A prognosis classification system for idiopathic membranous nephropathy based on multi-modal data fusion, characterized in that, It comprises: A data input module for receiving multi-modal data; A data preprocessing module for performing Z-score standardization of clinical data and ROI segmentation of pathological images; A feature extraction module for extracting single-modal features using deep neural networks and multi-instance learning mechanisms; A cross-modal fusion module for generating clinically guided multi-modal fusion features using cross-modal attention; A prediction output module for calculating risk probability and generating a visualized heat map according to the multi-modal fusion features.

Images