A method and system for predicting severe ischemic events in the nervous system caused by arteritis based on multimodal deep learning.

Abstract

This invention discloses a method and system for predicting severe ischemic events in the nervous system caused by large vessel arteritis based on multimodal deep learning. Belonging to the field of intelligent medical image analysis and clinical risk prediction technology, the method includes: acquiring the patient's magnetic resonance angiography data and clinical information; preprocessing the image data; extracting three-dimensional morphological feature vectors from the point cloud of the blood vessel surface based on a point cloud neural network; extracting blood vessel wall feature vectors from the images based on a three-dimensional convolutional neural network; encoding clinical information to generate clinical feature vectors based on a pre-trained basic model; fusing the above three feature vectors to generate a multimodal fusion feature representation; and outputting an individualized risk score based on this fusion feature through a discrete-time survival analysis model. This invention, through multimodal deep collaborative modeling, achieves automated deep fusion analysis of three-dimensional vascular morphology, blood vessel wall structure, and clinical information, significantly improving prediction accuracy, objectivity, and clinical applicability.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical image analysis and clinical risk prediction technology, and in particular relates to a method and system for predicting severe ischemic events of the nervous system in arteritis based on multimodal deep learning. Background Technology

[0002] Takayasu arteritis (TAK) is an autoimmune vasculitis characterized by chronic inflammation of large and medium-sized arteries, commonly affecting the brachiocephalic trunk, common carotid artery, and subclavian artery, leading to thickening of the vessel walls, narrowing or even occlusion of the lumen, resulting in chronic or acute ischemia of the brain and optic nerve system. Currently, risk assessment for severe neurological ischemic events in TAK patients mainly relies on clinical experience, analysis of single or limited clinical indicators, and traditional statistical models. These methods typically stratify risk based on factors such as patient age, disease activity score, history of ischemic events, laboratory inflammatory markers, and the number of affected vessels. Furthermore, magnetic resonance angiography (MRI), as an important imaging modality for TAK, can non-invasively and intuitively display vessel wall structure, lumen morphology, and lesion extent, and has been widely used in disease diagnosis and follow-up assessment. With the development of artificial intelligence technology, medical image analysis methods based on convolutional neural networks have also made progress in tasks such as lesion detection, segmentation, and classification, providing technical support for the auxiliary assessment of vascular diseases.

[0003] However, existing risk assessment methods still have significant limitations. First, traditional statistical models typically rely on linear assumptions, making it difficult to characterize the complex nonlinear relationships between various clinical indicators and imaging features. Furthermore, they have limited ability to handle the heterogeneity of different patients' imaging presentations, vascular morphology, and clinical conditions. Second, in clinical practice, magnetic resonance angiography data primarily relies on manual image interpretation for qualitative or semi-quantitative analysis. The analysis is often limited to simple features such as the degree of local stenosis and the location of the most severe lesion, making it difficult to systematically uncover the potential risk information contained in changes in vessel wall signals and the overall three-dimensional morphological structure. In addition, existing deep learning-based medical image analysis methods mostly focus only on two-dimensional or local features of the images, lacking the ability to characterize the overall three-dimensional geometry of the complex tubular structure of blood vessels. Simultaneously, existing technologies suffer from insufficient modeling capabilities in fusing imaging information with multidimensional clinical data, failing to achieve truly multimodal collaborative analysis. Moreover, the clinical information of patients with aortitis is diverse in source, complex in dimensions, and inconsistent in format. Existing methods struggle to uniformly encode and deeply utilize this heterogeneous clinical information, thus limiting the accuracy and generalization ability of predictive models. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method and system for predicting severe ischemic events in the nervous system caused by large vessel arteritis based on multimodal deep learning, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for predicting severe ischemic events in the nervous system caused by large vessel arteritis based on multimodal deep learning, comprising: To obtain magnetic resonance angiography data and clinical information from patients with Takayasu arteritis; The magnetic resonance angiography data is preprocessed to obtain standardized three-dimensional vascular images; Based on point cloud neural networks, the point cloud of the blood vessel surface obtained by reconstruction from standardized three-dimensional blood vessel images is processed to extract the three-dimensional morphological feature vector of the blood vessel. Based on a three-dimensional convolutional neural network, the standardized three-dimensional vascular image is processed to extract the vascular wall feature vector; Based on the pre-trained basic model, the clinical information is encoded to generate a clinical feature vector; The three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector are fused to generate a multimodal fusion feature representation; Based on the multimodal fusion feature representation, an individualized risk score for severe neurological ischemic events in patients with aortitis is output through a discrete-time survival analysis model.

[0006] Preferably, the preprocessing step for magnetic resonance angiography data includes: Spatial resampling is performed on the magnetic resonance angiography data to unify the spatial resolution; Intensity normalization is performed on the resampled data; The region of interest containing the target blood vessel is cropped from the normalized data to obtain the standardized three-dimensional blood vessel image.

[0007] Preferably, the step of extracting the three-dimensional morphological feature vector of blood vessels includes: Layered sampling and local feature aggregation are performed on the point cloud of the blood vessel surface; A global feature vector representing the overall three-dimensional spatial geometry of the blood vessel is generated through a global feature extraction layer in a point cloud neural network, which serves as the three-dimensional morphological feature vector of the blood vessel.

[0008] Preferably, the step of extracting the feature vector of the blood vessel wall is as follows: Standardized 3D vascular images are input into a 3D residual convolutional neural network; After multiple convolutions and nonlinear mappings in the three-dimensional residual convolutional neural network, a global pooling operation is performed on the final output feature map to generate the blood vessel wall feature vector.

[0009] Preferably, the step of generating clinical feature vectors includes: Clinical information, including symptom descriptions, laboratory test results, and disease activity scores, is integrated into a text description; The text description is encoded using a pre-trained base model text encoder, which outputs a high-dimensional semantic vector as the clinical feature vector.

[0010] Preferably, the step of generating a multimodal fusion feature representation includes: The three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector are concatenated to obtain the concatenated feature vector; The concatenated feature vector is input into a feature mapping unit containing a fully connected layer and a nonlinear activation function for dimensionality reduction and fusion, and the multimodal fusion feature representation is output.

[0011] Preferably, the construction and execution of the discrete-time survival analysis model includes: The continuous follow-up time is discretized into multiple ordered time intervals; Construct a prediction network whose input is the multimodal fusion feature representation and whose output is the predicted risk probability of the patient experiencing a severe neurological ischemic event in each time interval. The individualized risk score is calculated based on the predicted risk probability.

[0012] Preferably, the steps for calculating the individualized risk score are as follows: The predicted risk probabilities within each of the aforementioned time intervals are weighted and summed, with the weights corresponding to the importance of each time interval.

[0013] Preferably, the method further includes a risk stratification step: Based on a preset risk threshold, the individualized risk score is divided into a high-risk level or a low-risk level.

[0014] Secondly, the present invention also provides a large-scale arteritis neurological severe ischemic event prediction system based on multimodal deep learning, for implementing the method described in the first aspect, the system comprising: The image data acquisition and preprocessing module is used to acquire and preprocess magnetic resonance angiography data; The 3D morphological feature extraction module for blood vessels is used to extract 3D morphological feature vectors of blood vessels based on point cloud neural networks. The blood vessel wall feature extraction module is used to extract blood vessel wall feature vectors based on a three-dimensional convolutional neural network. The clinical information feature encoding module is used to encode clinical information based on a pre-trained basic model and generate clinical feature vectors. A multimodal feature fusion module is used to fuse the three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector to generate a multimodal fused feature representation; The risk prediction and stratification module is used to output individualized risk scores and perform risk stratification based on the multimodal fusion feature representation and discrete-time survival analysis model.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention, through its technical features of "extracting 3D morphological feature vectors of blood vessels based on point cloud neural networks," "extracting feature vectors of blood vessel walls based on 3D convolutional neural networks," "generating clinical feature vectors based on a pre-trained basic model," and "fusion of the three feature vectors," for the first time collaboratively models point cloud features reflecting the macroscopic spatial geometry of blood vessels, image features reflecting changes in the microstructure of blood vessel walls, and clinical semantic features reflecting the overall condition of patients. This deep fusion mechanism can comprehensively capture the multidimensional pathological information of aortitis, overcoming the shortcomings of existing technologies that rely on a single data source or simple feature combinations, resulting in insufficient modeling and one-sided information utilization. Therefore, it achieves significantly higher prediction consistency index (C-index) and time-dependent AUC values ​​than single-feature models in both internal and external system validation.

[0016] This invention employs point cloud neural networks, three-dimensional convolutional neural networks, and pre-trained basic models for feature extraction and encoding, respectively, to achieve automated and intelligent processing of complex morphological and structural features in raw magnetic resonance angiography data, as well as heterogeneous clinical text information. This process eliminates the need for human experience in feature delineation, measurement, or subjective scoring, effectively removing subjective biases introduced by human interpretation differences in traditional methods. This results in more objective and stable risk assessment results and facilitates standardized promotion across different medical institutions.

[0017] This invention utilizes a specific technical feature of "extracting three-dimensional morphological feature vectors of blood vessels based on point cloud neural networks." Specifically targeting blood vessels, a continuous tubular structure, it employs a neural network architecture dedicated to processing three-dimensional point set data. This method can directly learn the coordinates of the point cloud on the blood vessel surface, effectively extracting and quantifying overall spatial topological and morphological features such as stenosis degree, curvature changes, and branching angles. This overcomes the shortcomings of existing methods based on two-dimensional slicing or simple three-dimensional measurement, which suffer from coarse depiction of the overall geometric morphology of blood vessels and limited feature representation capabilities.

[0018] This invention utilizes a technique of "generating clinical feature vectors based on pre-trained base models" to perform high-dimensional semantic encoding of textual clinical information such as symptom descriptions and examination results using base models (such as the CLIP text encoder) pre-trained on large-scale corpora. This method can understand the contextual semantic relationships in clinical texts, transforming unstructured information into semantically rich feature representations, thus overcoming the limitations of existing technologies that can only process structured numerical variables and are unable to effectively utilize rich clinical information such as textual descriptions.

[0019] The technical feature of this invention, "outputting individualized risk scores through discrete-time survival analysis models," employs a discrete-time survival analysis framework to construct a predictive model. This framework can correctly handle right-censored data commonly found in clinical follow-up studies and output the risk probabilities of patients across multiple future time intervals, ultimately culminating in an individualized score. Compared to existing methods that only output the probability of a single event or perform simple binary classification, this provides more refined and dynamic risk information that better meets the time-series requirements of disease management, thus providing a more timely basis for clinical development of individualized follow-up intervention strategies. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a diagram illustrating the overall architecture and workflow of an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the working principle of three different modal feature extraction methods according to an embodiment of the present invention; Figure 3 This is a comparison chart of Kaplan-Meier survival curves of the multi-feature fusion model and the single-feature model in the internal validation set of this invention embodiment; Figure 4 This is a time-dependent ROC curve of the multi-feature fusion model at different time points in the internal validation set of this invention embodiment; Figure 5 This is a comparison chart of Kaplan-Meier survival curves for different combinations of clinical features in the internal validation set of this invention. Figure 6 This is a time-dependent ROC curve of different combinations of clinical features at different time points in the internal validation set of this invention. Figure 7 This is a Kaplan-Meier survival curve and time-dependent ROC curve at different time points on an external validation set, according to an embodiment of the present invention. Figure 8This is a schematic diagram showing the comparison between model prediction results and clinical outcomes for three typical cases (high / low risk, internal / external data) in an embodiment of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0023] Example 1 like Figure 1 As shown, this embodiment provides a method for predicting severe ischemic events in the nervous system caused by large vessel arteritis based on multimodal deep learning, including: Step 1: Obtain magnetic resonance angiography data and clinical information from patients with Takayasu arteritis; Specifically, this embodiment provides two independent data queues: (1) Internal Development and Validation Cohort (ZS Cohort): Data from 359 patients diagnosed with TAK and with supraclavicular artery involvement were collected from Zhongshan Hospital affiliated with Fudan University. This included baseline clinical data and magnetic resonance angiography (MRA) data, and follow-up was conducted to record neurological SIE endpoint events.

[0024] (2) External independent validation cohort: Data from 53 eligible TAK patients were collected from three external medical centers to form a multicenter validation set.

[0025] In this embodiment, the ZS queue is randomly divided into a training set (n=300) and an internal validation set (n=59). The external validation queue (n=53) is completely independent and does not participate in any step of model building; it is only used to test the generalization ability of the model in the final test.

[0026] Step 2: Preprocess the magnetic resonance angiography data to obtain standardized three-dimensional vascular images; Furthermore, the preprocessing steps for magnetic resonance angiography data include: Spatial resampling is performed on the magnetic resonance angiography data to unify the spatial resolution; Intensity normalization is performed on the resampled data; The region of interest containing the target blood vessel is cropped from the normalized data to obtain the standardized three-dimensional blood vessel image.

[0027] Specifically, the automated standardization process for raw MRA data includes the following steps: (1) Spatial resampling: All three-dimensional MRA volume data were uniformly resampled to an isotropic resolution of 1.0mm x 1.0mm x 1.0mm.

[0028] (2) Intensity normalization: For each patient's MRA data, the mean voxel intensity is calculated in the background region. and standard deviation The entire image is then normalized using z-score to generate a standardized image.

[0029] (3) Region of Interest (ROI) cropping: Based on the preset anatomical coordinate range or by using a simple threshold method to initially locate the vascular region, the three-dimensional bounding box containing all target supraclavicular arteries is automatically cropped to obtain a standardized input image block. .

[0030] (4) Data augmentation during training: During model training, the input 3D image patches are augmented. Apply random data augmentation transformation in real time This includes small-angle 3D rotation, random translation, and the addition of values ​​with a mean of 0 and a standard deviation of 0. Gaussian noise .

[0031] Step 3: Based on the point cloud neural network, process the point cloud of the blood vessel surface obtained by reconstructing the standardized three-dimensional blood vessel image to extract the three-dimensional morphological feature vector of the blood vessel. Furthermore, the steps for extracting the three-dimensional morphological feature vector of blood vessels include: Layered sampling and local feature aggregation are performed on the point cloud of the blood vessel surface; A global feature vector representing the overall three-dimensional spatial geometry of the blood vessel is generated through a global feature extraction layer in a point cloud neural network, which serves as the three-dimensional morphological feature vector of the blood vessel.

[0032] Specifically, the workflow for extracting features from the geometry of blood vessels consists of two stages: Phase 1: Vascular Segmentation and Point Cloud Generation: First, a professional physician manually delineates the lumens of the five arteries above the clavicle on the MRA image to obtain a three-dimensional segmentation mask. Subsequently, the segmentation mask is converted into a triangular mesh surface model, from which samples are uniformly mined. These points form a three-dimensional point cloud data representing the surface of the blood vessel. .

[0033] The second stage: Feature extraction based on point cloud neural networks, specifically including: (1) Network architecture selection: PointNet++[8] network is adopted, the core of which is the hierarchical "SetAbstraction (SA)" layer.

[0034] (2) Input: The input of the module is the above-mentioned point cloud of the blood vessel surface. Each point It includes three-dimensional coordinates.

[0035] (3) Hierarchical feature learning: Each SA layer performs three steps: (4) Sampling: The farthest point sampling algorithm is used to sample the input point set { Select from} A centroid point { }

[0036] (5) Grouping: For each centroid In its radius Find within the neighborhood of the ball Neighboring points .

[0037] (6) Feature extraction: For each neighborhood Points within the region are used to extract local features through a shared-weight multilayer perceptron (MLP). After passing through multiple SA layers, the network gradually aggregates local information to obtain the hierarchical features of the point cloud.

[0038] (7) Feature generation: Finally, a global feature extraction layer (such as max pooling of all point features) is used to output a 1024-dimensional global feature vector, which encodes the overall three-dimensional morphological information of the blood vessels and serves as the second feature vector. ).

[0039] Step 4: Based on a three-dimensional convolutional neural network, process the standardized three-dimensional vascular image to extract the vascular wall feature vector; Furthermore, the specific steps for extracting the feature vector of the blood vessel wall are as follows: Standardized 3D vascular images are input into a 3D residual convolutional neural network; After multiple convolutions and nonlinear mappings in the three-dimensional residual convolutional neural network, a global pooling operation is performed on the final output feature map to generate the blood vessel wall feature vector.

[0040] Specifically, deep image features of blood vessel walls are automatically extracted based on a three-dimensional convolutional neural network, including: (1) Network architecture selection: A pre-trained 3D-ResNet-50 is used as the backbone network. This architecture effectively trains deep networks through residual connections, and its core residual block computation can be expressed as: ,in For input, It combines operations such as convolution, normalization, and activation, effectively mitigating the gradient vanishing problem.

[0041] (2) Input and processing: The input of the module is the preprocessed 3D MRA image block. The image patch passes through a convolutional layer, a batch normalization layer, a ReLU activation function, and multiple residual blocks in sequence.

[0042] (3) Feature generation: At the end of the network, the feature map output by the last convolutional layer is generated. Perform global max pooling operation This is aggregated into a 512-dimensional feature vector. This vector serves as the first feature vector reflecting the characteristics of the blood vessel wall. ).

[0043] Step 5: Based on the pre-trained basic model, encode the clinical information to generate a clinical feature vector; Furthermore, the steps for generating clinical feature vectors include: Clinical information, including symptom descriptions, laboratory test results, and disease activity scores, is integrated into a text description; The text description is encoded using a pre-trained base model text encoder, which outputs a high-dimensional semantic vector as the clinical feature vector.

[0044] Specifically, heterogeneous clinical data is transformed into high-dimensional features that the model can understand, including: (1) Information integration: Integrate the various clinical information of each patient into a coherent text description. (For example: "Patient, female, 38 years old, with a history of hypertension, Kerr score of 3, erythrocyte sedimentation rate of 28 mm / H").

[0045] (2) Feature encoding: Text encoder using a pre-trained CLIP model Describe the text Input this encoder, output a Semantic feature vectors of 512 dimensions (e.g., 512 dimensions) : (1) For longer texts, they can be segmented into sentences. , , ..., After encoding each feature vector separately, the average of the feature vectors is taken: (2) (3) Output: The generated vector serves as the third feature vector comprehensively reflecting the patient's clinical status. ).

[0046] In this embodiment, a schematic diagram illustrating the working principle of three different modal feature extraction methods is shown, as follows: Figure 2 As shown.

[0047] Step 6: Fuse the three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector to generate a multimodal fusion feature representation; Furthermore, the steps for generating multimodal fusion feature representations include: The three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector are concatenated to obtain the concatenated feature vector; The concatenated feature vector is input into a feature mapping unit containing a fully connected layer and a nonlinear activation function for dimensionality reduction and fusion, and the multimodal fusion feature representation is output.

[0048] Specifically, this involves effectively integrating features from different sources, including: (1) Feature concatenation: The first feature vector is concatenated. Second eigenvector and the third eigenvector The features are concatenated along the feature dimension to obtain a merged feature vector. : (3) (2) Feature Mapping and Dimensionality Reduction: The concatenated feature vectors are input into a feature mapping unit consisting of fully connected layers. This unit contains one or more fully connected layers, along with ReLU activation and Dropout layers. Finally, it is mapped to the fusion feature space to generate the final multimodal fusion feature representation. ): (4) in , , , These are learnable parameters.

[0049] Step 7: Based on the multimodal fusion feature representation, output an individualized risk score for severe neurological ischemic events in the patients with aortitis using a discrete-time survival analysis model.

[0050] Furthermore, the construction and execution of the discrete-time survival analysis model includes: The continuous follow-up time is discretized into multiple ordered time intervals; Construct a prediction network whose input is the multimodal fusion feature representation and whose output is the predicted risk probability of the patient experiencing a severe neurological ischemic event in each time interval. The individualized risk score is calculated based on the predicted risk probability.

[0051] Furthermore, the specific steps for calculating an individualized risk score are as follows: The predicted risk probabilities within each of the aforementioned time intervals are weighted and summed, with the weights corresponding to the importance of each time interval.

[0052] Specifically, the core predictor in this embodiment adopts a discrete-time survival analysis framework, which includes: (1) Discretization of survival time: The continuous follow-up time t is divided into J discrete time intervals. ,in , This is the maximum follow-up time.

[0053] (2) Tag construction: For the first For each patient, the event time or censoring time is: The event indicator is ( =1 indicates that an event has occurred. =0 indicates censoring). Define discrete-time labels. : (3) Prediction network: The module receives fused features This is followed by a two-layer fully connected network. Network output. There are 10 nodes, each outputting a probability using the Sigmoid function. This represents the patient's condition within the corresponding time interval. Predicted risk probability of events occurring within the region.

[0054] (4) Loss Function and Training: The sum of binary cross-entropy losses over time intervals is used, which essentially maximizes the likelihood function of discrete-time survival data. Total Loss for: (5) The model is trained end-to-end using the Adam optimizer.

[0055] (5) Risk score output and stratification: After training, the model outputs risk scores for new patients. Each risk probability { A comprehensive, individualized risk score R is calculated using methods such as weighted summation or taking the maximum value. For example: (6) in Weights are used to reflect the importance of time intervals. In clinical applications, patients can be divided into "high-risk" and "low-risk" groups based on the distribution of the risk score R (such as the median or the optimal cutoff value determined through survival analysis).

[0056] In this embodiment, the performance of the trained system is evaluated using the internal validation set and the external independent validation set.

[0057] Evaluation indicators include: The C-index measures the accuracy of the model's ranking of patient risk, and is calculated as follows: (7) in, For indicator functions, , For patients Risk score.

[0058] Hazard Ratio (HR): Calculated using the Cox proportional hazards model, this ratio represents the risk of the high-risk group relative to the low-risk group, along with its 95% confidence interval.

[0059] Kaplan-Meier survival analysis: survival curves were plotted for the high-risk and low-risk groups, and the differences were compared using the Log-rank test.

[0060] Time-dependent ROC curve: calculated at a specific time point The AUC (e.g., 24 months) is used to evaluate the model's discriminative ability at a fixed time point.

[0061] Internal validation results: On the internal validation set (n=57), the multimodal feature fusion model achieved the best performance, with a C-index of 0.86 and a HR of 1.86 (95% CI: 1.28–2.70, p=0.0012). Kaplan-Meier curves showed that the model could significantly distinguish between high- and low-risk patients (Log-rank p=0.0019). The AUC at 24 months was 0.91. Results for different feature combinations are shown in Table 1. Figure 3 and Figure 4 As shown.

[0062] To further explore the role of individual clinical variables, this embodiment analyzed the model performance combining different clinical feature subgroups with overall vascular features (including vessel wall and morphology). Detailed results for different combinations of clinical features are shown in Table 2. All models combining a specific clinical subgroup with vascular features demonstrated excellent predictive performance (C-index > 0.77), indicating that each component provides valuable prognostic information for risk assessment. Notably, the model combining Kerr scores with vascular features was particularly effective, achieving a C-index of 0.85 (HR = 1.89, 95% CI: 1.20–2.96; p = 0.0057), demonstrating its robust value as a core clinical indicator. The corresponding Kaplan-Meier curves (e.g.) are shown in Table 2. Figure 5 The time-dependent ROC curve analysis further validated the effectiveness of these combined models in risk stratification of severe ischemic events in the nervous system (e.g., Figure 6 ).

[0063] External validation results: On an independent external validation set (n=53), the model demonstrated good generalization ability, with a C-index of 0.79 and an HR of 1.86 (95% CI: 1.13–3.06, p=0.016), proving its applicability in different medical centers, such as... Figure 7 As shown.

[0064] Table 1 Table 2 The application value of the system is demonstrated through three typical cases, such as... Figure 8 As shown.

[0065] Case A (internal cohort, high risk): The patient had a history of SIEs (stenotic vascular endothelial diseases), high clinical activity (Kerr=3), and MRA showed multi-vessel stenosis. The system integrated multimodal features such as significant vascular wall enhancement and extremely tortuous vascular morphology, predicting the patient as high-risk (risk score). >Threshold). The patient experienced an adverse event 3 months after the assessment.

[0066] Case B (internal cohort, low risk): The patient's clinical characteristics were similar to Case A, but MRA showed milder changes in vessel wall features and overall morphology. The system predicted this as low risk based on subtle imaging differences. <Threshold). No adverse events occurred in the patient during the subsequent 39-month follow-up.

[0067] Case C (External Cohort, High Risk): The patient, from an external center, presented with severe stenosis to occlusion of the left common carotid artery. The system accurately extracted features and classified the patient as high-risk even on unseen data. The patient experienced an adverse event 7 months later.

[0068] The beneficial effects of this embodiment include: (1) Multimodal deep fusion significantly improves prediction accuracy: This invention innovatively integrates three-dimensional vascular morphological features (extracted through point cloud neural networks), vascular wall image features (extracted through 3D-CNN), and clinical semantic features (encoded through a basic model) in TAK prognosis prediction for the first time. This collaborative modeling mechanism can comprehensively capture multi-dimensional information about the disease and overcome the limitations of modeling from a single data source. Experiments show that the C-index of the fusion model reaches 0.86, which is more than 20% higher than that of the single feature model (maximum 0.71). It also maintains an excellent performance of 0.79 in independent external validation, which is significantly better than existing clinical prediction methods.

[0069] (2) Automated feature extraction reduces human bias: The system realizes fully automated processing from raw DICOM data to risk scoring: Automatic image feature extraction: No manual drawing or measurement is required. The system directly learns the deep features of vascular morphology and vascular wall from MRA data through deep learning networks; Intelligent encoding of clinical information: Heterogeneous clinical data (text, numerical, categorical variables) are uniformly transformed into structured feature vectors; End-to-end training optimization: The entire system can be jointly trained to optimize the synergy between feature extraction and prediction. This feature greatly reduces the dependence of traditional methods on physician experience and improves the objectivity and repeatability of assessment results.

[0070] (3) Three-dimensional geometric modeling capability to comprehensively characterize vascular lesions: For complex tubular structures such as blood vessels, this invention uses a point cloud neural network (PointNet++) to specifically process the three-dimensional morphology of blood vessels: Local detail capture: Through hierarchical sampling and local feature aggregation, it accurately depicts local geometric changes such as stenosis, dilation, and tortuosity; Global topological understanding: It learns the overall spatial relationship of blood vessel branches and directions of travel; Scale adaptability: Multi-scale feature fusion mechanism, which pays attention to both subtle lesions and macroscopic morphological changes; This three-dimensional geometric modeling capability far exceeds traditional two-dimensional image analysis or simple three-dimensional measurement, providing a new paradigm for the quantitative analysis of vascular diseases.

[0071] (4) Discrete-time survival analysis framework for handling right-censored data: In view of the characteristics of clinical follow-up data, this invention designs a special processing mechanism: time discretization: dividing continuous survival time into clinically meaningful intervals and transforming it into an ordered classification problem; censored data processing: correctly processing right-censored data through the maximum likelihood estimation method; time-varying risk assessment: outputting the risk probability at different time points, rather than a single static score; This framework maintains the rigor of survival analysis and adapts to the optimization characteristics of deep learning models. The C-index of 0.86 proves its effectiveness.

[0072] Example 2 This embodiment provides a multimodal deep learning-based system for predicting severe ischemic events in the nervous system associated with arteritis, used to implement the method described in Embodiment 1. The system includes: The image data acquisition and preprocessing module is used to acquire and preprocess magnetic resonance angiography data; The 3D morphological feature extraction module for blood vessels is used to extract 3D morphological feature vectors of blood vessels based on point cloud neural networks. The blood vessel wall feature extraction module is used to extract blood vessel wall feature vectors based on a three-dimensional convolutional neural network. The clinical information feature encoding module is used to encode clinical information based on a pre-trained basic model and generate clinical feature vectors. A multimodal feature fusion module is used to fuse the three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector to generate a multimodal fused feature representation; The risk prediction and stratification module is used to output individualized risk scores and perform risk stratification based on the multimodal fusion feature representation and discrete-time survival analysis model.

[0073] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting severe ischemic events in the nervous system caused by large vessel arteritis based on multimodal deep learning, characterized in that, Includes the following steps: To obtain magnetic resonance angiography data and clinical information from patients with Takayasu arteritis; The magnetic resonance angiography data is preprocessed to obtain standardized three-dimensional vascular images; Based on point cloud neural networks, the point cloud of the blood vessel surface obtained by reconstruction from standardized three-dimensional blood vessel images is processed to extract the three-dimensional morphological feature vector of the blood vessel. Based on a three-dimensional convolutional neural network, the standardized three-dimensional vascular image is processed to extract the vascular wall feature vector; Based on the pre-trained basic model, the clinical information is encoded to generate a clinical feature vector; The three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector are fused to generate a multimodal fusion feature representation; Based on the multimodal fusion feature representation, an individualized risk score for severe neurological ischemic events in patients with aortitis is output through a discrete-time survival analysis model.

2. The method according to claim 1, characterized in that, The steps for preprocessing magnetic resonance angiography data include: Spatial resampling is performed on the magnetic resonance angiography data to unify the spatial resolution; Intensity normalization is performed on the resampled data; The region of interest containing the target blood vessel is cropped from the normalized data to obtain the standardized three-dimensional blood vessel image.

3. The method according to claim 1, characterized in that, The steps for extracting the three-dimensional morphological feature vector of blood vessels include: Layered sampling and local feature aggregation are performed on the point cloud of the blood vessel surface; A global feature vector representing the overall three-dimensional spatial geometry of the blood vessel is generated through a global feature extraction layer in a point cloud neural network, which serves as the three-dimensional morphological feature vector of the blood vessel.

4. The method according to claim 1, characterized in that, The specific steps for extracting the feature vector of the blood vessel wall are as follows: Standardized 3D vascular images are input into a 3D residual convolutional neural network; After multiple convolutions and nonlinear mappings in the three-dimensional residual convolutional neural network, a global pooling operation is performed on the final output feature map to generate the blood vessel wall feature vector.

5. The method according to claim 1, characterized in that, The steps for generating clinical feature vectors include: Clinical information, including symptom descriptions, laboratory test results, and disease activity scores, is integrated into a text description; The text description is encoded using a pre-trained base model text encoder, which outputs a high-dimensional semantic vector as the clinical feature vector.

6. The method according to claim 1, characterized in that, The steps for generating multimodal fusion feature representations include: The three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector are concatenated to obtain the concatenated feature vector; The concatenated feature vector is input into a feature mapping unit containing a fully connected layer and a nonlinear activation function for dimensionality reduction and fusion, and the multimodal fusion feature representation is output.

7. The method according to claim 1, characterized in that, The construction and execution of discrete-time survival analysis models include: The continuous follow-up time is discretized into multiple ordered time intervals; Construct a prediction network whose input is the multimodal fusion feature representation and whose output is the predicted risk probability of the patient experiencing a severe neurological ischemic event in each time interval. The individualized risk score is calculated based on the predicted risk probability.

8. The method according to claim 7, characterized in that, The specific steps for calculating an individualized risk score are as follows: The predicted risk probabilities within each of the aforementioned time intervals are weighted and summed, with the weights corresponding to the importance of each time interval.

9. The method according to claim 1, characterized in that, The method also includes a risk stratification step: Based on a preset risk threshold, the individualized risk score is divided into a high-risk level or a low-risk level.

10. A system for predicting severe ischemic events in the nervous system caused by large vessel arteritis based on multimodal deep learning, characterized in that, The system for implementing the method as described in any one of claims 1 to 9 comprises: The image data acquisition and preprocessing module is used to acquire and preprocess magnetic resonance angiography data; The 3D morphological feature extraction module for blood vessels is used to extract 3D morphological feature vectors of blood vessels based on point cloud neural networks. The blood vessel wall feature extraction module is used to extract blood vessel wall feature vectors based on a three-dimensional convolutional neural network. The clinical information feature encoding module is used to encode clinical information based on a pre-trained basic model and generate clinical feature vectors. A multimodal feature fusion module is used to fuse the three-dimensional morphological feature vector of the blood vessel, the feature vector of the blood vessel wall, and the clinical feature vector to generate a multimodal fused feature representation; The risk prediction and stratification module is used to output individualized risk scores and perform risk stratification based on the multimodal fusion feature representation and discrete-time survival analysis model.

Images