Cerebral stroke postoperative vascular recanalization effect prediction method and system based on multiple modes

Abstract

The invention relates to the technical field of intelligent disease prediction, in particular to a cerebral apoplexy postoperative vascular recanalization effect prediction method and system based on multiple modalities, and the method comprises the steps: obtaining clinical information and NCCT images of a patient; preprocessing the clinical information to obtain clinical modal features; preprocessing the NCCT image to obtain a denoised NCCT image; inputting the denoised NCCT image into a deep convolutional neural network to obtain modal features of the NCCT image; performing comparative learning according to the clinical modal features and the NCCT image modal features, and constructing a comparative loss function to obtain comparative learning loss; fusing the NCCT image modal features and the clinical modal features to obtain fused modal features, and calculating the loss of the fused modal features; and calculating dynamic total loss, and outputting a prediction result of vascular recanalization after thrombus extraction of the acute ischemic stroke patient. According to the method, accurate prediction of the effectiveness of postoperative vascular recanalization is realized through mechanisms such as deep neural network feature extraction, multi-head cross attention and comparative learning.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent disease prediction, and in particular to a multimodal method and system for predicting the effect of vascular recanalization after stroke surgery. Background Art

[0002] Acute ischemic stroke is caused by acute occlusion of cerebral blood vessels, resulting in ischemic damage to brain tissue. It is also the stroke subtype with the highest rates of disability and mortality. In clinical practice, even after effective interventional treatments such as thrombectomy, approximately 60% of patients with this type of acute ischemic stroke fail to achieve satisfactory recovery. In clinical practice, effective recanalization is considered effective if the patient achieves good vascular recanalization after surgery, while ineffective recanalization is considered ineffective. Non-contrast computed tomography (NCCT) utilizes an X-ray tube to scan slices around the body, measuring X-ray absorption data at each point. High-speed computer processing and image reconstruction techniques are then used to display the scanned slices. As a widely used advanced neuroimaging technique, NCCT can provide certain clinical information. However, determining effective recanalization after thrombectomy in patients with acute ischemic stroke is time-consuming and labor-intensive, and is subject to subjective variability among physicians. Clinical features can, to a certain extent, reflect various patient indicators, but they do not fully reflect the patient's condition. Data fusion between two modalities can effectively address these missing information. In recent years, deep learning and multimodal technologies have developed rapidly in the medical field and have been widely used in disease diagnosis, prognosis and treatment planning, and have important application value in the management of ischemic stroke. By integrating multi-source information such as imaging, clinical, and even genetic information, the model can capture deep features related to the recovery effect after thrombectomy from multiple angles, achieving more accurate automated diagnosis. Therefore, the present invention is based on NCCT fusion clinical feature training deep learning model, which can be effectively used for the prediction of recanalization of acute ischemic stroke, assist clinical diagnosis, provide relatively objective diagnostic results, and then formulate more personalized treatment plans for patients, and provide effective decision-making for clinicians.

[0003] Traditional prognostic assessment options include the following:

[0004] 1. Doctors directly interpret and judge images through experience, which is highly subjective and easily affected by individual differences.

[0005] 2. By generating two separate images, a perfusion color map and a blood volume color map, from imaging results, these images are used to determine whether vascular perfusion and blood volume are within threshold limits. This approach, combined with RAPID software, guides physician decision-making. This solution is costly and difficult to implement in resource-constrained settings, such as primary care hospitals.

[0006] 3. Directly predicting the prognosis after thrombectomy through a single modality of imaging or clinical features is difficult to fully capture the spatiotemporal heterogeneity and contextual relationships of the lesions.

[0007] For example, the Chinese invention patent with announcement number CN113724876A discloses a stroke in-hospital complication prediction model based on multimodal fusion and DFS-LLE algorithm. The model construction method includes the following steps: S1. Collecting the medical treatment data of stroke patients during hospitalization before the occurrence of complications. The medical treatment data includes two modalities: clinical structured data and medical image data; S2. Preprocessing the data of the two modalities separately; S3. Extracting imaging genomics features from medical images and performing multimodal fusion with clinical structured data; S4. Using DFS technology to synthesize the features fused in step S3; S5. Using LLE to reduce the dimension of the features synthesized in step S4; S6. Constructing a stroke complication prediction model based on a deep neural network based on the above data.

[0008] Separately, James P. Diprose et al. collected clinical data and preoperative NCCT and computed tomography angiography (CTA) imaging data from 975 patients undergoing EVT for acute ischemic stroke. They first tokenized the DICOM files, converted them to NiFTI format, and resampled them to ensure consistent voxel size. NCCT and CTA images were preprocessed separately to create brain masks and perform image registration, followed by data augmentation on the NCCT images. The clinical data were interpolated for missing values and normalized for continuous features using the scikit-learn pipeline. For imaging modalities, the DenseNet121 model architecture was used to train single-channel (NCCT) and dual-channel (NCCT + CTA) models. For clinical feature modalities, a network consisting of three fully connected layers was used. After processing, the data was input into a fully connected layer for classification and prognostic results. However, the clinical information feature extraction module proposed by the institute only extracts features through a three-layer fully connected network, without considering the importance of features and their relevance to the final task. It is unable to fully explore the complex relationships and potential patterns in clinical data. In addition, the fully connected network has limited ability to capture nonlinear relationships and is largely unable to extract high-order features with rich semantics, thus affecting prediction performance. In multimodal fusion, this study directly stacked the features of the two modalities through simple splicing. It failed to consider the differences in distribution, dimension, and semantics between NCCT images and clinical features, which easily led to information redundancy or imbalance. NCCT images are more high-dimensional, allowing high-dimensional image features to dominate the fusion results, while the role of clinical information is weakened. On the other hand, simple stacking cannot fully model and calibrate the complementarity between NCCT and patient clinical information. The quality of the fused feature representation is low, and there is a lack of more sophisticated feature alignment and interaction strategies. As a result, the model cannot fully utilize the complementary information between the modalities during joint learning, thus affecting the final prediction accuracy.

[0009] Yongkai Liu et al. first preprocessed stroke patient data. NCCT images were first standardized using a template, and images were constrained to the range of 0–100 HU, followed by mean normalization. Clinical variables were converted to binary variables (0 or 1), with missing clinical variables imputed using the median of the dataset. During model construction, the imaging modality was modified based on the ResNet architecture of a three-dimensional convolutional neural network. The fourth layer was removed, a dropout layer was added to reduce overfitting, and the number of neurons in the final fully connected layer was reduced for regression analysis. The model accepted NCCT images as input and output a continuous prediction of the 90-day mRS score. In the clinical information modality, a support vector machine regression model was used to predict the 90-day mRS score using age, sex, baseline and 24-hour NIHSS scores, and a history of hypertension, diabetes, and atrial fibrillation as variables. In the multimodal fusion approach employed, clinical model variables and imaging model predictions were directly input into the support vector regression framework. Predicted values exceeding 6 or falling below 0 were reassigned to 6 and 0, respectively. The fusion model accepts clinical variables and imaging model predictions as input and outputs continuous predictions of 90-day mRS scores. However, the clinical information feature extraction module proposed by the institute uses relatively few clinical variables, which cannot fully reflect the patient's clinical condition. Compared with the multiple indicators that may be involved in actual clinical practice, the information is relatively limited. Only a support vector machine regression model is used to process clinical data. Although SVM performs well under small sample conditions, its simple model structure makes it difficult to capture complex nonlinear relationships and interactions between variables. The lack of modules for feature engineering or deep feature extraction of clinical data may result in some implicit information not being fully mined, which in turn hinders the improvement of overall prediction accuracy. For imaging feature extraction, the model is based on a modified ResNet architecture based on a three-dimensional convolutional neural network. The removal of the fourth layer makes the network insufficiently deep, resulting in inadequate capture of high-level, fine-grained spatial features. The lack of an interaction mechanism between the two modalities makes it impossible to distinguish feature similarity between the two modalities and fails to reduce the model's sensitivity to feature redundancy and noise. Summary of the Invention

[0010] The purpose of the present invention is to overcome the significant limitations of existing technologies in clinical information feature extraction and multimodal fusion: on the one hand, clinical feature extraction modules generally adopt shallow networks (such as three-layer fully connected) or simple models (such as SVM), which fail to fully explore the complex nonlinear relationships and high-order semantic information between features, and have problems such as insufficient coverage of clinical variables and lack of feature importance assessment mechanism; on the other hand, multimodal fusion often adopts a coarse-grained splicing strategy, ignoring the differences in dimensional distribution and semantic level between NCCT images and clinical data, resulting in insufficient information interaction between modalities and lack of feature calibration. At the same time, the image feature dominance phenomenon causes the weakening of clinical information. In addition, the insufficient network depth leads to limited capture of fine-grained spatial features, and the lack of a contrastive learning mechanism exacerbates feature redundancy and noise interference. These defects jointly restrict the model's utilization of the complementary advantages of multimodal data and the improvement of prediction performance. To this end, a multimodal-based method and system for predicting the effect of vascular recanalization after stroke surgery is provided.

[0011] In a first aspect, the present invention provides a method for predicting the effect of vascular recanalization after stroke surgery based on multimodality, comprising the following steps:

[0012] Obtain the patient's clinical information characteristics and NCCT images;

[0013] Preprocessing the clinical information features, calculating the importance index of each preprocessed clinical information feature and ranking them to obtain clinical modality features;

[0014] Preprocessing the NCCT image to obtain a denoised NCCT image, and inputting the denoised NCCT image into a deep convolutional neural network to obtain NCCT image modality features;

[0015] Performing comparative learning on the clinical modality features and the NCCT imaging modality features and calculating the NCCT imaging modality feature loss, the clinical modality feature loss, and the comparative learning loss;

[0016] Obtaining a fused modality feature by fusing the NCCT image modality feature and the clinical modality feature and calculating the fused modality feature loss;

[0017] The dynamic total loss is calculated based on the NCCT image modality feature loss, clinical modality feature loss, fusion modality feature loss, and contrastive learning loss, and the dynamic total loss is input into the deep convolutional neural network until the deep convolutional neural network converges. After convergence, the deep convolutional neural network predicts the newly input clinical information features and NCCT images, and outputs the prediction results of vascular recanalization after thrombectomy.

[0018] Preferably, the specific steps of preprocessing the clinical information features include:

[0019] Set ranges for continuous variables in clinical information, identify them through binning and boxplots, and perform corrections and delete outliers;

[0020] Fields with missing values greater than the preset missing value threshold are discarded, and the remaining categorical fields with missing values are filled with the mode;

[0021] One-hot encoding was used for multi-class unordered variables in clinical information, and PCA dimensionality reduction was performed on highly correlated features.

[0022] Preferably, the specific steps of obtaining clinical modality features include:

[0023] The importance features of each clinical information feature are calculated and sorted, structural features are screened out, and the structural features are concatenated to form a character string, which is converted into a text that conforms to the natural language format, and clinical modality features are extracted through a pre-trained model.

[0024] Preferably, the specific steps of obtaining NCCT imaging modality features include:

[0025] Extracting local features of the denoised NCCT image;

[0026] Normalizing the activation value of each channel of the denoised NCCT image;

[0027] The standardized denoised NCCT images are subjected to channel compression and deep features are extracted to obtain multi-scale features.

[0028] Further extracting and integrating local features of the multi-scale features;

[0029] The feature map of the integrated multi-scale features is reduced in dimensionality, and high-order semantic features are further extracted after filtering out noise, so that the network can capture deeper image features;

[0030] The size of the extracted multi-scale feature map is reduced, and the three-dimensional features of the multi-scale feature map are converted into a one-dimensional vector to obtain the NCCT image modality features.

[0031] Preferably, the specific steps of performing comparative learning based on the clinical modality features and the NCCT imaging modality features include:

[0032] A group of samples is selected, and the clinical modality features and NCCT imaging modality features of the same patient in the samples are regarded as positive sample pairs, and the clinical modality features and NCCT imaging modality features of different patients in the samples are regarded as negative sample pairs;

[0033] Comparative learning is performed on all positive and negative sample pairs in a set of samples to capture the similarities between different modality features of the same patient and the differences between modality features of different patients.

[0034] Preferably, the step of calculating the fusion modality feature includes:

[0035] Taking clinical modality features as Q and NCCT imaging modality features as K and V, the cross-headed multi-head attention mechanism is used to screen out the NCCT imaging modality features that are most relevant to the clinical modality features, and multiple attention heads are used to capture the correlation of cross-modality information;

[0036] The NCCT image modality features and clinical table modality features are weightedly fused by calculating the cross-attention score to obtain the fused modality features.

[0037] In a second aspect, the present invention provides a multimodal post-stroke vascular recanalization effect prediction system, comprising a clinical information preprocessing module, a clinical feature extraction module, a 3D image preprocessing module, a 3D image feature extraction module, a contrast learning and feature alignment module, a multimodal feature fusion module, and a prediction module.

[0038] The clinical information preprocessing module is used to preprocess clinical information features;

[0039] The clinical feature extraction module is used to calculate the importance index of each preprocessed clinical information feature and sort them to obtain clinical modality features;

[0040] The 3D image preprocessing module is used to preprocess the NCCT image to obtain a denoised NCCT image;

[0041] The 3D image feature extraction module extracts NCCT image modality features based on a deep convolutional neural network;

[0042] The contrastive learning and feature alignment module is used to perform contrastive learning based on the clinical modality features and the NCCT image modality features and calculate the NCCT image modality feature loss, the clinical modality feature loss and the contrastive learning loss;

[0043] The multimodal feature fusion module is used to fuse the NCCT image modality features and the clinical modality features to obtain fused modality features and calculate the fused modality feature loss;

[0044] The prediction module is used to predict the newly input clinical information features and NCCT images, and output the prediction results of the patient's vascular recanalization after thrombectomy.

[0045] Preferably, the deep convolutional neural network includes a 3D convolution layer, a normalization layer, a first dense block, a 3D convolution layer, a first pooling layer, a second dense block, a second pooling layer and a fully connected layer;

[0046] The 3D convolution layer is used to extract local features of the denoised NCCT image and simultaneously learn local contrast and edge information in the denoised NCCT image;

[0047] The normalization layer is used to standardize the activation value of each channel of the denoised NCCT image;

[0048] The first dense block is used to compress the number of channels of the standardized denoised NCCT image and extract deep features to obtain multi-scale features;

[0049] The 3D convolution layer is used to further extract and integrate local features of the multi-scale features;

[0050] The first pooling layer is used to reduce the dimension of the feature map of the integrated multi-scale features and filter out noise before inputting it into the second dense block;

[0051] The second dense block is used to further extract high-order semantic features, so that the network can capture deeper image features, and the extracted multi-scale features are input into the second pooling layer;

[0052] The second pooling layer is used to reduce the size of the feature map;

[0053] The fully connected layer is used to convert three-dimensional features into one-dimensional vectors.

[0054] Preferably, the clinical feature extraction module uses XGBoost to calculate the importance index; uses the text encoder of the CLIP pre-trained model to extract clinical modality features; and uses an upsampling layer to enhance the resolution of the clinical modality features;

[0055] In the process of preprocessing the NCCT image to obtain the denoised NCCT image by the 3D image preprocessing module, brain region segmentation is performed using the U-Net pre-trained deep learning model to remove non-brain tissue information;

[0056] The contrastive learning loss is calculated using a contrastive loss function, and the NCCT imaging modality feature loss, clinical modality feature loss, and fusion modality feature loss are calculated using binary cross entropy.

[0057] Preferably, the prediction module is composed of a deep learning network, which includes an image modality branch, a clinical information modality branch and a fusion modality branch. The image modality branch, the clinical information modality branch and the fusion modality branch are all composed of three linear layers, a ReLU activation function and a Softmax classification layer.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. This invention provides a multimodal method and system for predicting postoperative vascular recanalization efficacy after stroke surgery. This method uses a convolutional neural network to extract features from images. It also extracts features from clinical information based on traditional machine learning methods and pre-trained large models. It innovatively employs a contrastive learning approach to integrate imaging modalities with clinical information modalities, enhancing the model's ability to distinguish features and extracting both NCCT imaging features and structured clinical feature information.

[0060] 2. The present invention provides a multimodal method and system for predicting the effect of vascular recanalization after stroke surgery. By adopting a cross-multi-head attention mechanism for fusion, high-dimensional image features are matched with low-dimensional clinical indicators. The image modality loss and clinical table modality loss are weighted separately and added with the fusion loss to form a total loss, which is then used for back propagation, thereby solving the modality bias problem of images and clinical indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of a method for predicting the effect of vascular recanalization after stroke surgery based on multimodality according to Example 1 of the present invention;

[0062] Figure 2 This is a simplified flowchart of the method for predicting the effect of vascular recanalization after stroke surgery based on multimodality according to Example 1 of the present invention;

[0063] Figure 3 Schematic diagram of the neural network structure of Example 1 of the present invention;

[0064] Figure 4 This is a medical treatment logic diagram of Example 1 of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further described in detail below with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments, as all technologies implemented based on the present invention fall within the scope of the present invention.

[0066] In addition, the expressions “first”, “second”, “third”, etc. in the terms are merely used to distinguish the description of the same or similar components, and should not be understood as emphasizing or implying the relative importance of specific components.

[0067] In addition, in the description of the embodiments of the present invention, "several," "plurality," and "a number" represent at least two. It can also be any number such as two, three, four, five, six, seven, eight, nine, or even more than nine.

[0068] Example 1

[0069] A multimodal method and system for predicting the effect of vascular recanalization after stroke surgery. The method flow chart is as follows: Figure 1 As shown, the following steps are included:

[0070] Obtain the patient's clinical information characteristics and NCCT images;

[0071] Preprocessing the clinical information features, calculating the importance index of each preprocessed clinical information feature and ranking them to obtain clinical modality features;

[0072] Preprocessing the NCCT image to obtain a denoised NCCT image, and inputting the denoised NCCT image into a deep convolutional neural network to obtain NCCT image modality features;

[0073] Performing comparative learning on the clinical modality features and the NCCT imaging modality features and calculating the NCCT imaging modality feature loss, the clinical modality feature loss, and the comparative learning loss;

[0074] Obtaining a fused modality feature by fusing the NCCT image modality feature and the clinical modality feature and calculating the fused modality feature loss;

[0075] The dynamic total loss is calculated based on the NCCT image modality feature loss, clinical modality feature loss, fusion modality feature loss, and contrastive learning loss, and the dynamic total loss is input into the deep convolutional neural network until the deep convolutional neural network converges. After convergence, the deep convolutional neural network predicts the newly input clinical information features and NCCT images, and outputs the prediction results of vascular recanalization after thrombectomy.

[0076] As a specific embodiment, the patient's clinical information characteristics and NCCT images, as well as the corresponding label data, are obtained, such as Figure 2 As shown, the first step is data preprocessing, first of all, clinical information feature preprocessing:

[0077] By observing and visually analyzing the characteristics of clinical tables, we first deleted completely duplicate records by hospitalization number to ensure data uniqueness. We set reasonable ranges for indicators such as age, blood pressure, and biochemistry, and used binning and boxplots to identify, correct, and delete outliers. For gender and anesthesia method, we ensured coding consistency and deleted illegal values.

[0078] Preferably, if gender only contains 0 / 1, delete other numbers and characters. Delete irrelevant fields such as name. Discard fields with a missing rate greater than 50%, and fill in the median for other fields with missing values, including ALT and creatinine, and the mode for categorical fields such as atrial fibrillation and anesthesia. Binary categorical variables such as intravenous thrombolysis and stent implantation are directly retained as 0 / 1 coding, and multi-category unordered variables such as responsible vessels and infarction types are encoded using one-hot encoding. Standardize model-sensitive continuous variables such as age and blood pressure, normalize fields that need to limit the range, construct the pulse pressure difference by subtracting diastolic pressure from systolic pressure, and classify the time from admission to imaging into "urgent treatment" or "non-urgent treatment". Perform PCA dimensionality reduction on highly correlated features such as LDL-C and TC.

[0079] Next, NCCT images were preprocessed. Patients with repeated brain scan sequences were screened and the highest-quality NCCT was selected, ensuring that only one complete set of brain NCCT images was included for each patient. A U-Net pretrained deep learning model was used for precise segmentation, skull stripping, and removal of non-brain tissue interference, allowing subsequent feature extraction to focus more on the brain. An image resampling algorithm was used to resample all NCCT images to a uniform voxel size of 1 × 1 × 1 mm. The HU values of the NCCT images were truncated and normalized to facilitate convergence during subsequent model training. The normalized images were binarized, and a fixed threshold (HU greater than a preset value) was used to generate a binary image. A precise mask was generated based on this contour, retaining only regions with HU values above the threshold. Irrelevant background and noise were removed, retaining only the region of interest. A fixed cropping of the image was performed to a uniform width and height of 128 × 128, respectively. Because the number of slices in each patient's NCCT images varied, interpolation was used to normalize the slices to the median. Finally, the cropped image data was converted into a numpy array using the numpy library for input into the deep learning model training.

[0080] like Figure 3 As shown, the second step is feature extraction. As a specific embodiment, clinical information feature extraction is first performed:

[0081] The clinical information features obtained from the preprocessing are input into the clinical feature extraction module. Unlike imaging data, a suitable method is required for feature extraction. In order to better capture the relationship between patients' physiological indicators, the present invention adopts a network structure that combines XGBoost feature selection, CLIP pre-training model and upsampling layer to fully mine the potential information of clinical data and map it to a high-dimensional feature space for prediction. First, since not all variables are equally important to the final prediction task, the present invention first uses XGBoost (eXtreme Gradient Boosting) to calculate the importance of each feature and screen the most critical variables. The calculation formula for importance is expressed as:

[0082]

[0083] in, Representative feature f i The sum of the split gains over all decision trees is used as an indicator of feature importance. TR is the total number of decision trees trained by XGBoost; S t is the total number of split nodes in the t-th tree; is the feature f i The gain brought by the s-th split node in the t-th tree. Calculate the gain of all features And sort them, select the top-ranked features and input them into the subsequent deep learning model to improve the prediction performance and reduce redundant information interference.

[0084] The most critical features screened out by the present invention include the NIHSS score for measuring the severity of stroke and the ASPECT score for assessing the scope of cerebral infarction. In view of the fact that some features of the clinical information used contain multiple values, the present invention innovatively concatenates the structural features that have passed the screening into a character string and converts it into a text that conforms to the natural language format. For example, the table structure NIHSS score of 7, age of 37, ASPECT score of 8, and systolic blood pressure of 10 is converted into the text "NIHSS score {7}, age {37}, ASPECT score {8}, systolic blood pressure {10}". The text after format conversion is input into the text encoder of the CLIP (Contrastive Language-Image Pretraining) pre-training model. The text in the natural semantic format allows CLIP to fully understand the actual meaning of the numerical value. Compared with directly using numerical data, the method of extracting text features using CLIP can better learn the complex relationship between these medical terms and the prediction target. The CLIP pre-training model can better model the intrinsic relationship between these features through the experience of large-scale text-image joint training, and the extracted clinical feature vector z clinical , the formula is:

[0085]

[0086] Where T is the input model feature, and D is the text embedding dimension of the CLIP pre-trained model. This vector contains deep semantic information of the text, such as the correlation between NIHSS score and ASPECT score, and how they jointly affect vascular recanalization. T It is a deep neural network based on Transformer, which converts the input text into a high-dimensional vector. represents a D-dimensional real vector space.

[0087] The extracted clinical features require further feature enhancement to enhance their representational capabilities. To this end, the present invention employs an upsampling layer to increase the resolution of the features, enabling them to have better representational capabilities in high-dimensional space.

[0088] Next, 3D image feature extraction is performed. The denoised NCCT images obtained after preprocessing are fed into an image convolution extraction module with a size of 128×128×28 (width, height, number of slices). This module extracts deep features from the patient images to predict the outcome of vascular recanalization after thrombectomy in patients with acute ischemic stroke. The entire network architecture consists of a 3D convolutional layer, a normalization layer, a dense block 1, a 3D convolutional layer, a pooling layer, a dense block 2, a pooling layer, and finally a fully connected layer, which maps the output to a one-dimensional vector. The first layer of the network is a 3D convolutional layer. It uses a 3×3×3 convolutional kernel, sliding in three-dimensional space with a stride of 1, using zero padding to maintain the original size, and outputs 64 feature channels. The primary function of this first 3D convolutional layer is to extract local spatial information, including structural features of brain tissue and blood vessels. It also learns local contrast and edge information in NCCT images, which are crucial for vascular changes after thrombectomy. After the convolutional layer, the normalization layer normalizes the activation values of each channel. By subtracting the mean and dividing by the standard deviation, the data distribution becomes more stable, helping to reduce internal covariate shift and accelerating network convergence. The normalized NCCT image features enter the first dense block. A dense block consists of multiple small convolutional units, each of which extracts new features from the outputs of all previous layers. Its dense connections enable more efficient information transfer, improve gradient flow, and enable deeper networks without vanishing gradients. The basic structure of each DenseBlock layer consists of batch normalization, a ReLU activation function, a 1×1×1 convolution kernel for dimensionality reduction, and a 3×3×3 convolution kernel for extracting deeper features. In this DenseBlock, the growth rate is 16, and there are three layers, increasing the number of output channels. After processing the DenseBlock, the data enters a 3D convolutional layer with similar parameters to the previous convolutional layer. This layer continues to extract deeper local features while integrating the information output by the DenseBlock. Since the same zero-padding strategy is used, the size of the output feature map remains unchanged. Subsequently, the feature map enters a 3D maximum pooling layer, which uses a pooling kernel of size 2×2×2 and a stride of 2 to reduce the dimension of the feature map, reduce the amount of calculation, and extract the main information of the image while filtering out some local noise. After the pooling operation, the size of the feature map is reduced. After the pooling is completed, the data enters the second DenseBlock, which has the same structure as the first DenseBlock. It still uses multiple densely connected 3D convolutional layers with a growth rate of 16. After 3 layers, the number of channels increases again. This stage further extracts high-order semantic features, allowing the network to capture deeper image features, such as changes in vascular morphology and texture information of ischemic areas. These features are crucial for predicting vascular recanalization.At this point, the deep interaction of NCCT features has been completed. The data then passes through a 3D max pooling layer with a kernel size of 2×2×2 and a stride of 2, further reducing the size of the feature map. This not only reduces computational complexity but also enables the network to integrate feature information at a higher spatial scale. Finally, the network enters a fully connected layer, flattening the 3D features into a one-dimensional vector.

[0089] Efficient feature preprocessing and selection in the first and second steps: 3D NCCT images and clinical table data are efficiently preprocessed using U-Net pre-trained model segmentation, feature normalization, PCA dimensionality reduction, and XGBoost screening to ensure the quality and validity of input data, thereby improving the stability of model training and prediction accuracy.

[0090] The present invention achieves efficient and accurate preprocessing of NCCT image data and clinical table data, ensuring data quality consistency and information integrity, and improving the stability and predictive ability of the model. In clinical information processing, redundant data is removed, missing values are filled, continuous variables are standardized, and methods such as one-hot encoding and PCA dimensionality reduction are used to optimize the data structure, enabling the model to more effectively learn key features. In image processing, U-Net is used for precise brain tissue segmentation, voxel size is resampled to standardize, and regions of interest are extracted using HU value truncation and binarization to ensure consistency and high-quality input of image features. This series of optimizations significantly reduces the interference of noisy data, improves data availability, enables deep learning models to converge more quickly, improves prediction accuracy, and provides more reliable support for clinical decision-making. This solution adopts a series of optimization strategies for feature extraction, enabling the model to efficiently and accurately extract key information, thereby improving predictive performance and generalization ability. In terms of 3D NCCT image feature extraction, convolutional layers are combined with DenseBlock structures to extract local and global features layer by layer, and batch normalization and pooling operations are used to enhance the stability and robustness of feature expression. In terms of clinical information feature extraction, key variables are first screened through XGBoost sorting to reduce the interference of redundant information. Subsequently, the structured clinical data is converted into text format and input into the CLIP pre-trained model for feature encoding. This allows the tabular data to be integrated into the deep learning model in the form of high-dimensional semantic information, fully leveraging the powerful representation capabilities of CLIP in multimodal learning. In terms of feature interaction, comparative learning is used to effectively align the feature distributions of image and clinical table data, enhance cross-modal information correlation, improve the model's discriminative ability and generalization performance, and thus optimize the prediction accuracy of postoperative vascular recanalization effects in patients with acute ischemic stroke. This series of optimization measures greatly enhances the model's adaptability to complex data, improves prediction accuracy, and enhances the model's robustness to data uncertainty, providing a more reliable auxiliary decision-making tool for predicting postoperative vascular recanalization effects in patients with acute ischemic stroke.

[0091] The third step is comparative learning and feature fusion. First, comparative learning is performed. Both imaging modalities and clinical modalities describe the patient's condition, but the information sources of the two are different. Therefore, in the task of predicting the effect of vascular recanalization, the distance between each sample in the feature space is different. It is necessary to make the imaging features and clinical features closer in similar situations and farther apart in different situations.

[0092] Therefore, the present invention adopts a contrastive learning method for feature alignment, and regards the imaging features and clinical features of the same patient as positive sample pairs, so that the model can learn the correlation between different modal features of the same patient. The imaging features and clinical features of different patients are regarded as negative sample pairs, so that the model can distinguish the feature differences between different patients. By contrasting and learning positive and negative sample pairs within a batch, the similarities between different modal features of the same patient and the differences between the features of different patients are captured. The contrastive loss function is used to calculate the loss during the training process, so that the image and clinical information of the same patient are closer in the feature space, while the feature vectors of different patients are far away from each other. The contrastive loss function used in model training is as follows:

[0093]

[0094] Among them, z image，i is the i-th imaging modality feature in the same batch, z clinical，j is the jth clinical information sample in the same batch, where i, j = 1, 2, ..., N represents all samples in a batch, N is the batch size, sim(z image , z clinical ) is the cosine similarity, which is used to measure the similarity between the image vector and the clinical information vector. The formula is expressed as:

[0095]

[0096] Where τ is the temperature coefficient, which is used to adjust the smoothness of the similarity distribution.

[0097] Secondly, feature fusion is performed. After obtaining the features of NCCT imaging modality and clinical table modality, the information of these two modalities needs to be fused to fully utilize the complementarity of imaging and clinical data and improve the prediction performance of the model. Since imaging data can provide intuitive brain structure information and the distribution of infarct areas, and clinical data contains key clinical indicators such as NIHSS score, ASPECT score, age, etc., the combination of the two can more comprehensively reflect the patient's condition. Therefore, the present invention innovatively adopts a cross-multi-head attention mechanism for fusion. Among them, since clinical data represents the doctor's score, medical history information, etc., which are usually used to guide medical decision-making, in the fusion process, the present invention uses clinical features as queries (Q) to find the most relevant information from the imaging data. On the other hand, imaging data contains rich spatial structural information. The present invention uses clinical features as the "query" source, and imaging features as keys (K) and values (V), and uses the attention mechanism to filter out the imaging information most relevant to the clinical features. And through 16 attention heads to capture the correlation of cross-modal information, so as to achieve efficient feature fusion. The query, key and value are represented as follows:

[0098] Q=W q z clinical ,K=W k z image , V=W v z image

[0099] W q 、W k 、W v is a learnable projection matrix. clinical , z image They represent the clinical information feature vector and NCCT image feature vector respectively, and Q, K, and V represent query, key, and value respectively.

[0100] The present invention uses 16 attention heads to enhance the expressiveness of the model.

[0101] Then, calculate the cross attention score:

[0102]

[0103] Among them, CrossAttention(Q,K,V) is the cross attention score, QK T To calculate the similarity between the query and the key, the correlation between clinical features and imaging features is expressed. is a scaling factor that prevents excessively large values from causing gradient vanishing. The Softmax operation normalizes the weights, giving higher weight to the most relevant image information. After multiplying by V, the present invention obtains the weighted sum of the clinical features and the image features, i.e., the final feature representation of the fusion of image features and clinical information features.

[0104] The cross-modal feature fusion mechanism in this step proposes a feature fusion method based on multi-head cross attention for medical imaging and clinical feature information, using clinical information as Q and imaging features as K and V to achieve deep interaction between the two modal information, so that the model can dynamically focus on specific areas in the image that are related to clinical information, thereby improving prediction accuracy. The present invention utilizes a multi-head cross attention mechanism to fuse brain 3D NCCT imaging data and clinical table information data, effectively solving the modality bias problem between NCCT images and low-dimensional clinical features. It achieves fine-grained association of semantic information between modalities, enabling the fused features to more accurately reflect the patient's disease development trend. This method not only improves the predictive performance of the fusion model, but also significantly improves the robustness of feature representation, providing an efficient data integration solution for the field of multimodal medical imaging analysis.

[0105] The fourth step is to predict whether the blood vessels are effectively recanalized. In order to more accurately predict the effect of blood vessel recanalization after thrombectomy in patients with acute ischemic stroke, a deep learning network structure consisting of three independent branches: image modality, clinical information modality, and fusion modality was designed. Each branch consists of three linear layers, a ReLU activation function, and a Softmax classification layer. The corresponding loss is calculated using binary cross entropy, and the formula is:

[0106]

[0107] Represents the loss value, which measures the gap between the model prediction result and the true label. true is the true label, indicating the true category of the sample, with a value of 0 or 1. P is the probability predicted by the model, that is, the probability that the sample belongs to the positive class, ranging from (0, 1). The final loss is obtained by creatively combining contrastive learning loss, image loss, clinical information loss, and multimodal fusion loss:

[0108]

[0109] in, For the final loss, represents the image model training loss, represents the clinical information model training loss, represents the contrastive learning loss, represents the multimodal fusion model training loss, w1(t) and w2(t) are the weights dynamically adjusted over time t.

[0110] The total loss The data is passed back to each layer in the model through the back-propagation algorithm, and the weights and biases of the model are updated. The above process is repeated. Through multiple iterations, the total loss is continuously reduced, and the model gradually converges. As the number of iterations increases, the total loss gradually decreases. When the loss value no longer decreases significantly or reaches the preset number of iterations, the network is considered to have converged. The converged model can predict new input data and output the results of whether the blood vessels of patients with acute ischemic stroke are effectively restored after thrombectomy.

[0111] In the fourth step, an innovative loss function is designed to construct a multimodal collaborative loss, including image modality loss, clinical information modality loss, fusion modality loss, and contrastive learning loss. The loss weights are optimized through a dynamic weight adjustment mechanism so that different modality features can be fully utilized during the training process. This loss design can not only enhance the synergy between modalities and improve the model's ability to capture key features, but also effectively suppress the noise interference that may exist in a single modality, improve the robustness and generalization ability of the model, and thus more accurately predict the effect of postoperative vascular recanalization in patients with acute ischemic stroke.

[0112] The medical treatment logic diagram of the present invention is as follows Figure 4 As shown, the above combination Figure 4 The present invention describes a medical logic for predicting the prognosis of ischemic stroke that can be used to implement the present invention. It should be understood that the medical logic herein is merely exemplary and the implementation of the present invention is not limited thereto but may be modified without departing from the spirit of the present invention.

[0113] It should be noted that although the operations of the present method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the operations shown must be performed to achieve the desired results. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0114] It should be understood that the terms "include" and "comprising" used in the description and claims of this application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal method for predicting the effect of vascular recanalization after stroke surgery, characterized in that: The following steps are involved: Obtain the patient's clinical information characteristics and NCCT images; Preprocessing the clinical information features, calculating the importance index of each preprocessed clinical information feature and ranking them to obtain clinical modality features; Preprocessing the NCCT image to obtain a denoised NCCT image, and inputting the denoised NCCT image into a deep convolutional neural network to obtain NCCT image modality features; Performing comparative learning on the clinical modality features and the NCCT imaging modality features and calculating the NCCT imaging modality feature loss, the clinical modality feature loss, and the comparative learning loss; Obtaining a fused modality feature by fusing the NCCT image modality feature and the clinical modality feature and calculating the fused modality feature loss; The dynamic total loss is calculated based on the NCCT image modality feature loss, clinical modality feature loss, fusion modality feature loss, and contrastive learning loss, and the dynamic total loss is input into the deep convolutional neural network until the deep convolutional neural network converges. After convergence, the deep convolutional neural network predicts the newly input clinical information features and NCCT images, and outputs the prediction results of vascular recanalization after thrombectomy.

2. The method for predicting the effect of vascular recanalization after stroke surgery based on multimodality according to claim 1, characterized in that: The specific steps of preprocessing the clinical information features include: Set ranges for continuous variables in clinical information, identify them through binning and boxplots, and perform corrections and delete outliers; Fields with missing values greater than the preset missing value threshold are discarded, and the remaining categorical fields with missing values are filled with the mode; One-hot encoding was used for multi-class unordered variables in clinical information, and PCA dimensionality reduction was performed on highly correlated features.

3. The method for predicting the effect of vascular recanalization after stroke surgery based on multimodality according to claim 1, characterized in that: The specific steps of obtaining the clinical modality features include: The importance features of each clinical information feature are calculated and sorted, structural features are screened out, and the structural features are concatenated to form a character string, which is converted into a text that conforms to the natural language format, and clinical modality features are extracted through a pre-trained model.

4. The method for predicting the effect of vascular recanalization after stroke surgery based on multimodality according to claim 1, characterized in that: The specific steps of obtaining the NCCT imaging modality features include: Extracting local features of the denoised NCCT image; Normalizing the activation value of each channel of the denoised NCCT image; The standardized denoised NCCT images are subjected to channel compression and deep features are extracted to obtain multi-scale features. Further extracting and integrating local features of the multi-scale features; The feature map of the integrated multi-scale features is reduced in dimensionality, and high-order semantic features are further extracted after filtering out noise, so that the network can capture deeper image features; The size of the extracted multi-scale feature map is reduced, and the three-dimensional features of the multi-scale feature map are converted into a one-dimensional vector to obtain the NCCT image modality features.

5. The method for predicting the effect of vascular recanalization after stroke surgery based on multimodality according to claim 1, characterized in that: The specific steps of performing comparative learning based on the clinical modality features and the NCCT image modality features include: A group of samples is selected, and the clinical modality features and NCCT imaging modality features of the same patient in the samples are regarded as positive sample pairs, and the clinical modality features and NCCT imaging modality features of different patients in the samples are regarded as negative sample pairs; Comparative learning is performed on all positive and negative sample pairs in a set of samples to capture the similarities between different modality features of the same patient and the differences between modality features of different patients.

6. The method for predicting the effect of vascular recanalization after stroke surgery based on multimodality according to claim 1, characterized in that: The calculation steps of the fusion modality feature include: Taking clinical modality features as Q and NCCT imaging modality features as K and V, the cross-headed multi-head attention mechanism is used to screen out the NCCT imaging modality features that are most relevant to the clinical modality features, and multiple attention heads are used to capture the correlation of cross-modality information; The NCCT image modality features and clinical table modality features are weightedly fused by calculating the cross-attention score to obtain the fused modality features.

7. A multimodal post-stroke vascular recanalization effect prediction system, characterized by: It includes clinical information preprocessing module, clinical feature extraction module, 3D image preprocessing module, 3D image feature extraction module, contrast learning and feature alignment module, multimodal feature fusion module and prediction module. The clinical information preprocessing module is used to preprocess clinical information features; The clinical feature extraction module is used to calculate the importance index of each preprocessed clinical information feature and sort them to obtain clinical modality features; The 3D image preprocessing module is used to preprocess the NCCT image to obtain a denoised NCCT image; The 3D image feature extraction module extracts NCCT image modality features based on a deep convolutional neural network; The contrastive learning and feature alignment module is used to perform contrastive learning based on the clinical modality features and the NCCT image modality features and calculate the NCCT image modality feature loss, the clinical modality feature loss and the contrastive learning loss; The multimodal feature fusion module is used to fuse the NCCT image modality features and the clinical modality features to obtain fused modality features and calculate the fused modality feature loss; The prediction module is used to predict the newly input clinical information features and 3DCT images, and output the prediction results of the patient's vascular recanalization after thrombectomy.

8. The multimodal post-stroke vascular recanalization effect prediction system according to claim 7, characterized in that: The deep convolutional neural network includes a 3D convolution layer, a normalization layer, a first dense block, a 3D convolution layer, a first pooling layer, a second dense block, a second pooling layer and a fully connected layer; The 3D convolution layer is used to extract local features of the denoised NCCT image and simultaneously learn local contrast and edge information in the denoised NCCT image; The normalization layer is used to standardize the activation value of each channel of the denoised NCCT image; The first dense block is used to compress the number of channels of the standardized denoised NCCT image and extract deep features to obtain multi-scale features; The 3D convolution layer is used to further extract and integrate local features of the multi-scale features; The first pooling layer is used to reduce the dimension of the feature map of the integrated multi-scale features and filter out noise before inputting it into the second dense block; The second dense block is used to further extract high-order semantic features, so that the network can capture deeper image features, and the extracted multi-scale features are input into the second pooling layer; The second pooling layer is used to reduce the size of the feature map; The fully connected layer is used to convert the three-dimensional features into a one-dimensional vector.

9. The multimodal post-stroke vascular recanalization effect prediction system according to claim 7, characterized in that: The clinical feature extraction module uses XGBoost to calculate importance indicators; uses the text encoder of the CLIP pre-trained model to extract clinical modality features; and uses an upsampling layer to enhance the resolution of clinical modality features; In the process of preprocessing the NCCT image to obtain the denoised NCCT image by the 3D image preprocessing module, brain region segmentation is performed using the U-Net pre-trained deep learning model to remove non-brain tissue information; The contrastive learning loss is calculated using a contrastive loss function, and the NCCT imaging modality feature loss, clinical modality feature loss, and fusion modality feature loss are calculated using binary cross entropy.

10. The multimodal post-stroke vascular recanalization effect prediction system according to claim 7, characterized in that: The prediction module is composed of a deep learning network, which includes an image modality branch, a clinical information modality branch and a fusion modality branch. The image modality branch, the clinical information modality branch and the fusion modality branch are all composed of three linear layers, a ReLU activation function and a Softmax classification layer.

Images