Thrombolysis prediction method based on deep learning and multi-modal fusion

Abstract

The invention discloses a thrombolysis prediction method based on deep learning and multi-modal fusion, and belongs to the technical field of thrombolysis prediction, and the method comprises the steps: S1, collecting an MRI image of a patient with ischemic cerebral apoplexy, and constructing a cerebral apoplexy automatic segmentation model; s2, carrying out image registration; s3, lesion segmentation; and S4, thrombolysis prediction. According to the cerebral apoplexy automatic segmentation model based on the gating parallel Mama network, the long-distance dependency relationship in the image can be effectively captured, ischemic cerebral apoplexy lesions can be rapidly and accurately segmented, and the lesion identification efficiency and accuracy are greatly improved; according to the algorithm provided by the invention, the linear time complexity is realized, the computing resource consumption is greatly reduced, and the bottleneck of the prior art is broken through; according to the cerebral apoplexy thrombolysis prediction model based on the multi-modal data, scientific automatic decision support is provided for judging whether the patient is suitable for thrombolysis treatment or not, and the quality and safety of medical decision are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of thrombolysis prediction, and in particular to a thrombolysis prediction method based on deep learning and multimodal fusion. Background Art

[0002] Stroke is a very common cerebrovascular disease worldwide and one of the main causes of death and disability. It is characterized by high morbidity and mortality. In the United States, for example, there are approximately 795,000 new cases or recurrent stroke patients each year, and the prevalence of stroke in adults is as high as 3%.

[0003] Ischemic stroke is the main type of stroke, accounting for about 70-80% of all strokes. Thrombolytic therapy, as one of the effective means of treating ischemic stroke, can dissolve blood clots and restore blood flow to the brain, thereby saving damaged brain tissue. Although the lesions revealed by magnetic resonance imaging (MRI) in brain imaging examinations are usually regarded as irreversible ischemic damage (i.e., ischemic core), some patients are still suitable for thrombolytic therapy.

[0004] How to accurately predict which patients are suitable for thrombolytic therapy has always been a challenge for clinicians. In the current medical technology context, automatic segmentation of lesions is an indispensable basic link in predictive analysis. However, existing automatic segmentation methods (such as models based on convolutional neural networks (CNN) or Transformer) have obvious shortcomings. These methods perform poorly in capturing long-distance dependencies between image pixels and have high requirements for computing resources, resulting in heavy computational burdens and efficiency bottlenecks in practical applications. This technical limitation has seriously hindered its widespread application in clinical settings, especially when processing large-scale stroke imaging data.

[0005] To address this gap, the present invention proposes a thrombolysis prediction method based on deep learning and multimodal fusion, thereby providing efficient and accurate decision support for whether the patient is suitable for thrombolytic therapy. Summary of the invention

[0006] The purpose of the present invention is to provide a thrombolysis prediction method based on deep learning and multimodal fusion to solve the problems in the background technology.

[0007] To achieve the above object, the present invention provides a thrombolysis prediction method based on deep learning and multimodal fusion, which specifically includes the following steps:

[0008] S1. Construct a segmentation model: collect MRI images of ischemic stroke patients, construct an automatic stroke segmentation model based on gated parallel Mamba, and obtain the segmentation model weights;

[0009] S2. Image registration: align multiple MRI images before and after label-free thrombolysis to the same spatial reference system for image registration;

[0010] S3, lesion segmentation: the MRI image after S2 image registration is inferred using the segmentation model weights obtained in S1 to extract the predicted label;

[0011] S4. Thrombolysis prediction: Combined with the prediction labels obtained in S3, a multimodal stroke thrombolysis prediction model is constructed. The three-dimensional MRI, clinical characteristics and imaging features before thrombolysis to be predicted are used as input to predict thrombolysis and determine whether the patient is suitable for thrombolytic therapy.

[0012] Preferably, the S1 specifically comprises the following steps:

[0013] S11. Data acquisition: A dataset of multimodal MRI images of multiple ischemic stroke patients was collected. Each MRI image included four original images of diffusion-weighted imaging, apparent diffusion coefficient, T2-weighted imaging, and T2*-weighted imaging, and a labeled image. The labeled image included two label categories: background area and lesion area.

[0014] S12. Build a gated parallel Mamba-based stroke automatic segmentation model: Based on the standard encoder-decoder architecture in nnUNet, set the gated parallel Mamba module in the BottleNeck layer;

[0015] S13, model training and verification: The nnUNet self-configuration method is used to automatically configure hyperparameters for the dataset, and the dataset is used as input for model training. After the training is completed, the model generates a set of segmentation model weights.

[0016] Preferably, in S12, the gated parallel Mamba module adjusts the information flow of the Mamba module through the gate node and the two parallel branches to further compress the image features; the image features include batch size, number of channels, image height, image width and image depth;

[0017] The specific process is as follows: the image features are passed through a residual block and then Flattened to obtain a vector v of size (B, H×W×D, C), where B, C, H, W, and D are the batch size, number of channels, image height, image width, and image depth, respectively;

[0018] The vector v is represented as (B, L, C). After layer normalization, the vector v enters two branches. In one branch, it passes through the Mamba module and the gating unit to obtain the weight ω, ω∈(0,1). The obtained weight is multiplied with the new vector obtained from the other branch to obtain the vector F:

[0019]

[0020] Where v1 and v2 are the feature vectors input to the two branches, σ(·) represents the process of generating weights through the gating unit, and M(·) is the calculation process of the Mamba module.

[0021] After reshape, vector F becomes (B ′ ,H ′ ×W ′ ×D ′ ,C ′ ), and finally output after transposition (B ′ ,C ′ ,H ′ ,W ′ ,D ′ )-sized vector.

[0022] Preferably, the S2 specifically comprises the following steps:

[0023] S21. In order to avoid the convergence problem of subsequent registration due to excessive deviation of the starting position, the spatial centroid alignment of multiple label-free MRI images before and after thrombolysis is performed, which is expressed as:

[0024] T=C f -C m ;

[0025] Where T is the distance the image needs to be translated, C f is the centroid coordinate of the fixed image, C m is the centroid coordinate of the moving image;

[0026] S22, perform rigid body transformation on the MRI image processed by S21, expressed as:

[0027] T(x) = R x + t;

[0028] Where T(x) is the coordinate of the image to be registered after transformation, x is the coordinate of the image to be registered, R is the rotation matrix, and t is the translation vector;

[0029] S23, perform Mattes mutual information measurement on the MRI image processed by S22, expressed as:

[0030] MI(F,M)=H(F)+H(M)-H(F,M);

[0031] Where MI(F,M) is the mutual information measure between images F and M, H(F) and H(M) are the entropies of images F and M respectively, and H(F,M) is the joint entropy of images F and M;

[0032] S24, performing linear interpolation calculation on the MRI image processed by S23, expressed as:

[0033]

[0034] Where I(x,y) is the pixel value of the new coordinate point, a and b are the offsets between the coordinate and the nearest neighbor point, and x0, y0, x1, and y1 are the integer coordinates of the neighboring points of the pixel to be interpolated in the original grid.

[0035] Preferably, the S3 specifically includes the following steps:

[0036] The MRI image after the S2 image registration is inferred using the segmentation model weights obtained in S1 to obtain the corresponding label image, which is manually corrected by experts, and the difference in lesion volume before and after thrombolysis is used as the predicted label, expressed as:

[0037]

[0038] In the formula, L(A v ,B v ) is the predicted label, A v is the volume of the lesion after thrombolysis in this case, B v This is the lesion volume before thrombolysis in this case.

[0039] Preferably, in S4, the input of the multimodal stroke thrombolysis prediction model includes three groups of features of different modalities, specifically three-dimensional MRI before thrombolysis, clinical characteristics and imaging genomics features. Each group of features is extracted through its own processing module, and finally fused and spliced ​​in the fully connected layer to obtain the final feature vector. The obtained feature vector is processed and finally the prediction result for classification is output.

[0040] Preferably, the multimodal stroke thrombolysis prediction model includes a three-dimensional MRI processing branch, a clinical feature processing branch, and an imaging genomics feature processing branch. In the three-dimensional MRI data processing branch, a residual module and an attention module are introduced to extract features. The calculation formula of the residual module is as follows:

[0041] y=F(x,{W i})+x;

[0042] Where x and y are input and output respectively, F(x,{W i}) represents the features extracted by the convolutional layer and batch normalization layer.

[0043] Therefore, the present invention provides a thrombolysis prediction method based on deep learning and multimodal fusion, which has the following beneficial effects:

[0044] (1) The present invention proposes an automatic stroke segmentation model based on a gated parallel Mamba network. By introducing the Mamba structure and combining it with the local feature extraction capability of the CNN convolutional layer, it can effectively capture the long-distance dependencies in the image, quickly and accurately segment ischemic stroke lesions, and greatly improve the efficiency and accuracy of lesion identification.

[0045] (2) The algorithm proposed in the present invention achieves linear time complexity, greatly reduces the consumption of computing resources, and breaks through the bottleneck of the existing technology.

[0046] (3) The stroke thrombolysis prediction model based on multimodal data proposed in the present invention makes comprehensive use of the patient's MRI images, clinical data and radiomics information to achieve accurate prediction of lesion recoverability, thereby providing scientific and automated decision support for whether the patient is suitable for thrombolytic therapy. At the same time, the model is flexible in design, can properly handle various data combination inputs, effectively deal with the problem of missing modal information, enhance the comprehensiveness and reliability of prediction, and plays an important role in clinical medicine.

[0047] (4) Compared with the traditional method of making decisions based solely on the volume of ischemic stroke lesions, this technology effectively breaks through its limitations by comprehensively considering information from more dimensions, providing doctors with more scientific and comprehensive decision-making support, helping to more accurately determine whether patients are suitable for thrombolytic therapy, thereby optimizing the formulation of treatment plans and improving the quality and safety of medical decisions.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The overall process framework diagram of the method proposed in the present invention;

[0050] Figure 2 This is a diagram of the architecture of an automatic stroke segmentation model according to an embodiment of the present invention;

[0051] Figure 3 This is a diagram of the gated parallel Mamba module architecture of an embodiment of the present invention;

[0052] Figure 4 This is a diagram of the architecture of a multimodal stroke thrombolysis prediction model according to an embodiment of the present invention;

[0053] Figure 5 1 is a diagram showing the architecture of a residual module according to an embodiment of the present invention;

[0054] Figure 6 This is a diagram of the attention module architecture of an embodiment of the present invention;

[0055] Figure 7A comparison diagram of the segmentation results and true labels of the stroke automatic segmentation model in some test samples according to an embodiment of the present invention;

[0056] Figure 8 The ROC curve diagram of the prediction model of the embodiment of the present invention in which only the three-dimensional MRI data before thrombolysis is input;

[0057] Fig. 9 ROC curve diagram for inputting only clinical feature data into the prediction model of the embodiment of the present invention;

[0058] Fig.10 The ROC curve diagram of the prediction model of the embodiment of the present invention in which only the radiomics feature data is input;

[0059] Fig.11 ROC curve diagram of inputting three-dimensional MRI and clinical characteristic data before thrombolysis into the prediction model of the embodiment of the present invention;

[0060] Fig.12 ROC curve diagram of inputting three-dimensional MRI and radiomics feature data before thrombolysis into the prediction model of the embodiment of the present invention;

[0061] Fig.13 ROC curve diagram of clinical characteristics and radiomics feature data input to the prediction model of the embodiment of the present invention;

[0062] Fig.14 ROC curve diagram of the three-dimensional MRI data before thrombolysis, clinical characteristics and radiomics characteristics input into the prediction model of the embodiment of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0065] Example

[0066] like Figure 1 As shown, the present invention provides a thrombolysis prediction method based on deep learning and multimodal fusion. The method uses multimodal magnetic resonance images from the Department of Imaging of Tianjin Huanhu Hospital to test the effect of multimodal data fusion on model performance. The steps are as follows:

[0067] S1. Construct a segmentation model: Collect MRI images of ischemic stroke patients, construct an automatic stroke segmentation model based on gated parallel Mamba, and obtain the segmentation model weights; specifically:

[0068] S11. Data acquisition: A total of 172 multimodal MRI images were collected from the Department of Imaging of Tianjin Huanhu Hospital. Each image included four original images of diffusion-weighted imaging, apparent diffusion coefficient, T2-weighted imaging, and T2*-weighted imaging, and a labeled image. The labeled image included two label categories: background area and lesion area.

[0069] S12. Constructing an automatic stroke segmentation model based on gated parallel Mamba:

[0070] like Figure 2 As shown in the figure, based on the standard encoder-decoder architecture in nnUNet, a gated parallel Mamba module is set and implemented in the BottleNeck layer. This module can capture local features while capturing long-distance dependencies in the image, and compress the extracted information to increase the expressiveness of the model.

[0071] The model input is a 4×16×128×128 vector, and after the segmentation model output is two channels of 0 and 1, corresponding to the two label categories of the label image in S11.

[0072] like Figure 3 As shown, the gated parallel Mamba module specifically includes the following steps:

[0073] Through the gate node and two parallel branches, the information flow of the Mamba module is adjusted to further compress the image features and increase the expressiveness of the model. The image features B, C, H, W and D are the batch size, number of channels, image height, image width and image depth respectively;

[0074] The specific process is as follows: the image features pass through a residual block and then a Flatten operation to obtain a vector v of size (B, H×W×D, C), expressed as (B, L, C). The vector v is normalized by the layer and enters two branches. In one branch, it passes through the Mamba module and the gating unit to obtain the weight ω, ω∈(0,1). The obtained weight is multiplied with the new vector obtained from the other branch to obtain the vector F:

[0075]

[0076] Where v1 and v2 are the feature vectors input to the upper and lower branches respectively, σ(·) represents the process of generating weights through the gating unit, and M(·) is the calculation process of the Mamba module;

[0077] After reshape, vector F becomes (B ′ ,H ′ ×W ′ ×D ′,C ′ ), and finally output after transposition (B ′ ,C ′ ,H ′ ,W ′ ,D ′ ) size vector;

[0078] S13, model training and validation: nnUNet self-configuration method is used to automatically configure hyperparameters for the dataset, with batch size set to 8, epoch to 800, and initial learning rate to 0.01; 5-fold cross validation method is used for validation;

[0079] like Figure 7 As shown in the figure, the 2D and 3D segmentation effects of this method on some samples of the dataset used are shown. It can effectively capture the features and information of each modality image on lesions of different sizes and shapes, and achieve accurate segmentation of lesions.

[0080] S2. Image registration: Align multiple unlabeled MRI images before and after thrombolysis to the same spatial reference system for image registration. Register another 690 unlabeled MRI data before and after thrombolysis provided by Huanhu Hospital. Each case includes five modalities: diffusion-weighted imaging, apparent diffusion coefficient, T2-weighted imaging, T2*-weighted imaging, and fluid inversion recovery. Specifically:

[0081] S21. In order to make the approximate position of the MRI image consistent in the physical space and avoid the convergence problem caused by the large deviation of the starting position in the subsequent registration, the spatial centroid alignment is required, which is expressed as:

[0082] T=C f -C m ;

[0083] Where T is the distance the image needs to be translated, C f is the centroid coordinate of the fixed image, C m is the centroid coordinate of the moving image.

[0084] S22, perform a rigid body transformation on the MRI image processed by S21, that is, the MRI image is registered by rotation and translation, without scaling or shearing. This transformation can keep the shape of the image structure unchanged and only adjust the position in space, which is expressed as:

[0085] T(x) = R x + t;

[0086] Where T(x) is the coordinate of the image to be registered after transformation, x is the coordinate of the image to be registered, R is the rotation matrix, and t is the translation vector;

[0087] S23, perform Mattes mutual information measurement on the MRI image processed by S22. By maximizing the mutual information, the algorithm can find the transformation that makes the MRI images most correlated after alignment, expressed as:

[0088] MI(F,M)=H(F)+H(M)-H(F,M);

[0089] Where MI(F,M) is the mutual information measure between images F and M, H(F) and H(M) are the entropies of images F and M respectively, and H(F,M) is the joint entropy of images F and M.

[0090] S24, linear interpolation: After S21, S22, and S23, the pixel coordinates of the MRI image may no longer be integer coordinates, and the image needs to be resampled. Therefore, the linear interpolation method is used to interpolate the MRI image processed by S23 to obtain a new pixel value, which is expressed as:

[0091]

[0092] Where I(x,y) is the pixel value of the new coordinate point, a and b are the offsets between the coordinate and the nearest neighbor point, and x0, y0, x1, and y1 are the integer coordinates of the neighboring points of the pixel to be interpolated in the original grid.

[0093] S3, lesion segmentation: The MRI image after S2 image registration is inferred using the segmentation model weights obtained in S1 to extract the predicted label; specifically:

[0094] The 690 data after S2 image registration were inferred using the segmentation model weights obtained in S1 to obtain the corresponding label images, which were manually corrected by experts. The difference in lesion volume before and after thrombolysis in the label image was used as the predicted label, expressed as:

[0095]

[0096] In the formula, L(A v ,B v ) is the predicted label, A v is the volume of the lesion after thrombolysis in this case, B v is the lesion volume before thrombolysis in this case. The calculation result of 0 represents suitable for thrombolytic therapy, and 1 represents unsuitable for thrombolytic therapy.

[0097] S4, thrombolysis prediction: Combined with the prediction labels obtained in S3, a multimodal stroke thrombolysis prediction model is constructed, and the three-dimensional MRI, clinical characteristics and imaging omics features before thrombolysis to be predicted are used as input to predict thrombolysis and determine whether the patient is suitable for thrombolytic therapy. Specifically:

[0098] S41. Construction of a multimodal stroke thrombolysis prediction model:

[0099] like Figure 4 As shown in the figure, the input includes features of three different modalities: three-dimensional MRI before thrombolysis, clinical features and radiomics features, which correspond to the three-dimensional MRI processing branch, the clinical feature processing branch and the radiomics feature processing branch respectively. Each feature is extracted through its own processing module and finally fused and spliced ​​in the fully connected layer to obtain the final feature vector. The obtained feature vector is processed and the prediction result for classification is output. The output size is 2, corresponding to the two categories of whether it is suitable for thrombolytic therapy in S3.

[0100] The 3D MRI processing branch before thrombolysis is processed through a series of 3D convolution, residual modules, and attention modules. The input size is (C, H, W, D), where C is the number of channels. In this method, 5 modalities are used, so C is 5. H, W, and D represent the height, width, and depth of 3D MRI, respectively. In the first layer of image feature extraction, the input image passes through a 3D convolution layer with a convolution kernel size of 3×3×3 and a padding of 1, and the dimension becomes (32, 128, 128, 128);

[0101] like Figure 5-Figure 6 As shown in the figure, the MRI data is then processed by two residual blocks. The residual blocks introduce residual connections, which enable the network to retain the features of the lower layers in the deep layers, thereby alleviating the gradient vanishing problem. The core idea is to learn the residual function, which is expressed as:

[0102] y=F(x,{W i})+x;

[0103] Where x and y are input and output respectively, F(x,{W i}) represents the features extracted by the convolution layer and batch normalization layer;

[0104] The residual block performs a series of convolution and batch normalization operations, and adjusts the number of channels, and the output dimension becomes (128, 128, 128, 128). After that, a weight mask is generated through the attention module, and the feature map at each position is weighted. The output dimension remains the same as the input, that is, (128, 128, 128, 128).

[0105] In order to reduce the spatial dimension of the feature map, the model uses an adaptive average pooling operation to reduce the spatial dimension from (128, 128, 128) to (128, 4, 4, 4). The 3D feature map after pooling is flattened and converted into a one-dimensional vector, which is then reduced to 256 dimensions through a fully connected layer.

[0106] The model can also process clinical features and radiomic features. These two types of data are reduced in dimension through linear layers. Clinical data include gender and age. Radiomics data are extracted by Pyradiomics tool, and the 10 most valuable features are selected as the input of the prediction model by SelectKbest method. Clinical data and radiomics information are each passed through two layers of fully connected networks, and the final output dimension is 64.

[0107] After processing the image features, clinical features, and radiomic features, the model concatenates these three features and concatenates them in the feature dimension to form a final feature vector, and finally outputs the prediction results for classification.

[0108] This model can flexibly select different data combinations for input according to needs. You can choose to use images, clinical or radiomics information alone, or combine multiple data sources. According to the marker variables specified during input, the corresponding feature extraction module is selectively executed, and the output is feature spliced.

[0109] If only image features are used, the size of the concatenated feature vector is 256; if image, clinical and radiomic features are used at the same time, the size of the concatenated feature vector is 384. The concatenated feature vector is then processed through several layers of fully connected networks, and the final output is a prediction result for classification. The final output size is 2, corresponding to the two categories of whether or not thrombolytic therapy is suitable in S3.

[0110] S42, Model training: The model adopts 5-fold cross validation, uses Adam optimizer, sets the learning rate to 0.01, the batch size to 4, trains for 50 epochs, and uses the cross entropy loss function in the training process, expressed as:

[0111]

[0112] In the formula, is the cross entropy loss calculated by the true label encoding and output probability, y is the encoding of the true label, is the output probability of the model, and C is the number of predicted categories.

[0113] like Figure 8-Figure 14As shown in the figure, this embodiment explores the impact of different modality data combinations on the performance of the prediction model for 690 cases of data, as well as the ROC curve obtained after five-fold cross-validation. The results show that the model integrating multimodal data shows significant performance improvement compared to the single modality model. The model accuracy using MRI data is 77%, and after further combining MRI data with imaging omics data, the accuracy of the model is increased to 90%. When MRI data, clinical data and imaging omics data are combined, the prediction accuracy of the model reaches the highest 92%, reflecting the significant enhancement effect of multimodal data fusion on model performance.

[0114] By combining MRI data, clinical data, and radiomics data, the model can accurately assess whether a patient is suitable for thrombolytic therapy. In addition, the model also demonstrates good predictive stability and ability when dealing with complex situations such as missing clinical data.

[0115] Therefore, the present invention proposes a thrombolysis prediction method based on deep learning and multimodal fusion, and proposes an automatic stroke segmentation model based on a gated parallel Mamba network. By introducing the Mamba structure and combining the local feature extraction capability of the CNN convolutional layer, it can effectively capture the long-distance dependencies in the image, quickly and accurately achieve the segmentation of ischemic stroke lesions, and greatly improve the efficiency and accuracy of lesion identification.

[0116] The algorithm proposed in the present invention achieves linear time complexity, greatly reduces the consumption of computing resources, and breaks through the bottleneck of the prior art.

[0117] The stroke thrombolysis prediction model based on multimodal data proposed in this invention makes comprehensive use of the patient's MRI images, clinical data and radiomics information to achieve accurate prediction of the recoverability of the lesion, thereby providing scientific and automated decision support for whether the patient is suitable for thrombolytic therapy. At the same time, the model is flexible in design, can properly handle various data combination inputs, effectively deal with the problem of missing modal information, enhance the comprehensiveness and reliability of the prediction, and plays an important role in clinical medicine.

[0118] Compared with the traditional method of making decisions based solely on the volume of ischemic stroke lesions, this technology effectively breaks through its limitations by comprehensively considering information from more dimensions, providing doctors with more scientific and comprehensive decision-making support, and helping to more accurately determine whether patients are suitable for thrombolytic therapy, thereby optimizing the formulation of treatment plans and improving the quality and safety of medical decisions.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A thrombolysis prediction method based on deep learning and multimodal fusion, characterized in that: The specific steps include: S1. Construct a segmentation model: collect MRI images of ischemic stroke patients, construct an automatic stroke segmentation model based on gated parallel Mamba, and obtain the segmentation model weights; S2. Image registration: align multiple MRI images before and after label-free thrombolysis to the same spatial reference system for image registration; S3, lesion segmentation: the MRI image after S2 image registration is inferred using the segmentation model weights obtained in S1 to extract the predicted label; S4. Thrombolysis prediction: Combined with the prediction labels obtained in S3, a multimodal stroke thrombolysis prediction model is constructed. The three-dimensional MRI, clinical characteristics and imaging features before thrombolysis to be predicted are used as input to predict thrombolysis and determine whether the patient is suitable for thrombolytic therapy.

2. The thrombolysis prediction method based on deep learning and multimodal fusion according to claim 1, characterized in that: The S1 specifically includes the following steps: S11. Data acquisition: A dataset of multimodal MRI images of multiple ischemic stroke patients was collected. Each MRI image included original images of four modalities: diffusion-weighted imaging, apparent diffusion coefficient, T2-weighted imaging, and T2*-weighted imaging, and a labeled image. The labeled image included two label categories: background area and lesion area. S12. Build a gated parallel Mamba-based stroke automatic segmentation model: Based on the standard encoder-decoder architecture in nnUNet, set the gated parallel Mamba module in the BottleNeck layer; S13, model training and verification: The nnUNet self-configuration method is used to automatically configure hyperparameters for the dataset, and the dataset is used as input for model training. After the training is completed, the model generates a set of segmentation model weights.

3. The thrombolysis prediction method based on deep learning and multimodal fusion according to claim 2, characterized in that: In S12, the gated parallel Mamba module adjusts the information flow of the Mamba module through the gate node and the two parallel branches to further compress the image features; the image features include batch size, number of channels, image height, image width and image depth; The specific process is as follows: the image features are Flattened after passing through a residual block to obtain a vector v of size (B, H×W×D, C), where B, C, H, W, and D are the batch size, number of channels, image height, image width, and image depth respectively; The vector v is represented as (B, L, C). After layer normalization, the vector v enters two branches. In one branch, it passes through the Mamba module and the gating unit to obtain the weight ω, ω∈(0, 1). The obtained weight is multiplied with the new vector obtained from the other branch to obtain the vector F: Where v1 and v2 are the feature vectors input to the two branches, σ(·) represents the process of generating weights through the gating unit, and M(·) is the calculation process of the Mamba module. After Reshape, the vector F is converted to (B′, H′×W′×D′, C′), and finally the vector of size (B′, C′, H′, W′, D′) is output after transposition.

4. The thrombolysis prediction method based on deep learning and multimodal fusion according to claim 3, characterized in that: The S2 specifically includes the following steps: S21. In order to avoid the convergence problem of subsequent registration due to excessive deviation of the starting position, the spatial centroid alignment of multiple label-free MRI images before and after thrombolysis is performed, which is expressed as: T=C f -C m ; Where T is the distance the image needs to be translated, C f is the centroid coordinate of the fixed image, C m is the centroid coordinate of the moving image; S22, perform rigid body transformation on the MRI image processed by S21, expressed as: T(x) = R x + t; Where T(x) is the coordinate of the image to be registered after transformation, x is the coordinate of the image to be registered, R is the rotation matrix, and t is the translation vector; S23, perform Mattes mutual information measurement on the MRI image processed by S22, expressed as: MI(F,M)=H(F)+H(M)-H(F,M); Where MI(F, M) is the mutual information measure between images F and M, H(F) and H(M) are the entropies of images F and M respectively, and H(F, M) is the joint entropy of images F and M; S24, performing linear interpolation calculation on the MRI image processed by S23, expressed as: Where I(x, y) is the pixel value of the new coordinate point, a and b are the offsets between the coordinate and the nearest neighbor point, and x0, y0, x1, and y1 are the integer coordinates of the neighboring points of the pixel to be interpolated in the original grid.

5. The thrombolysis prediction method based on deep learning and multimodal fusion according to claim 4, characterized in that: The S3 specifically includes the following steps: The MRI images after S2 image registration were inferred using the segmentation model weights obtained in S1 to obtain the corresponding label images, which were manually corrected by experts and the difference in lesion volume before and after thrombolysis was used as the predicted label, expressed as: In the formula, L(A v , B v ) is the predicted label, A v is the volume of the lesion after thrombolysis in the corresponding case, B v is the lesion volume before thrombolysis in the corresponding case.

6. The thrombolysis prediction method based on deep learning and multimodal fusion according to claim 5, characterized in that: In S4, the input of the multimodal stroke thrombolysis prediction model includes three groups of features of different modalities, specifically three-dimensional MRI before thrombolysis, clinical characteristics and imaging genomics features. Each group of features is extracted by its own processing module before input, and finally fused and spliced ​​in the fully connected layer to obtain the final feature vector. The obtained feature vector is processed and finally the prediction result for classification is output.

7. The thrombolysis prediction method based on deep learning and multimodal fusion according to claim 6, characterized in that: The multimodal stroke thrombolysis prediction model includes a three-dimensional MRI processing branch, a clinical feature processing branch, and an imaging genomics feature processing branch. In the three-dimensional MRI data processing branch, a residual module and an attention module are introduced to extract features. The calculation formula of the residual module is as follows: y=F(x,{W i })+x; Where x and y are input and output respectively, F(x, {W i }) represents the features extracted by the convolutional layer and batch normalization layer.

Images