Multimodal cerebral apoplexy prediction method based on text-3D image

Abstract

The invention discloses a multimodal cerebral apoplexy prediction method based on a text-3D image, and the process comprises the following steps: collecting sample data and preprocessing: obtaining the MRI samples of a cerebral apoplexy patient and the natural language description information of the focus area of each sample, and converting the text description into vector representation as the input of a subsequent model; constructing a model: using a convolutional neural network (CNN) as an image feature coding module, and adopting a U-Net architecture; model optimization: weighting of Soft Dice Loss and Cross Entry Loss is adopted as a loss function of the model, so that the model can be better optimized; according to the multi-modal fusion, visual information and language information are fused through a Transform structure, and deep interaction of cross-modal features is achieved. And model evaluation: evaluating the performance of the segmentation model by using four evaluation indexes including a Dice coefficient, accuracy, a recall rate and HD95. And the segmentation result is continuously optimized by comparing the performance of the model under different evaluation indexes.

Description

Technical Field

[0001] The technical field relates to disease prediction in bioinformatics, especially the study of text-image multimodal data. The method is specifically a text-3D image-based multimodal stroke prediction method, which aims to accurately segment the stroke lesion area based on 3D-MRI images and the natural description language of the lesion location in the image. Background Art

[0002] Stroke is a series of diseases caused by blockage or ischemia of blood vessels supplying the brain. After a stroke, T1-weighted MRI is a commonly used non-invasive neuroimaging technique that can clearly show the morphological changes of brain tissue and lesions. By observing MRI images, radiologists can determine the presence, location, and size of lesions and make a preliminary diagnosis. However, stroke lesions are usually irregular in shape and small in size, making manual delineation of lesions time-consuming and dependent on the radiologist's experience level [3]. Different radiologists may have different classifications of the same lesion, and the same radiologist may have inconsistent classifications of the same lesion at different times. This subjectivity and variability increase diagnostic uncertainty and errors.

[0003] In order to solve the problems caused by manual segmentation, many researchers use deep learning to build automatic stroke lesion segmentation models. The goal is to automatically identify and segment the stroke area through algorithms, thereby assisting doctors in diagnosis and treatment. This task usually involves complex image processing technology and advanced deep learning models. In recent years, multimodal models have received increasing attention. In the field of automatic image segmentation, natural language processing (NLP) and computer vision (CV) technologies are mainly combined to more accurately segment the target area in the image. This technology is called Referring Image Segmentation, which refers to the technology of locating and segmenting specific image areas through natural language descriptions [9]. Automatic stroke MRI image segmentation combined with Referring Image Segmentation can not only improve the accuracy of segmentation, but also make the model more flexible and practical. It has broad application prospects in medical image analysis, especially in scenarios where specific lesions need to be accurately located. Summary of the invention

[0004] In order to solve the problems caused by manual segmentation, many researchers use deep learning to build automatic stroke lesion segmentation models. The goal is to automatically identify and segment the stroke area through algorithms, thereby assisting doctors in diagnosis and treatment. This task usually involves complex image processing technology and advanced deep learning models. In recent years, multimodal models have received increasing attention. In the field of automatic image segmentation, natural language processing (NLP) and computer vision (CV) technologies are mainly combined to more accurately segment the target area in the image. This technology is called Referring Image Segmentation, which refers to the technology of locating and segmenting specific image areas through natural language descriptions [9]. Automatic stroke MRI image segmentation combined with Referring Image Segmentation can not only improve the accuracy of segmentation, but also make the model more flexible and practical. It has broad application prospects in medical image analysis, especially in scenarios where specific lesions need to be accurately located.

[0005] The following is a technical solution to achieve the purpose of the present invention, comprising the following steps: 1) Collect sample data: obtain MRI samples of stroke patients and natural language description information of the lesion area of ​​each sample; 2) Image preprocessing: First, use methods such as median filtering to denoise the MRI images to improve image quality. Then, resize all images to the same size to ensure input consistency; 3) Text preprocessing and feature extraction: standardize the descriptive language to ensure consistency in text format and writing. Then, use the BERT model to extract high-dimensional features from the text description and convert them into vector representations as input for subsequent models; 4) Model construction: Use convolutional neural network (CNN) as the image feature encoding module, all using U-Net architecture, including encoder and decoder. This hierarchical approach can better extract image features and improve the accuracy of lesion segmentation; 5) Model optimization: We use the weighted combination of Soft Dice Loss and Cross Entropy Loss as the loss function of the model to better optimize the model; 6) Multimodal fusion: We use the Transformer structure to fuse visual and language information to achieve deep interaction of cross-modal features. This method can make full use of the information in MRI images and descriptive language to improve the segmentation effect of the lesion area; 7) Model evaluation: Use four evaluation indicators, namely Dice coefficient, precision, recall rate, and HD95, to evaluate the performance of the segmentation model. By comparing the performance of the model under different evaluation indicators, the segmentation results can be continuously optimized. Beneficial Effects

[0006] This model uses one-hot encoding and nucleotide chemical properties (NCP) for encoding. An improved Dense Net is used to extract high-level features from the original encoding. The importance of high-level features is evaluated using the self-attention module and then input into the fully connected neural network to obtain the prediction probability. Finally, the final classification results are integrated through voting. The experimental results show that we have achieved significant performance improvement compared with existing methods. The ACC values ​​and MCC values ​​of all datasets are higher than the state-of-the-art methods, which shows that our proposed enhancer prediction model has strong generalization ability while having high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 3D-MMSUnet model architecture diagram Figure 2 Multimodal fusion module diagram Figure 3 Visualization of small lesion segmentation results Figure 4 Visualization of large lesion segmentation results Figure 5 Ablation experiment results DETAILED DESCRIPTION

[0008] The specific implementation modes of the present invention are explained below in conjunction with the accompanying drawings. The accompanying drawings are for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0009] Figure 1 The overall process of the 3D-MMSUNet model we proposed is shown. The model uses the Transformer structure to fuse visual and language information to achieve deep interaction of cross-modal features. Given a 3D image and a natural language expression describing the distribution of lesions in the image as input, the model outputs a pixel-by-pixel mask that accurately marks the distribution area of ​​the lesions.

[0010] First, we preprocess the data. For images, we crop them to a size of 128*128*128 for easy input. For text, we extract its language features. We use the BERT model to encode the input natural language description and embed the low-dimensional language expression into a high-dimensional vector space.

[0011] Secondly, after obtaining these language features, we continuously fuse visual features with language features through a staged structure of five convolutional blocks and a multimodal fusion module to achieve deeper cross-modal information interaction. Figure 2 As shown in Figure 1. This process consists of two main steps: In the first step, we perform pixel-by-voxel and word-by-word attention operations on image features and text features, that is, we calculate the correlation between each text feature and each pixel. In the second step, we introduce a language gate, the purpose of which is to prevent Fi from suppressing the visual signal in Vi and allow an adaptive amount of language to flow to the next stage of encoding. At each stage, language-aware vision can be continuously optimized through these two steps.

[0012] Then, the 3D segmentation mask is obtained through its decoder, and the generalization ability of the model is further evaluated on an independent test set.

[0013] We compared the performance of MMSUnet and existing state-of-the-art methods in the task of stroke 3D MRI segmentation. We divided the dataset into a training set, a validation set, and an independent test set. The training set and validation set were divided using a 5-fold cross-validation method, and the performance of different models in terms of Dice coefficient, Precision, Recall, and HD95 were compared. The experimental results are shown in Table 1. The experimental results show that 3D-MMSUnet outperforms nnUnet3D and other existing methods in most cases, especially in the Dice coefficient and HD95 indicators, demonstrating its superiority in segmentation accuracy and lesion boundary accuracy.

[0014] In order to further test the generalization ability of the model, an independent test set is usually used to evaluate the performance of each model to ensure the model's inference ability on unseen data. In this study, we further verified the advantages of our proposed model by comparing the performance of multiple models on an independent test set. From the results in Table 2, it can be seen that our model performs better than the second-best nnUnet3D on HD95 and Recall, especially in terms of accuracy and boundary processing.

[0015] Table 1 Comparison with other models on the validation set Dice↑ Precision↑ Recall↑ HD95↓ nnUnet3D 0.606 0.650 0.639 26.51 Ensemble pp 0.606 0.651 0.637 26.51 LKA-E 3D 0.570 0.604 0.615 30.38 LKA-ED 3D 0.578 0.607 0.628 29.28 SwinUNETR 0.522 0.502 0.679 27.48 3D-MMSUnet 0.621 0.652 0.650 23.81 .

[0016] Table 2 Comparison with other models on independent test sets Dice↑ Precision↑ Recall↑ HD95↓ nnUnet3D 0.603 0.659 0.633 30.22 Ensemble pp 0.601 0.3855 0.637 30.05 LKA-E 3D 0.576 0.620 0.612 32.83 LKA-ED 3D 0.561 0.585 0.617 29.10 SwinUNETR 0.568 0.607 0.611 32.76 3D-MMSUnet 0.607 0.653 0.642 28.61 .

[0017] In this section, we conducted two parts of ablation experiments. The first part of the experiment tested whether the 3D-PWAM as a bottleneck has any effect, and whether its number of layers has a significant impact on the model performance, see Figure 5 (Table 3). The second is to test different backbones and compare the performance of the traditional CNN module and the Vision Transformer module. Figure 5 (Table 4 part).

[0018] Table 3 Comparative experiments with different numbers of layers

[0019] Table 4 Comparative experiments of different backbones.

Claims

1. A multimodal stroke prediction method based on text and 3D images, 3D-MMSUnet, comprises the following steps: 1) Collect sample data: obtain MRI samples of stroke patients and natural language description information of the lesion area of ​​each sample; 2) Image preprocessing: First, use methods such as median filtering to denoise the MRI images to improve image quality. Then, resize all images to the same size to ensure input consistency; 3) Text preprocessing and feature extraction: standardize the descriptive language to ensure consistency in text format and writing. Then, use the BERT model to extract high-dimensional features from the text description and convert them into vector representations as input for subsequent models; 4) Model construction: Use convolutional neural network (CNN) as the image feature encoding module, all using U-Net architecture, including encoder and decoder. This hierarchical approach can better extract image features and improve the accuracy of lesion segmentation; 5) Model optimization: We use the weighted combination of Soft Dice Loss and Cross Entropy Loss as the loss function of the model to better optimize the model; 6) Multimodal fusion: We use the Transformer structure to fuse visual and language information to achieve deep interaction of cross-modal features. This method can make full use of the information in MRI images and descriptive language to improve the segmentation effect of the lesion area; 7) Model evaluation: Use four evaluation indicators, namely Dice coefficient, precision, recall rate, and HD95, to evaluate the performance of the segmentation model. By comparing the performance of the model under different evaluation indicators, the segmentation results can be continuously optimized.

2. This method fuses and aligns 3D MRI image features with the natural description language features of the stroke area in the image, improving the prediction accuracy of the model while preventing the model from ignoring small lesions. It has superior recognition performance compared to the most advanced methods.

3. The multimodal stroke prediction method based on text and 3D images according to claim 1, characterized in that: The construction of the basic segmentation model and multimodal fusion described in step 4) and step 6) can not only improve the segmentation accuracy of the image, but also ensure that the model will not ignore small lesion areas, making the model more flexible and practical in practical applications.

Images