Brain tumor patient survival analysis method and system

Abstract

The application relates to a brain tumor patient survival analysis method and system, the method comprising the following steps: collecting clinical information of tumor patients to establish a data set; converting, resampling, registering and standardizing image data in the data set; dividing the data set into a training set and a test set; establishing a Transformer neural network, pre-training the training set to extract global scale image features; cropping a tumor region, inputting the tumor region into the network to extract local scale image features, and obtaining divided image blocks; sorting all the divided image blocks to obtain part image blocks with higher contribution degrees to extract fine-grained image features; performing cross attention operation on the global scale image features, the local scale image features and the fine-grained image features to output a prediction result; training the model, selecting a good result as a final model, and saving the final model. The application can fully utilize whole brain images, and improves the accuracy and interpretability of the model.

Description

Technical Field

[0001] This invention relates to a method and system for analyzing the survival of patients with brain tumors. Background Technology

[0002] Malignant tumors are among the most prevalent diseases worldwide. Brain tumors, in particular, grow rapidly and are prone to recurrence after surgery, resulting in generally poor prognoses. Accurate survival risk assessments are crucial to assist doctors in developing personalized treatment plans, such as radiotherapy, chemotherapy, the need for tumor resection, and the extent of tumor removal.

[0003] Traditional survival analysis methods use statistical methods to predict survival time. Models utilizing deep learning and medical image data are relatively rare, and those using multi-scale information from medical image data are even rarer. Medical image-based survival models can be categorized into three types: first, radiomics models based on manually designed features; second, deep feature-based radiomics models; and third, models based on convolutional neural networks or other deep networks. Radiomics models extract hundreds or thousands of manually designed features from medical images, but these low-order features may not fully represent the image heterogeneity within lesion areas. Deep feature-based radiomics models simply use convolutional neural networks as feature extractors and then feed the extracted features into a Cox proportional hazards model. Due to their hierarchical nature, deep learning-based models can represent more abstract, high-order features compared to manually designed models, so survival models built with deep networks generally have better performance.

[0004] Current deep learning-based survival analysis models are built using convolutional neural networks. However, convolutional neural networks only have local receptive fields. Therefore, it is necessary to manually segment the tumor region in advance and crop the whole brain image to retain the tumor part. This process requires a lot of manpower to manually segment the tumor region and can only use a small area after cropping, which cannot utilize information from multiple scales of the whole brain image.

[0005] However, images with multi-scale information contain both global and fine-grained information, and information such as non-tumor regions and tumor locations in whole-brain images are related to tumor progression. Therefore, there is an urgent need for a method or system that can make full use of whole-brain images and incorporate multi-scale information from medical images into the model, thereby further improving the accuracy and interpretability of the model. Summary of the Invention

[0006] In view of this, it is necessary to provide a method and system for survival analysis of brain tumor patients that can make full use of whole-brain imaging and incorporate multi-scale information from medical images into the model, thereby further improving the accuracy and interpretability of the model.

[0007] This invention provides a method for survival analysis of brain tumor patients, comprising the following steps: a. collecting clinical information of tumor patients and establishing a dataset, wherein the clinical information includes: MRI image data, survival status, and survival time; b. converting, resampling, and registering the image data in the dataset; c. using whole-brain images of patients in the dataset as input to the model and standardizing the data in the dataset; d. dividing the dataset into training and testing sets according to the patients' clinical information; e. establishing a Transformer neural network, pre-training the training set to extract global-scale image features, and obtaining a class activation map; f. cropping the tumor region according to the obtained class activation map, inputting it into the network, and extracting local-scale image features to obtain segmented image patches; g. sorting all segmented image patches according to the global scale, and extracting fine-grained image features from the image patches with higher contribution; h. performing a cross-attention operation based on the obtained global-scale image features, local-scale image features, and fine-grained image features to fuse information from multiple scales and output prediction results; i. training the model, selecting the best-performing result as the final model, and saving it.

[0008] Specifically, the magnetic resonance imaging data includes multimodal magnetic resonance images; the magnetic resonance images of each patient contain four commonly used modes; the four commonly used modes are T1, T2, T1C, and Flair modes.

[0009] Specifically, step b includes:

[0010] First, the DICOM format is converted to NIFTI format; then the image is resampled; then the image is registered, matching points corresponding to the same spatial location at multiple time points one by one. Rigid registration mode is used during registration, and mutual information is used as the image similarity measure.

[0011] Step c includes:

[0012] The image data in the dataset is standardized using grayscale normalization and histogram equalization.

[0013] Specifically, step e includes:

[0014] A global visual Transformer pre-trained network and a visual Transformer network with three branches are constructed. Three-dimensional whole-brain images are divided into K fixed-size three-dimensional image patches, mapped to one-dimensional vectors of fixed length D, which serve as inputs to the global visual Transformer model. The four commonly used modalities are processed as four image channels. The model optimizes parameters using negative log-likelihood loss to extract global features. The self-attention mechanism in the global visual Transformer pre-trained network focuses attention on the tumor region. After training, the model with the best performance is saved, and the relevant class activation map obtained based on the parameter distribution of this model is used to locate the approximate tumor region.

[0015] Specifically, step g includes:

[0016] The sorting module sorts all the segmented image patches according to the global scale output, and extracts fine-grained image features from the image patches with higher contribution. The sorting module integrates the local information of each input one-dimensional vector through a linear projection layer, and then performs global pooling to obtain a vector containing global information. Finally, the global vector and each one-dimensional vector are concatenated and input into the linear classification module to predict the contribution of each vector. Features are extracted from the corresponding patch with the highest contribution and overlapping with the region through the fine-grained branch of the Transformer.

[0017] This invention also provides a brain tumor patient survival analysis system, which includes a dataset establishment unit, a data processing unit, a partitioning unit, a global-scale image feature extraction unit, a local-scale image feature extraction unit, a fine-grained image feature extraction unit, an output unit, and a selection unit. The dataset establishment unit collects clinical information from tumor patients to establish a dataset, including MRI image data, survival status, and survival time. The data processing unit performs format conversion, resampling, and registration of the image data in the dataset. The data processing unit also uses whole-brain images of the patients in the dataset as input to the model to standardize the data. The partitioning unit divides the dataset into a training set and a test set based on the patients' clinical information. The global-scale image feature extraction unit extracts global-scale image features, local-scale image features, and fine-grained image features. The output ... The feature extraction unit is used to build a Transformer neural network, pre-train the training set to extract global-scale image features, and obtain a class activation map. The local-scale image feature extraction unit is used to crop the tumor region according to the obtained class activation map, input it into the network, and extract local-scale image features to obtain segmented image patches. The fine-grained image feature extraction unit is used to sort all segmented image patches according to the global scale and extract fine-grained image features from the image patches with higher contribution. The output unit is used to perform cross-attention operation based on the obtained global-scale image features, local-scale image features, and fine-grained image features, fuse information from multiple scales, and output the prediction result. The selection unit is used to train the model, select the best-performing result as the final model, and save it.

[0018] Specifically, the magnetic resonance imaging data includes multimodal magnetic resonance images; the magnetic resonance images of each patient contain four commonly used modes; the four commonly used modes are T1, T2, T1C, and Flair modes.

[0019] Specifically, the data processing unit is used for:

[0020] First, the DICOM format is converted to NIFTI format; then the image is resampled; then the image is registered, matching points corresponding to the same spatial location at multiple time points one by one. Rigid registration mode is used during registration, and mutual information is used as the image similarity measure.

[0021] In addition, grayscale normalization and histogram equalization methods are used to standardize the image data in the dataset.

[0022] Specifically, the global-scale image feature extraction unit is used for:

[0023] A global visual Transformer pre-trained network and a visual Transformer network with three branches are constructed. Three-dimensional whole-brain images are divided into K fixed-size three-dimensional image patches, mapped to one-dimensional vectors of fixed length D, which serve as inputs to the global visual Transformer model. The four commonly used modalities are processed as four image channels. The model optimizes parameters using negative log-likelihood loss to extract global features. The self-attention mechanism in the global visual Transformer pre-trained network focuses attention on the tumor region. After training, the model with the best performance is saved, and the relevant class activation map obtained based on the parameter distribution of this model is used to locate the approximate tumor region.

[0024] Specifically, the fine-grained image feature extraction unit is used for:

[0025] The sorting module sorts all the segmented image patches according to the global scale output, and extracts fine-grained image features from the image patches with higher contribution. The sorting module integrates the local information of each input one-dimensional vector through a linear projection layer, and then performs global pooling to obtain a vector containing global information. Finally, the global vector and each one-dimensional vector are concatenated and input into the linear classification module to predict the contribution of each vector. Features are extracted from the corresponding patch with the highest contribution and overlapping with the region through the fine-grained branch of the Transformer.

[0026] This application constructs a deep learning model that uses medical image data and can be used for tumor survival prediction and analysis. It acquires whole-brain magnetic resonance imaging (MRI) images at multiple scales through cropping, slicing, and reconstruction, and replaces ordinary convolutional neural networks with a visual Transformer to better capture overall and fine-grained information. It combines information from multiple scales in medical images for survival analysis, further improving the model's accuracy and interpretability. Attached Figure Description

[0027] Figure 1 This is a flowchart of the brain tumor patient survival analysis method of the present invention;

[0028] Figure 2 This is a schematic diagram of the brain tumor patient survival analysis system of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0030] See Figure 1 The diagram shown is a flowchart of a preferred embodiment of the brain tumor patient survival analysis method of the present invention.

[0031] Step S1: Collect clinical information of cancer patients and establish a dataset. The clinical information includes: MRI image data, survival status, and survival time. Specifically:

[0032] This study collects MRI images, survival status, and survival time from cancer patients to create a dataset. Data for each patient includes MRI images, survival status, and survival time. MRI images include four commonly used modalities: T1, T2, T1C, and FLAIR. The patient images obtained in this step come from patient image datasets jointly collected by hospitals and TCIA (The Cancer Imaging Archive) and TCGA (The Cancer Genome Atlas). This application does not limit the size of the dataset; a larger dataset generally results in stronger generalization ability.

[0033] Step S2 involves converting, resampling, and registering the image data in the dataset.

[0034] Specifically:

[0035] DICOM (Digital Imaging and Communications in Medicine) is a set of common standard protocols for medical image processing, storage, printing, and transmission. Data acquired by medical devices is in DICOM format. First, the DICOM format is converted to NIFTI (Neuroimaging Informatics Technology Initiative) format. Then, the image is resampled to improve resolution. Next, the image is registered, mapping points at the same spatial location across multiple time points. Rigid registration is used, employing mutual information as the image similarity metric. After resampling and registration, the spatial resolution of the image is 1 mm.

[0036] Step S3: Use the whole-brain images of patients in the dataset as input to the model, and standardize the data in the dataset. Specifically:

[0037] Whole-brain images of patients in the dataset were used as model input, and the data in the dataset were standardized. Methods such as grayscale normalization and histogram equalization were used to standardize the data in the dataset.

[0038] Step S4: Divide the dataset into a training set and a test set based on the patient's clinical information. Specifically:

[0039] The dataset was divided into training and test sets based on patients' clinical information. The dataset was divided into training and test sets according to patients' gender, age, whether they received radiotherapy or chemotherapy, genotype, survival status, and survival time. A correlation coefficient was used as a threshold for this division; for example, 0.05 was used. A threshold greater than 0.05 indicates no significant difference between the training and test sets. When the correlation coefficients of all the aforementioned clinical information were greater than 0.05, the division into training and test sets was considered reasonable. Functions used to calculate correlation included, but were not limited to, Pearson correlation coefficient and Spearman correlation coefficient.

[0040] Step S5: Establish a Transformer neural network, pre-train the training set to extract global-scale image features, and obtain a class activation map. Specifically:

[0041] A Transformer neural network is established, and class activation maps are obtained by pre-training on whole-brain data in the training set. A global visual Transformer pre-trained network and a visual Transformer network with three branches are constructed. The 3D whole-brain image is divided into K fixed-size 3D image patches, which are mapped to one-dimensional vectors of fixed length D and used as input to the global visual Transformer model. The four commonly used modalities involved in this application are processed as four image channels. The model optimizes the parameters with negative log-likelihood loss to extract global features. The self-attention mechanism in the global visual Transformer focuses attention on the tumor region. After training, the model with the best performance is saved, and the relevant class activation maps obtained based on the parameter distribution of this model are used to locate the approximate tumor region.

[0042] Step S6: Based on the acquired class activation map, the tumor region is cropped, input into the network, and local-scale image features are extracted to obtain segmented image patches. Specifically:

[0043] The tumor region is cropped from the original image based on the global-scale class activation map, and local-scale image features are extracted. Since the size of the tumor and peritumoral region varies from patient to patient, the cropped region is padded with zero values. An adaptive embedding method is used to ensure that the sequence length of the input transformer for each patient is fixed at L, serving as the input to the local visual Transformer model. To simultaneously process both whole-image and region inputs and minimize training time, the pre-trained model parameters are loaded into the global branch of the multi-scale model.

[0044] Step S7: Sort all the segmented image patches according to the global scale, and extract fine-grained image features from the image patches with higher contribution. Specifically:

[0045] The sorting module ranks all the segmented image patches according to the global scale output, extracting fine-grained image features from the patches with higher contributions. The sorting module then integrates the local information of each input one-dimensional vector through a linear projection layer, followed by global pooling to obtain a vector containing global information. Finally, the global vector and each one-dimensional vector are concatenated and input into the linear classification module to predict the contribution of each vector. Furthermore, features are extracted from the corresponding patch with the highest contribution and overlapping with that region through the fine-grained branch of the Transformer.

[0046] Step S8: Based on the acquired global-scale image features, local-scale image features, and fine-grained image features, perform a cross-attention operation to fuse information from multiple scales and output the prediction result.

[0047] Specifically:

[0048] Based on the acquired global-scale image features, local-scale image features, and fine-grained image features, a cross-attention operation is performed to fuse information from multiple scales, ultimately outputting the prediction result. First, for the local-scale image features and fine-grained image features, Q and K, V are designed as attention mechanisms, respectively, and attention operations are performed. The output is the same as that performed on the global-scale features, ultimately yielding the risk value of the output result.

[0049] Step S9: Train the model, select the best-performing model as the final model, and save it. Specifically:

[0050] First, in step S91, model training and selection are performed:

[0051] During training, data augmentation methods, including but not limited to rotation, translation, scaling, and cropping, are used to improve the model's generalization ability.

[0052] The network is trained using stochastic gradient descent as the optimizer, with a negative log-likelihood function as the loss function, and weight decay is used to prevent overfitting. For the input image data, the network outputs a risk value, which is used as a predictor.

[0053] The consistency coefficient is calculated using the risk values ​​output by the network, and the consistency coefficient is used as the evaluation criterion for the model.

[0054] Next, in step S92, after the model training is completed, the KM curve (Kaplan-Meier curve), nomogram, calibration curve, etc. are plotted to evaluate the model's performance.

[0055] See Figure 2The diagram shown is a hardware architecture diagram of the brain tumor patient survival analysis system 10 of the present invention. The system includes: a dataset creation unit 101, a data processing unit 102, a partitioning unit 103, a global-scale image feature extraction unit 104, a local-scale image feature extraction unit 105, a fine-grained image feature extraction unit 106, an output unit 107, and a selection unit 108. Wherein:

[0056] The dataset creation unit 101 is used to collect clinical information from cancer patients and create a dataset. The clinical information includes: magnetic resonance imaging data, survival status, and survival time. Specifically:

[0057] The dataset creation unit 101 collects MRI images, survival status, and survival time of tumor patients to create a dataset. Data for each patient includes MRI images, survival status, and survival time. The MRI images include four commonly used modalities: T1, T2, T1C, and FLAIR. The patient images obtained in this step come from patient image datasets jointly collected by hospitals and TCIA (The Cancer Imaging Archive) and TCGA (The Cancer Genome Atlas). This application does not limit the size of the dataset; the larger the dataset, the stronger the generalization ability.

[0058] The data processing unit 102 is used for format conversion, resampling, and registration of image data in the dataset. Specifically:

[0059] DICOM (Digital Imaging and Communications in Medicine) is a set of common standard protocols for medical image processing, storage, printing, and transmission. Data acquired by medical devices is in DICOM format. The data processing unit 102 first converts the DICOM format to NIFTI (Neuroimaging Informatics Technology Initiative) format; then, it resamples the image to improve its resolution; next, it registers the image, mapping points at the same spatial location across multiple time points. Rigid registration is used during registration, employing mutual information as the image similarity metric. After resampling and registration, the spatial resolution of the image is 1 mm.

[0060] The data processing unit 102 is further configured to use whole-brain images of patients in the dataset as input to the model, and to standardize the data in the dataset. Specifically:

[0061] The data processing unit 102 takes whole-brain images of patients in the dataset as model input and standardizes the data in the dataset. Methods such as grayscale normalization and histogram equalization are used to standardize the data in the dataset.

[0062] The partitioning unit 103 is used to partition the dataset into a training set and a test set based on the patient's clinical information. Specifically:

[0063] The partitioning unit 103 divides the dataset into training and testing sets based on the patient's clinical information. The dataset is divided into training and testing sets based on clinical information such as the patient's gender, age, whether they received radiotherapy or chemotherapy, genotype, survival status, and survival time. A correlation coefficient is used as a threshold for this partitioning, for example, 0.05. A threshold greater than 0.05 indicates no significant difference between the training and testing sets. When the correlation coefficients of all the clinical information are greater than 0.05, the partitioning of the training and testing sets is considered reasonable. Functions used to calculate the correlation include, but are not limited to, Pearson correlation coefficient and Spearman correlation coefficient.

[0064] The global-scale image feature extraction unit 104 is used to establish a Transformer neural network, pre-train the training set to extract global-scale image features, and obtain a class activation map. Specifically:

[0065] The global-scale image feature extraction unit 104 establishes a Transformer neural network and pre-trains on whole-brain data in the training set to obtain class activation maps. A global visual Transformer pre-training network and a visual Transformer network with three branches are constructed. The three-dimensional whole-brain image is divided into K fixed-size three-dimensional image patches, mapped to one-dimensional vectors of fixed length D, which serve as inputs to the global visual Transformer model. The four commonly used modalities involved in this application are processed as four image channels. The model optimizes parameters using negative log-likelihood loss, extracts global features, and the self-attention mechanism in the global visual Transformer focuses attention on the tumor region. After training, the model with the best performance is saved, and the relevant class activation maps obtained based on the parameter distribution of this model are used to locate the approximate tumor region.

[0066] The local-scale image feature extraction unit 105 is used to crop the tumor region based on the acquired class activation map, input it into the network, and extract local-scale image features to obtain segmented image patches. Specifically:

[0067] The local-scale image feature extraction unit 105 crops the tumor region on the original image based on the global-scale class activation map and extracts local-scale image features. Since the size of the tumor and peritumoral region varies from patient to patient, this application pads the cropped region with zero values ​​and uses an adaptive embedding method to ensure that the sequence length of the input transformer for each patient is fixed at L, serving as the input to the local visual Transformer model. To simultaneously process both whole-image input and region input and minimize training time, the pre-trained model parameters are loaded into the global branch of the multi-scale model.

[0068] The fine-grained image feature extraction unit 106 is used to sort all the divided image patches according to the global scale and extract fine-grained image features from the image patches with higher contribution. Specifically:

[0069] The fine-grained image feature extraction unit 106 sorts all the segmented image patches according to the global scale output through the sorting module, and extracts fine-grained image features from the image patches with higher contribution. The sorting module integrates the local information of each input one-dimensional vector through a linear projection layer, and then performs global pooling to obtain a vector containing global information. Finally, the global vector and each one-dimensional vector are concatenated and input into the linear classification module to predict the contribution of each vector. Furthermore, features are extracted from the corresponding patch with the highest contribution and overlapping with that region through the fine-grained branch of the Transformer.

[0070] The output unit 107 is used to perform a cross-attention operation based on the acquired global-scale image features, local-scale image features, and fine-grained image features, fusing information from multiple scales to output a prediction result. Specifically:

[0071] The output unit 107 performs a cross-attention operation based on the acquired global-scale image features, local-scale image features, and fine-grained image features, fusing information from multiple scales to finally output the prediction result. First, for the local-scale image features and fine-grained image features, Q and K, V in the attention mechanism are designed respectively, and attention operations are performed. The output performs the same operation as the global-scale features, and finally the risk value of the output result is obtained.

[0072] The selection unit 108 is used to train the model, select the best-performing result as the final model, and save it. Specifically:

[0073] First, the selection unit 108 performs model training and selection:

[0074] During training, data augmentation methods, including but not limited to rotation, translation, scaling, and cropping, are used to improve the model's generalization ability.

[0075] The network is trained using stochastic gradient descent as the optimizer, with a negative log-likelihood function as the loss function, and weight decay is used to prevent overfitting. For the input image data, the network outputs a risk value, which is used as a predictor.

[0076] The consistency coefficient is calculated using the risk values ​​output by the network, and the consistency coefficient is used as the evaluation criterion for the model.

[0077] Next, after the model training of the selection unit 108 is completed, the KM curve (Kaplan Meier curve), nomogram, calibration curve and other evaluation methods are plotted to evaluate the model's performance.

[0078] This application is the first to combine visual Transformer networks with multi-scale information, solving the problems of existing deep learning survival prediction methods based on convolutional neural networks that require manual tumor segmentation and can only utilize single-scale information from image data, unable to utilize multi-scale information.

[0079] Although the present invention has been described with reference to the present preferred embodiments, those skilled in the art should understand that the above preferred embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for survival analysis of brain tumor patients, characterized in that, The method includes the following steps: a. Collect clinical information of cancer patients and establish a dataset, including: MRI image data, survival status and survival time; b. Convert, resample, and register the image data in the dataset; c. Use the whole-brain images of patients in the dataset as input to the model, and standardize the data in the dataset; d. Divide the dataset into training and testing sets based on the patients' clinical information; e. Establish a Transformer neural network, pre-train the training set to extract global-scale image features, and obtain a class activation map; f. Based on the obtained class activation map, the tumor region is cropped, input into the network, and local scale image features are extracted to obtain the segmented image patches; g. The sorting module sorts all the segmented image patches according to the global scale output, and extracts fine-grained image features from the image patches with higher contribution. The sorting module integrates the local information of each input one-dimensional vector through a linear projection layer, and then performs global pooling to obtain a vector containing global information. Finally, the global vector and each one-dimensional vector are concatenated and input into the linear classification module to predict the contribution of each vector. Features are extracted from the corresponding patch with the highest contribution and overlapping with the region through the fine-grained branch of the Transformer. h. Based on the acquired global-scale image features, local-scale image features, and fine-grained image features, perform a cross-attention operation to fuse information from multiple scales and output the prediction result; i. Train the model, select the best-performing result as the final model, and save it; where: The MRI image data includes multimodal MRI images; the MRI images of each patient contain four commonly used modalities; the four commonly used modalities are T1, T2, T1C, and Flair modalities; Step e specifically includes: A global visual Transformer pre-trained network and a visual Transformer network with three branches are constructed. Three-dimensional whole-brain images are divided into K fixed-size three-dimensional image patches, mapped to one-dimensional vectors of fixed length D, which serve as inputs to the global visual Transformer model. The four commonly used modalities are processed as four image channels. The model optimizes parameters using negative log-likelihood loss to extract global features. The self-attention mechanism in the global visual Transformer pre-trained network focuses attention on the tumor region. After training, the model with the best performance is saved, and the relevant class activation map obtained based on the parameter distribution of this model is used to locate the tumor region.

2. The method for analyzing the survival of brain tumor patients as described in claim 1, characterized in that: Step b includes: First, the DICOM format is converted to NIFTI format; then the image is resampled; then the image is registered, matching points at multiple time points that correspond to the same spatial location one by one. Rigid registration mode is used during registration, and mutual information is used as the image similarity measure. Step c includes: The image data in the dataset is standardized using grayscale normalization and histogram equalization.

3. A survival analysis system for brain tumor patients, characterized in that, The system includes a dataset creation unit, a data processing unit, a partitioning unit, a global-scale image feature extraction unit, a local-scale image feature extraction unit, a fine-grained image feature extraction unit, an output unit, and a selection unit, wherein: The dataset creation unit is used to collect clinical information of cancer patients and create a dataset. The clinical information includes: magnetic resonance imaging data, survival status and survival time. The data processing unit is used for format conversion, resampling, and registration of image data in the dataset; The data processing unit is also used to standardize the data in the dataset by using whole-brain images of patients in the dataset as input to the model. The partitioning unit is used to divide the dataset into a training set and a test set based on the patient's clinical information; The global scale image feature extraction unit is used to establish a Transformer neural network, pre-train the training set to extract global scale image features, and obtain a class activation map. The local scale image feature extraction unit is used to crop the tumor region based on the acquired class activation map, input it into the network, and extract local scale image features to obtain the segmented image blocks. The fine-grained image feature extraction unit is used to sort all the segmented image patches according to the global scale output by the sorting module, and extract fine-grained image features from the image patches with higher contribution. The sorting module integrates the local information of each input one-dimensional vector through a linear projection layer, and then performs global pooling to obtain a vector containing global information. Finally, the global vector and each one-dimensional vector are concatenated and input into the linear classification module to predict the contribution of each vector. Features are extracted from the corresponding patch with the highest contribution and overlapping with the region through the fine-grained branch of the Transformer. The output unit is used to perform cross-attention operation based on the acquired global-scale image features, local-scale image features and fine-grained image features, to fuse information from multiple scales and output the prediction result. The selection unit is used to train the model, select the best-performing result as the final model, and save it; wherein: The MRI image data includes multimodal MRI images; the MRI images of each patient contain four commonly used modalities; the four commonly used modalities are T1, T2, T1C, and Flair modalities; The global-scale image feature extraction unit is specifically used for: A global visual Transformer pre-trained network and a visual Transformer network with three branches are constructed. Three-dimensional whole-brain images are divided into K fixed-size three-dimensional image patches, mapped to one-dimensional vectors of fixed length D, which serve as inputs to the global visual Transformer model. The four commonly used modalities are processed as four image channels. The model optimizes parameters using negative log-likelihood loss to extract global features. The self-attention mechanism in the global visual Transformer pre-trained network focuses attention on the tumor region. After training, the model with the best performance is saved, and the relevant class activation map obtained based on the parameter distribution of this model is used to locate the tumor region.

4. The brain tumor patient survival analysis system as described in claim 3, characterized in that, The data processing unit is specifically used for: First, the DICOM format is converted to NIFTI format; then the image is resampled; then the image is registered, matching points at multiple time points that correspond to the same spatial location one by one. Rigid registration mode is used during registration, and mutual information is used as the image similarity measure. In addition, grayscale normalization and histogram equalization methods are used to standardize the image data in the dataset.

Images