Method and System for Predicting Prognostic Survival Period of Glioma Patients Based on Multimodal Imaging

Abstract

The present invention discloses a method and system for predicting the prognosis survival period of glioma patients based on multimodal images, which relates to the technical field of data processing. The key points of its technical solution are as follows: inputting multimodal magnetic resonance image data into a decision tree to determine the tumor grade of the target object; matching the cov-Split transformer model according to the tumor grade; respectively extracting the global features in the multimodal magnetic resonance image data and nuclear magnetic resonance spectroscopy data through the cov-Split transformer model; performing linearization processing on the global features, and adding the age information of the target object and then performing full connection to predict the prognosis survival period of the target object. The present invention can deeply extract the features of images, expand the receptive field at the same time, add position information, and avoid feature loss to a certain extent; and there is no need to supplement the geometric characteristics of the tumor, making the extracted features relatively lightweight and improving the prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and more specifically, to a method and system for predicting the prognosis survival period of glioma patients based on multi-modal images. Background Art

[0002] Multi-modal magnetic resonance imaging (mMRI) is the most commonly used medical image in current clinical diagnosis, and is also often used in the treatment process and postoperative follow-up of gliomas. Compared with other imaging devices, MRI can display richer and clearer details of brain structure information, and among its multiple modalities (T1w, T2w, T1wce, Flair), each modality highlights different parts of the organizational structure and can provide complementary information.

[0003] The prior art records a technique for extracting MR image features of the tumor enhancement area and non-enhancement area from the T1 enhanced weighted image, and using the tumor geometric characteristics as a supplement to the image features to achieve the prediction and analysis of the patient's survival period. However, when analyzing the prognosis survival period of glioma patients in the prior art, it mainly uses the segmented tumor region and extracts features from the tumor region again. For the method of segmenting first and then extracting features, the accuracy of feature selection depends on the accuracy of segmentation. If the segmentation is inaccurate and incomplete, it is also easy to cause errors in the extraction of classification features. In addition, the existing survival period prediction methods have the limitation of being unable to fully utilize the tumor information rich in the images, such as information loss caused by the limitation of the receptive field and position information, etc. Therefore, the tumor geometric characteristics are selected for supplementation. Generally, the number of features based on radiomics is large, and using all of them for prediction often easily leads to overfitting of the model and low accuracy of the prediction analysis results.

[0004] Therefore, how to research and design a method and system for predicting the prognosis survival period of glioma patients based on multi-modal images that can overcome the above defects is an urgent problem for us to solve currently. Summary of the Invention

[0005] To solve the deficiencies in the prior art, the purpose of the present invention is to provide a method and system for predicting the prognosis survival period of glioma patients based on multi-modal images. The cov-Split transformer model is used to extract features from multi-modal magnetic resonance imaging data and nuclear magnetic spectroscopy data, which can deeply extract the features of the image, expand the receptive field at the same time, add position information, and avoid feature loss to a certain extent; and there is no need to supplement the tumor geometric characteristics, making the extracted features relatively lightweight and improving the prediction accuracy.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions:

[0007] In a first aspect, a method for predicting the prognosis survival period of glioma patients based on multimodal imaging is provided, including the following steps:

[0008] Obtain the multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data of the target object;

[0009] After inputting the multimodal magnetic resonance imaging data into the decision tree, determine the tumor grade of the target object;

[0010] Match the corresponding pre-constructed cov-Split transformer model according to the tumor grade;

[0011] Extract the global features in the multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data respectively through the cov-Split transformer model;

[0012] Linearly process the global features in the multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data, and add the age information of the target object and then perform a full connection to predict the prognosis survival period of the target object.

[0013] The present invention uses the cov-Split transformer model to extract features from the multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data, which can deeply extract the features of the image, expand the receptive field at the same time, add the position information, and avoid the loss of features to a certain extent; combining the features in the nuclear magnetic resonance spectroscopy data with the morphological features provided by the MRI image can more effectively reflect the tumor development of the patient, without supplementing the tumor geometric characteristics, making the extracted features relatively lightweight and improving the prediction accuracy.

[0014] Further, the multimodal magnetic resonance imaging data includes four modal data of T1w, T2w, T1wce, and Flair.

[0015] Further, the decision tree divides the tumor grade according to biochemical characteristics.

[0016] Further, the biochemical characteristics include IDH, ATRX, 1p / 19q, CDKN2A / B, TERT_EGFR, H3.3G34R / V, and H3 K27M.

[0017] Further, the tumor grades include four levels of glioblastoma WHO1, WHO2, WHO3, and WHO4.

[0018] Furthermore, the cov-Split transformer model includes a 2D convolutional block, a pooling block, three Bottleneck0, one Bottleneck1, two matrix addition function blocks, a segmentation labeling module, a linear projection module, and a Transformer Encoder module.

[0019] Furthermore, the data import module is used to receive the input multi-modal magnetic resonance imaging data and nuclear magnetic spectroscopy data;

[0020] The 2D convolutional block is used to project the three-dimensional multi-modal magnetic resonance imaging data onto a two-dimensional plane;

[0021] The pooling block is used to perform pooling on the data;

[0022] The Bottleneck0 consists of three convolutional layers;

[0023] The Bottleneck1 consists of three convolutional layers;

[0024] The matrix addition function block is used to perform an addition operation on the convolution results

[0025] The segmentation labeling module is used to divide the image into nine equal-sized small squares and label them;

[0026] The linear projection module is used to perform a linear projection on the image;

[0027] The Transformer Encoder module is used to model the high-dimensional global features.

[0028] In a second aspect, a glioma patient prognosis survival period prediction system based on multi-modal images is provided, including:

[0029] A data acquisition module, which is used to acquire the multi-modal magnetic resonance imaging data and nuclear magnetic spectroscopy data of a target object;

[0030] A level division module, which is used to input the multi-modal magnetic resonance imaging data into a decision tree to determine the tumor level of the target object;

[0031] A model matching module, which is used to match a pre-constructed cov-Split transformer model according to the tumor level;

[0032] A feature extraction module, which is used to respectively extract the global features in the multi-modal magnetic resonance imaging data and nuclear magnetic spectroscopy data through the cov-Split transformer model;

[0033] A prediction analysis module is used to linearly process the global features in multi-modal magnetic resonance imaging data and nuclear magnetic spectroscopy data, and perform a fully connected operation after adding the age information of the target object to predict the prognosis survival period of the target object.

[0034] In a third aspect, a computer terminal is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for predicting the prognosis survival period of glioma patients based on multi-modal images as described in any one of the first aspects.

[0035] In a fourth aspect, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can implement the method for predicting the prognosis survival period of glioma patients based on multi-modal images as described in any one of the first aspects.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The method for predicting the prognosis survival period of glioma patients based on multi-modal images provided by the present invention uses the cov-Split transformer model to extract features from multi-modal magnetic resonance imaging data and nuclear magnetic spectroscopy data, which can deeply extract the features of the image, expand the receptive field at the same time, add the position information, and avoid the loss of features to a certain extent; and there is no need to supplement the geometric characteristics of the tumor, making the extracted features relatively lightweight and improving the prediction accuracy.

[0038] 2. The present invention uses a decision tree to logically judge the biochemical characteristics of the target object, pre-determine the tumor grade of the target alignment, and thus match the corresponding cov-Split transformer model, making the accuracy and reliability of feature extraction higher, which is beneficial to enhancing the accuracy of predicting the prognosis survival period of patients.

[0039] 3. When predicting the prognosis survival period of patients, the present invention also considers the age information, and more comprehensively considers the influencing factors of survival period prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, form a part of this application, and do not limit the embodiments of the present invention. In the drawings:

[0041] Figure 1 is the flowchart in Embodiment 1 of the present invention;

[0042] Figure 2 is the flowchart for dividing the tumor grade in Embodiment 1 of the present invention;

[0043] Figure 3It is the network structure diagram of the cov-Split transformer model in Embodiment 1 of the present invention. a is the cov part, b is the Split part, and c is the transformer part;

[0044] Figure 4 It is the system block diagram in Embodiment 2 of the present invention. Detailed implementation manners

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in combination with embodiments and drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and do not limit the present invention.

[0046] Embodiment 1: A method for predicting the prognosis survival period of glioma patients based on multimodal images, as Figure 1 shown, includes the following steps:

[0047] Step S1: Obtain multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data of the target object;

[0048] Step S2: Input the multimodal magnetic resonance imaging data into a decision tree to determine the tumor grade of the target object;

[0049] Step S3: Match the corresponding pre-constructed cov-Split transformer model according to the tumor grade;

[0050] Step S4: Extract the global features in the multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data respectively through the cov-Split transformer model;

[0051] Step S5: Linearly process the global features in the multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data, add the age information of the target object, and then perform a full connection to predict the prognosis survival period of the target object.

[0052] It should be noted that MRS is a technology that can non-invasively observe the metabolism and biochemical changes of living tissues. The nuclear magnetic resonance spectroscopy data obtained by the MRS technology can provide more accurate chemical substance components and their contents in the tumor site. Combined with the morphological features provided by the MRI images, it can more effectively reflect the tumor development of the patient.

[0053] The present invention uses a new cov-Split transformer model to extract the global features in the multimodal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data. Without the need to supplement the image geometric features, it deeply extracts more comprehensive features, making the extracted features relatively lightweight and improving the prediction accuracy.

[0054] The multi-modal magnetic resonance imaging data includes four modal data: T1w, T2w, T1wce, and Flair.

[0055] The present invention uses a decision tree to perform logical judgment on the biochemical characteristics of the target object, pre-determine the tumor grade targeted, and thus match the corresponding cov-Split transformer model, making the accuracy and reliability of feature extraction higher, which is beneficial to enhancing the accuracy of predicting the patient's prognosis survival period. Specifically, the decision tree divides the tumor grade based on one or more biochemical characteristics, and the biochemical characteristics include but are not limited to IDH, ATRX, 1p / 19q, CDKN2A / B, TERT_EGFR, H3.3G34R / V, and H3 K27M.

[0056] As Figure 2 shown, for example, by simultaneously considering the biochemical characteristics of IDH, ATRX, 1p / 19q, CDKN2A / B, TERT_EGFR, H3.3 G34R / V, and H3 K27M, the tumor grade can be divided into four levels: glioblastoma WHO1, WHO2, WHO3, and WHO4. The models for different tumor grades are trained based on the sample data of the corresponding grades.

[0057] As Figure 3 shown, the cov-Split transformer model includes 1 2D convolutional block, 1 pooling block, 3 Bottleneck0s, 1 Bottleneck1, 2 matrix addition function blocks, 1 segmentation labeling module, 1 linear projection module, and 1 Transformer Encoder module.

[0058] Among them, the data import module is used to receive the input multi-modal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data; the 2D convolutional block is used to project the three-dimensional multi-modal magnetic resonance imaging data onto two dimensions; the pooling block is used to perform pooling processing on the data, which can speed up the calculation speed and prevent overfitting; Bottleneck0 consists of three convolutional layers; Bottleneck1 consists of three convolutional layers; the matrix addition function block is used to perform addition operations on the convolutional results; the segmentation labeling module is used to divide the image into nine equal-sized small squares and label them; the linear projection module is used to perform linear projection on the image; the Transformer Encoder module is used to model the high-dimensional global features.

[0059] Generally, the input of the cov-Split transformer model is 3x224x224. After the cov part, the output is 512x28x28. Through 2D projection processing, 512x784 is obtained. The 512x784 is input into the Split part for splitting, and then input into the liner. Its position information and features are linearized and input into the transformer encoder. Finally, 512x784 is output, input into the linearization layer, and 512x1x1 is output. Age is added, and the survival period is output through full connection.

[0060] Example 2: A prognostic survival period prediction system for glioma patients based on multimodal images, which is used to implement the prognostic survival period prediction method for glioma patients based on multimodal images recorded in Example 1, as Figure 4 shown, including a data acquisition module, a level division module, a model matching module, a feature extraction module, and a prediction analysis module.

[0061] Among them, the data acquisition module is used to acquire the multimodal magnetic resonance image data and nuclear magnetic resonance spectroscopy data of the target object; the level division module is used to input the multimodal magnetic resonance image data into the decision tree to determine the tumor level of the target object; the model matching module is used to match the corresponding pre-constructed cov-Split transformer model according to the tumor level; the feature extraction module is used to extract the global features in the multimodal magnetic resonance image data and nuclear magnetic resonance spectroscopy data respectively through the cov-Split transformer model; the prediction analysis module is used to perform linearization processing on the global features in the multimodal magnetic resonance image data and nuclear magnetic resonance spectroscopy data, and perform full connection after adding the age information of the target object to predict the prognostic survival period of the target object.

[0062] Working principle: The present invention uses the cov-Split transformer model to extract features from multimodal magnetic resonance image data and nuclear magnetic resonance spectroscopy data, which can deeply extract the features of images, expand the receptive field at the same time, add position information, and avoid feature loss to a certain extent; and there is no need to supplement the geometric characteristics of tumors, making the extracted features relatively lightweight and improving the prediction accuracy; in addition, the decision tree is used to logically judge the biochemical characteristics of the target object to pre-determine the tumor level of the target alignment, so as to match the corresponding cov-Split transformer model, making the accuracy and reliability of feature extraction higher, which is beneficial to enhancing the accuracy of predicting the prognostic survival period of patients; in addition, when predicting the prognostic survival period of patients, age information is also considered, and the influencing factors of survival period prediction are considered more comprehensively.

[0063] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0064] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one or more of the processes Figure 1 or a plurality of processes and / or blocks

[0065] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the functions specified in Figure 1 one or more of the processes Figure 1 or a plurality of processes and / or blocks

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one or more of the processes Figure 1 or a plurality of processes and / or blocks

[0067] The above specific implementation manners further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above is only the specific implementation manner of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for predicting the prognosis survival period of glioma patients based on multimodal imaging, characterized in that, It includes the following steps: Obtain the multi-modal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data of the target object; After inputting the multi-modal magnetic resonance imaging data into the decision tree, determine the tumor grade of the target object; Match the corresponding pre-constructed cov-Split transformer model according to the tumor grade; Extract the global features in the multi-modal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data respectively through the cov-Split transformer model; Linearly process the global features in the multi-modal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data, add the age information of the target object, and then perform a full connection to predict the prognostic survival period of the target object; The cov-Split transformer model includes a data import module, a 2D convolution block, a pooling block, 3 Bottleneck0s, 1 Bottleneck1, 2 matrix addition function blocks, a segmentation labeling module, a linear projection module, and a Transformer Encoder module.

2. The prognostic survival period prediction method for glioma patients based on multimodal imaging according to claim 1, characterized in that, The multi-modal magnetic resonance imaging data includes four modal data of T1w, T2w, T1wce, and Flair.

3. The prognostic survival period prediction method for glioma patients based on multi-modal images according to claim 1, wherein The decision tree divides the tumor grade according to biochemical characteristics.

4. The prognostic survival period prediction method for glioma patients based on multimodal imaging according to claim 3, characterized in that, The biochemical characteristics include IDH, ATRX, 1p / 19q, CDKN2A / B, TERT_EGFR, H3.3 G34R / V, and H3 K27M.

5. The prognostic survival period prediction method for glioma patients based on multimodal imaging according to claim 1, wherein The tumor grades include four levels of glioblastoma WHO1, WHO2, WHO3, and WHO4.

6. The method for predicting the prognostic survival period of a glioma patient based on multi-modal images according to claim 1, characterized in that The data import module is used to receive the input multi-modal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data; The 2D convolution block is used to project the three-dimensional multi-modal magnetic resonance imaging data onto a two-dimensional plane and perform convolution processing on the two-dimensional nuclear magnetic resonance spectroscopy data; The pooling block is used to perform pooling processing on the data; The Bottleneck0 is composed of three convolutional layers; The Bottleneck1 is composed of three convolutional layers; The matrix addition function block is used to perform an addition operation on the convolution result; The segmentation labeling module is used to divide the processed two-dimensional image of the multi-modal magnetic resonance imaging data into nine small squares of equal size and label them; The linear projection module is used to perform a linear projection on the two-dimensional image; The Transformer Encoder module is used to model the high-dimensional global features.

7. A prognostic survival period prediction system for glioma patients based on multi-modal imaging, characterized in that, It includes: A data acquisition module for obtaining the multi-modal magnetic resonance imaging data and nuclear magnetic resonance spectroscopy data of the target object; A level division module for determining the tumor grade of the target object after inputting the multi-modal magnetic resonance imaging data into the decision tree; A model matching module for matching the corresponding pre-constructed cov-Split transformer model according to the tumor grade; A feature extraction module for separately extracting global features from multimodal magnetic resonance imaging data and nuclear magnetic spectroscopy data through a cov-Split transformer model; A prediction and analysis module for linearly processing the global features in the multimodal magnetic resonance imaging data and nuclear magnetic spectroscopy data, and performing a fully connected operation after adding the age information of the target object to predict the prognostic survival period of the target object; The cov-Split transformer model includes a data import module, a 2D convolution block, a pooling block, three Bottleneck0s, one Bottleneck1, two matrix addition function blocks, a segmentation label module, a linear projection module, and a Transformer Encoder module.

8. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for predicting the prognostic survival period of glioma patients based on multimodal images as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it can implement the method for predicting the prognostic survival period of glioma patients based on multimodal images as described in any one of claims 1-6.

Images