Undersampling brain tumor classification method based on Transform and multi-stage fusion

Abstract

The invention discloses an under-sampling brain tumor classification method based on Transform and multi-stage fusion, and belongs to the field of brain MRI (Magnetic Resonance Imaging). Acquiring full-sampling multi-coil k space data of the brain; the method comprises the following steps of: performing multiple operations on full-sampling multi-coil k-space data through coil sensitivity decoding and coding, a two-dimensional Fourier transform operator, a two-dimensional inverse Fourier transform operator and an under-sampling operator for selecting an acquisition position from a whole multi-coil k-space grid to obtain under-sampling multi-coil k-space data; the brain tumors are marked to obtain labels of all categories, and finally, the undersampled multi-coil k-space data and the corresponding labels of all categories form a training set, a verification set and a test set; the method comprises the following steps of: designing a joint network model which is expanded based on a Transform network and has multiple scales and multiple stages; constructing a joint loss function of the joint network model; solving the optimal parameters of the joint network model by using the training set, the verification set and the joint loss function; and for the test set, performing brain tumor prediction by using the optimal parameters of the combined network model.

Description

Technical Field

[0001] The present invention belongs to the field of brain MRI imaging, and in particular relates to an under-sampling brain tumor classification method based on Transformer and multi-stage fusion with a joint learning mechanism based on shared parameters. Background Art

[0002] Magnetic resonance imaging (MRI) is an imaging technology that does not emit ionizing radiation and can generate high-resolution images. However, the further development of MRI technology is currently restricted by two major factors. On the one hand, the long k-space data acquisition time often causes discomfort to patients and may lead to motion artifacts. On the other hand, in research fields such as brain tumor classification, research usually relies on fully sampled brain MRI images, which not only requires a lot of time for manual annotation, but also often ignores the potential impact of the image reconstruction process on subsequent analysis tasks. Summary of the invention

[0003] The purpose of the present invention is to address the problem of long MRI acquisition time, which is usually relied on for classification of fully sampled MR data. An under-sampled brain tumor classification method based on Transformer and multi-stage fusion is introduced. This method is specially designed for brain tumor classification from under-sampled MRI images, and by adopting a joint learning strategy with shared parameters, the reconstruction task and the classification task are cleverly aggregated, which significantly improves the brain tumor classification performance in the under-sampled MRI scenario.

[0004] To achieve the above object, the present invention comprises the following steps:

[0005] 1) Obtaining full-sampled multi-coil k-space data of the brain, performing multiple operations on the full-sampled multi-coil k-space data through coil sensitivity decoding and encoding, a two-dimensional Fourier transform operator, a two-dimensional Fourier inverse transform operator, and an undersampling operator that selects acquisition positions from the entire multi-coil k-space grid to obtain undersampled multi-coil k-space data, annotating brain tumors to obtain labels of various categories, and finally forming a training set, a validation set, and a test set with the undersampled multi-coil k-space data and the corresponding labels of various categories;

[0006] 2) Design a multi-scale, multi-stage joint network model based on the Transformer network;

[0007] 3) Constructing a joint loss function for the joint network model;

[0008] 4) using the training set and validation set obtained in step 1) and the joint loss function obtained in step 3), solving the optimal parameters of the joint network model;

[0009] 5) For the test set obtained in step 1), the optimal parameters of the joint network model obtained in step 4) are used to predict brain tumors.

[0010] In the step 1), the full-sampled multi-coil k-space data of the brain is obtained, and multiple operations are performed on the full-sampled multi-coil k-space data through coil sensitivity decoding and encoding, a two-dimensional Fourier transform operator, a two-dimensional Fourier inverse transform operator, and an undersampling operator that selects an acquisition position from the entire multi-coil k-space grid to obtain undersampled multi-coil k-space data, and brain tumors are annotated to obtain various category labels. Finally, the specific method of forming a training set, a validation set, and a test set with the undersampled multi-coil k-space data and the corresponding various category labels is as follows:

[0011] The brain's full-sample multi-coil k-space data is obtained from the magnetic resonance imaging instrument, and the S -1 and the two-dimensional inverse Fourier transform operator F -1 Get fully sampled multi-coil image data Represents the complex field, N = N h ×N w Represents the number of pixels in a two-dimensional image, N h and N w are the height and width of the two-dimensional image respectively; then the under-sampling multi-coil k-space data is obtained through the under-sampling operator P, coil sensitivity encoding S and two-dimensional Fourier transform operator F Defined as Y = PFSx,c th The undersampled k-space data of the coil is expressed as c=1,2,...,C,M represents the number of under-sampling points of single coil k-space data, M<<N, and finally forms the training set, validation set and test set with the corresponding category labels.

[0012] In the step 2), the joint network model is composed of a reconstruction model and a segmentation model, and a joint learning strategy is used to classify the undersampled input tumor image to classify the specific tumor type. The reconstruction model and the segmentation model share the same U-shaped network architecture. The U-shaped network architecture introduces a novel multi-stage fusion module MSF, which enhances feature representation by using different information levels between decoder stages for complementary fusion. The interaction module IA connects the shallow encoder features with the deep decoder features, and adopts a residual structure and deep separable convolution to promote better information exchange. In addition, a Swin-conv module SC is introduced to combine convolution and self-attention hierarchically for feature learning, and the hierarchical design of local and global feature affinities ensures effective representation learning and subsequent feature fusion.

[0013] A.Swin-conv module SC:

[0014] The input feature tensor X undergoes a 1×1 convolution and is then divided into two independent feature tensors X1 and X2; these two independent tensors X1 and X2 enter the multi-head self-attention Swin Transformer based on the shift window and two 3×3 convolution modules respectively;

[0015] B. Interaction Module IA:

[0016] The IA module extracts features from the input tensor through 1×1 convolution, depth-wise separable convolution, Gaussian error linear unit function, and residual connection operations;

[0017] C. Multi-stage fusion module MSF:

[0018] The MSF module consists of three complementary information fusion CIF modules based on multi-scale convolution kernels, which are used to efficiently integrate the image features of two inputs. The CIF module consists of 1×1 convolution, 3×3 convolution, 5×5 convolution and sigmoid function.

[0019] D. Joint learning strategy:

[0020] This method integrates MRI reconstruction and classification into a unified framework, using a shared parameter approach that enables the model to exploit detailed information learned during reconstruction to improve the classification performance of undersampled MRI data.

[0021] The definition of the joint loss function in step 3) consists of two parts, including the L1 norm and the cross entropy function; the L1 norm is one of the most commonly used indicators for evaluating the performance of medical reconstruction based on deep learning, and is expressed in a mathematical formula as follows:

[0022]

[0023] in represents the reconstructed image, x i represents the original image, T represents the number of images in the dataset;

[0024] The cross entropy function is used to measure the similarity between two distributions and is expressed as follows:

[0025]

[0026] in, Represents the classification prediction value, y i Represents the true label value;

[0027] The joint loss function L of the network Total is the loss function of the sum of the L1 norm and the cross entropy function:

[0028] L Total =L1+λLseg

[0029] Among them, λ is the joint loss weight parameter.

[0030] In step 4), the optimal parameters of the joint network model are solved by using the Adam optimizer in deep learning, and the network is trained and verified using the training set and verification set generated in step 1), and the joint loss function L in step 3) is minimized. Total The optimal parameter set θ° is obtained, where θ° represents the set of network parameters when the undersampling classification effect of the joint network model is optimal.

[0031] In step 5), the network prediction of brain tumors is performed using the test set obtained in step 1) and the optimal parameters of the joint network model obtained in step 4). The network prediction process can be expressed as:

[0032]

[0033] Among them, f overall (·) represents the prediction process of the entire network.

[0034] Compared with the prior art, the present invention has the following outstanding technical effects:

[0035] The present invention combines convolutional neural networks (CNN) with the Transformer architecture to achieve deep integration and utilization of multiple information. In addition, the present invention uniquely utilizes the outputs of different decoder stages to perform interactive feature fusion, thereby greatly improving the brain tumor classification performance in undersampled MRI scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram of the overall network architecture of the present invention;

[0037] Figure 2 A confusion matrix diagram of the present invention on a brain tumor dataset;

[0038] Figure 3 It is the ROC curve diagram of the present invention on the brain tumor data set. DETAILED DESCRIPTION

[0039] The following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example 1: Figure 1-3 As shown, the embodiment of the present invention includes the following steps:

[0041] Step 1: Obtain the full-sampled multi-coil k-space data of the brain, perform multiple operations on the full-sampled multi-coil k-space data through coil sensitivity decoding and encoding, two-dimensional Fourier transform operator, two-dimensional Fourier inverse transform operator, and undersampling operator that selects acquisition positions from the entire multi-coil k-space grid to obtain undersampled multi-coil k-space data, and annotate brain tumors to obtain labels of various categories. Finally, the undersampled multi-coil k-space data and the corresponding labels of various categories are combined into training sets, validation sets, and test sets. The specific method is as follows:

[0042] The brain's full-sample multi-coil k-space data is obtained from the magnetic resonance imaging instrument, and the S -1 and the two-dimensional inverse Fourier transform operator F -1 Get fully sampled multi-coil image data Represents the complex field, N = N h ×N w Represents the number of pixels in a two-dimensional image, N h and N w are the height and width of the two-dimensional image respectively; then the under-sampling multi-coil k-space data is obtained through the under-sampling operator P, coil sensitivity encoding S and two-dimensional Fourier transform operator F Defined as Y = PFSx,c th The undersampled k-space data of the coil is expressed as c=1,2,...,C,M represents the number of under-sampling points of single coil k-space data, M<<N, and finally forms the training set, validation set and test set with the corresponding category labels.

[0043] Step 2: The joint network model consists of a reconstruction model and a segmentation model, and a joint learning strategy is used to classify the undersampled input tumor images and classify the specific tumor types. The reconstruction model and the segmentation model share the same U-shaped network architecture. The U-shaped network architecture introduces a novel multi-stage fusion module MSF, which enhances feature representation by utilizing different information levels between decoder stages for complementary fusion. The interaction module IA connects the shallow encoder features with the deep decoder features, and adopts residual structure and deep separable convolution to promote better information exchange. In addition, the Swin-conv module SC is introduced to combine convolution and self-attention hierarchically for feature learning. The hierarchical design of local and global feature affinities ensures effective representation learning and subsequent feature fusion.

[0044] A.Swin-conv module SC:

[0045] The input feature tensor X undergoes a 1×1 convolution and is then divided into two independent feature tensors X1 and X2; these two independent tensors X1 and X2 enter the multi-head self-attention Swin Transformer based on the shift window and two 3×3 convolution modules respectively;

[0046] B. Interaction Module IA:

[0047] The IA module extracts features from the input tensor through 1×1 convolution, depth-wise separable convolution, Gaussian error linear unit function, and residual connection operations;

[0048] C. Multi-stage fusion module MSF:

[0049] The MSF module consists of three complementary information fusion CIF modules based on multi-scale convolution kernels, which are used to efficiently integrate the image features of two inputs. The CIF module consists of 1×1 convolution, 3×3 convolution, 5×5 convolution and sigmoid function.

[0050] D. Joint learning strategy:

[0051] This method integrates MRI reconstruction and classification into a unified framework, using a shared parameter approach that enables the model to exploit detailed information learned during reconstruction to improve the classification performance of undersampled MRI data.

[0052] Step 3: The joint loss function definition consists of two parts, including the L1 norm and the cross entropy function; the L1 norm is one of the most commonly used indicators for evaluating the performance of medical reconstruction based on deep learning, and is expressed in mathematical formula as follows:

[0053]

[0054] in represents the reconstructed image, x i represents the original image, T represents the number of images in the dataset;

[0055] The cross entropy function is used to measure the similarity between two distributions and is expressed as follows:

[0056]

[0057] in, Represents the classification prediction value, y i Represents the true label value;

[0058] The joint loss function L of the network Total is the loss function of the sum of the L1 norm and the cross entropy function:

[0059] L Total =L1+λL seg

[0060] Among them, λ is the joint loss weight parameter.

[0061] Step 4: Solve the optimal parameters of the joint network model using the Adam optimizer in deep learning, use the training set and validation set generated in step 1) to train and validate the network, and minimize the joint loss function L in step 3) Total The optimal parameter set θ° is obtained, where θ° represents the set of network parameters when the undersampling classification effect of the joint network model is optimal.

[0062] Step 5: Using the test set obtained in step 1), the optimal parameters of the joint network model obtained in step 4) are used to perform network prediction of brain tumors. The network prediction process can be expressed as:

[0063]

[0064] Among them, f overall (·) represents the prediction process of the entire network. Figure 1 middle The meaning is the softmax activation function.

[0065] MobileNetV2, AG-CNN and the method proposed in the present invention were tested on the test data set, and the classification results obtained are shown in Table 1. Compared with the comparison methods MobileNetV2 and AG-CNN, the average classification accuracy of the method proposed in the present invention was improved by 7.84% and 11.96%, respectively. The present invention has greatly improved the accuracy of under-sampled brain tumor classification and has great potential in disease prediction.

[0066] Table 1 Comparison of classification effects of different classification methods

[0067]

[0068] Figure 2 It is a confusion matrix used to evaluate the performance of the classification model, especially its prediction accuracy between different categories. The "True label" on the left represents the actual category, that is, the real classification label of the method proposed in the test data. The "Predicted label" at the bottom represents the prediction result given by the method proposed in the present invention. This figure shows the classification results of the method proposed in the present invention for three types of brain tumors, including meningioma, glioma and pituitary tumor. Figure 2 Each number in represents the model's prediction of different categories. For example, in actual meningioma cases, 109 were correctly predicted as meningiomas, 5 were incorrectly predicted as gliomas, and 4 were incorrectly predicted as pituitary tumors.

[0069] Figure 3 It is a ROC (Receiver Operating Characteristic) curve of a multi-class classification model, which is used to evaluate the performance of the method proposed in the present invention on different categories. The ROC curve is a commonly used evaluation tool, which shows the classification effect of the method proposed in the present invention by plotting the relationship between the True Positive Rate (TPR) and the False Positive Rate (FPR) under different classification thresholds. Figure 3 The horizontal axis represents the false positive rate (False Positive Rate, FPR), which represents the proportion of negative classes mistakenly predicted as positive classes, that is, the prediction errors. Ideally, the false positive rate should be close to zero, indicating that the method proposed in the present invention will hardly misjudge the actual negative class (such as patients without diseases) as positive class (such as patients with diseases). The vertical axis represents the true positive rate (True Positive Rate, TPR), also known as the recall rate or sensitivity, which represents the proportion of positive classes correctly predicted. Ideally, the true positive rate should be close to 1, indicating that the method proposed in the present invention can accurately identify positive classes (such as patients with diseases).

[0070] Here Figure 3 In the data, the method proposed in the present invention classifies three categories: Meningioma (meningioma), Glioma (glioma) and Pituitary (pituitary tumor). The corresponding ROC curves are: the ROC curve of Meningioma (meningioma), the area under the curve (AUC, Area Under Curve) is 0.990, close to 1, indicating that the method proposed in the present invention has a good recognition ability for this category; the ROC curve of Glioma (glioma), its AUC value is 0.996, which also shows that the method proposed in the present invention has good performance in identifying this category; the ROC curve of Pituitary (pituitary tumor), the AUC value is 0.999, which is almost a perfect classification performance, indicating that the method proposed in the present invention has almost no errors in identifying this category. In addition, Figure 3There are also micro-average ROC curves and macro-average ROC curves. The micro-average ROC curve is the ROC curve calculated by comprehensive statistics of all individual samples in each category, representing the overall average performance of the method proposed in the present invention. The macro-average ROC curve is the average value of the ROC curve of each category calculated separately. The AUC values ​​of the two average curves are both 0.996, indicating that the overall performance of the method proposed in the present invention in all categories is very outstanding.

[0071] Through analysis, it can be concluded that the method proposed in the present invention has a very good classification effect on three types of brain tumors (meningioma, glioma, pituitary tumor), and the AUC values ​​of almost all categories are close to 1, indicating that the method proposed in the present invention has a very strong distinguishing ability.

[0072] The present invention combines convolutional neural networks (CNN) with the Transformer architecture to achieve deep integration and utilization of multiple information. In addition, the present invention uniquely utilizes the outputs of different decoder stages to perform interactive feature fusion, thereby greatly improving the brain tumor classification performance in undersampled MRI scenarios.

[0073] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technology in the field within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. An undersampling brain tumor classification method based on Transformer and multi-stage fusion, Features: The steps include: 1) Obtaining full-sampled multi-coil k-space data of the brain, performing multiple operations on the full-sampled multi-coil k-space data through coil sensitivity decoding and encoding, a two-dimensional Fourier transform operator, a two-dimensional Fourier inverse transform operator, and an undersampling operator that selects acquisition positions from the entire multi-coil k-space grid to obtain undersampled multi-coil k-space data, annotating brain tumors to obtain labels of various categories, and finally forming a training set, a validation set, and a test set with the undersampled multi-coil k-space data and the corresponding labels of various categories; 2) Design a multi-scale, multi-stage joint network model based on the Transformer network; 3) Construct the joint loss function L of the joint network model Total ; 4) using the training set and validation set obtained in step 1) and the joint loss function obtained in step 3), solving the optimal parameters of the joint network model; 5) For the test set obtained in step 1), the optimal parameters of the joint network model obtained in step 4) are used to predict brain tumors.

2. The undersampling brain tumor classification method based on Transformer and multi-stage fusion according to claim 1, Features: The specific method of step 1) is: The brain's full-sample multi-coil k-space data is obtained from the magnetic resonance imaging instrument, and the S -1 and the two-dimensional inverse Fourier transform operator F -1 Get fully sampled multi-coil image data Represents the complex field, N = N h ×N w Represents the number of pixels in a two-dimensional image, N h and N w are the height and width of the two-dimensional image respectively; then the under-sampling multi-coil k-space data is obtained through the under-sampling operator P, coil sensitivity encoding S and two-dimensional Fourier transform operator F Defined as Y = PFSx,c th The undersampled k-space data of the coil is expressed as c=1,2,...,C,M represents the number of under-sampling points of single coil k-space data, M<<N, and finally forms the training set, validation set and test set with the corresponding category labels.

3. The undersampling brain tumor classification method based on Transformer and multi-stage fusion according to claim 1, It is characterized in that In the step 2), the joint network model is composed of a reconstruction model and a segmentation model, and a joint learning strategy is used to classify the undersampled input tumor image to classify the specific tumor type. The reconstruction model and the segmentation model share the same U-shaped network architecture. The U-shaped network architecture introduces a novel multi-stage fusion module MSF, which enhances feature representation by using different information levels between decoder stages for complementary fusion. The interaction module IA connects the shallow encoder features with the deep decoder features, and adopts a residual structure and deep separable convolution to promote better information exchange. In addition, a Swin-conv module SC is introduced to combine convolution and self-attention hierarchically for feature learning, and the hierarchical design of local and global feature affinities ensures effective representation learning and subsequent feature fusion. A.Swin-conv module SC: The input feature tensor X undergoes a 1×1 convolution and is then split into two independent feature tensors X 1 and X 2 ; These two independent tensors X 1 and X 2 Enter the multi-head self-attention Swin Transformer based on shift window and two 3×3 convolution modules respectively; B. Interaction Module IA: The IA module extracts features from the input tensor through 1×1 convolution, depth-wise separable convolution, Gaussian error linear unit function, and residual connection operations; C. Multi-stage fusion module MSF: The MSF module consists of three complementary information fusion CIF modules based on multi-scale convolution kernels, which are used to efficiently integrate the image features of two inputs. The CIF module consists of 1×1 convolution, 3×3 convolution, 5×5 convolution and sigmoid function. D. Joint learning strategy: We integrate MRI reconstruction and classification into a unified framework with a shared parameter approach, which enables the model to exploit detailed information learned during reconstruction to improve the classification performance of undersampled MRI data.

4. The undersampling brain tumor classification method based on Transformer and multi-stage fusion according to claim 1, It is characterized in that The definition of the joint loss function in step 3) consists of two parts, including L 1 Norm and cross entropy function; L 1 The norm is one of the most commonly used indicators for evaluating the performance of deep learning-based medical reconstruction, which can be expressed mathematically as follows: in represents the reconstructed image, x i represents the original image, T represents the number of images in the dataset; The cross entropy function is used to measure the similarity between two distributions and is expressed as follows: in, Represents the classification prediction value, y i Represents the true label value; The joint loss function L of the network Total YesL 1 The loss function is the sum of the norm and cross entropy function: THE Total =L 1 +λL seg Among them, λ is the joint loss weight parameter.

5. The undersampling brain tumor classification method based on Transformer and multi-stage fusion according to claim 1, It is characterized in that In step 4), the optimal parameters of the joint network model are solved by using the Adam optimizer in deep learning, and the network is trained and verified using the training set and verification set generated in step 1). The joint loss function L in step 3) is minimized. Total Get the optimal parameter set θ o ,θ o Represents the set of network parameters when the undersampling classification effect of the joint network model is optimal.

6. The undersampling brain tumor classification method based on Transformer and multi-stage fusion according to claim 5, It is characterized in that Using the test set obtained in step 1), the optimal parameters of the joint network model obtained in step 4) are used to perform network prediction of brain tumors. The network prediction process can be expressed as: Among them, f overall (·) represents the prediction process of the entire network.

Images