A meningioma automatic grading method based on multi-view image feature fusion

Abstract

The application discloses a meningioma automatic grading method based on multi-view image feature fusion, comprising the following steps: S1: for the model input end, the lesion image of Dicom format data is converted into PNG data without any image enhancement and data amplification processing, and is input into a multi-view image feature fusion model as three channels; S2: the full connection layer and the maximum pooling of ResNet are removed as the basic network of each view feature extraction, transfer learning is introduced, and a normalized attention module is added to the model to suppress the insignificant features of the related views, so that the best feature information of each view is extracted; S3: after the feature information extracted from each view is fused, the feature is re-extracted, and the interference information is further filtered, and the pre-training residual neural network model of the three different channels and the fusion network are used to compose the model, and the fusion model is trained in an end-to-end mode, so that the effect of accurately distinguishing low-grade meningioma and high-grade meningioma can be achieved.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and medical data classification and detection technology, and in particular to an automatic grading method for meningiomas based on multi-view image feature fusion. Background Technology

[0002] Meningiomas are the most common primary intracranial tumors in adults, accounting for approximately 33.8% of all intracranial tumors. According to the 2016 World Health Organization (WHO) criteria, they can be classified into grades I, II, and III. Generally, low-grade meningiomas (grade I) can achieve good survival outcomes with complete surgical resection, while high-grade meningiomas (grades II and III) are more prone to recurrence, with recurrence rates of 30%-40% and 50%-80%, respectively. The five-year overall survival rates are 78.4% and 44.0%, respectively. Detailed studies have shown that grade II and III meningiomas are associated with an increased risk of recurrence and invasiveness, requiring additional treatment strategies such as radiotherapy or chemotherapy even after complete resection. Therefore, accurate preoperative grading is crucial for meningioma patients in the clinical setting.

[0003] Magnetic resonance imaging (MRI) provides detailed information on the anatomy, cellular structure, and vascular supply of brain tumors, making it a valuable tool for disease diagnosis, treatment, and detection. Some researchers have demonstrated that MRI images can provide valuable information for the assessment of meningiomas; however, the radiographic features of different grades of meningiomas largely overlap, which can lead to misdiagnosis and inappropriate treatment strategies. Artificial intelligence may be an effective solution for accurately grading meningiomas using information provided by MRI images.

[0004] In recent research, deep learning has been successfully applied to the grading of meningiomas. Deep learning can utilize large amounts of labeled data to improve predictive performance, outperforming traditional methods in medical classification. It has also been widely applied in research aimed at achieving accurate grading of meningiomas, yielding significant results. However, no method combines MRI images from different orientations. Meningioma MRI images from different scanning orientations contain different feature information. Fusing complementary information from various views is an effective way to address the current low accuracy of meningioma grading. Therefore, there is an urgent need to develop an automatic meningioma grading method based on multi-view image feature fusion to assist physicians in preoperative pathological diagnosis of patients.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to provide an automatic grading method for meningiomas based on multi-view image feature fusion. The method uses three pre-trained residual neural network models with different pathways and a fusion network to form a model. The fusion model is trained end-to-end, which can accurately distinguish between low-grade and high-grade meningiomas. The grading accuracy is high, the application prospects are broad, and it is conducive to promotion and application.

[0007] To achieve the above objectives, the present invention provides an automatic grading method for meningiomas based on multi-view image feature fusion, comprising the following steps:

[0008] S1: For the model input, the lesion image in Dicom format is converted into PNG data without any image enhancement or data amplification processing, and then used as the input of the three channels into the multi-view image feature fusion model.

[0009] S2: Remove the fully connected layers and max pooling of ResNet and use it as the base network for feature extraction of each view. Introduce transfer learning and add a normalized attention module to the model to suppress insignificant features of related views in order to extract the best feature information of each view.

[0010] S3: After fusing the feature information extracted from each view, feature re-extraction is performed to further filter out interference information.

[0011] Preferably, in step S1, a new model input method is adopted, in which NMR data from different directions are input into the network in parallel with different input terminals, and their respective feature information is extracted at the same time.

[0012] Preferably, in step S2, feature extraction in different directions uses networks of different depths, and a NAM channel attention mechanism module is added for two of these directions. The specific steps are as follows:

[0013] S21: The branch network of the cross-sectional input adopts the shallow network ResNet 18, and the fully connected layer is removed at the end of the network;

[0014] S22: In the two input pathways of coronal and sagittal planes, a deep network ResNet 101 is used, and a NAM channel attention mechanism module is added after each layer to suppress insignificant feature information in each pathway. The fully connected layer is removed at the end of the network.

[0015] S23: After extracting features in the coronal and sagittal planes, a transformation layer consisting of 7×7 convolutional layers is used, along with a batch normalization layer and a ReLU activation function. The convolutional layers are used to reduce the dimensionality of the upper-layer features, the normalization layers are used to enhance the network's backpropagation ability, and the activation function is added to better improve the nonlinearity of the neural network.

[0016] Preferably, in step S3, a combination of two different feature fusion methods is used, and different feature re-extraction layers are designed after the two fusions. The specific steps are as follows:

[0017] S31: The feature information extracted from the three viewpoint images is fused using two different feature fusion methods: concat and add. The concat method concatenates the feature information from the three views without changing any individual features of each view. The number of feature channels after fusion is three times that of a single view. The add method adds the corresponding elements of the feature information from the three views, resulting in a feature channel count equal to that of a single view. The expressions after convolution for both fusion methods are as follows:

[0018]

[0019]

[0020] In the formula: Fc represents the output of the concat method, Fa represents the output of the add method, Xi, Yi and Zi are the inputs of each parallel path, K represents the convolution kernel, and c represents the number of channels of the input feature;

[0021] S32: After concat and add, feature re-extraction layer A and feature re-extraction layer B are designed respectively. The feature re-extraction layers are composed of 1×1 convolutional layer, 3×3 convolutional layer and 1×1 convolutional layer in series. The number of convolutional kernels of A are 1024, 1024 and 512 respectively, and the number of convolutional kernels of B are 256, 256 and 512 respectively. The function of the feature re-extraction layer is to filter the fused high-level features again to avoid the influence of interfering features.

[0022] S33: After different feature extraction layers A and B, the extracted features are concatenated using the concat method, and then hierarchical detection is performed through two fully connected layers.

[0023] The present invention provides an automatic grading method for meningiomas based on multi-view image feature fusion, which has the following beneficial effects.

[0024] This invention can achieve the integration of meningioma feature information from three views without image enhancement or data amplification, and clearly distinguish between low-grade meningioma (LGM) and high-grade meningioma (HGM) on T1WI enhanced images. The grading accuracy is high, which can assist doctors in the preoperative pathological diagnosis of patients. Attached Figure Description

[0025] Figure 1 The first column shows T1WI enhanced images of LGM in three directions, and the second column shows T1WI enhanced images of HGM in three directions.

[0026] Figure 2 Here is a diagram of the BasicBlock structure;

[0027] Figure 3 Diagram of the Bottleneck Block structure;

[0028] Figure 4 This is a diagram of the NAM channel attention structure.

[0029] Figure 5 This is a diagram of the transformation layer structure;

[0030] Figure 6 This is a diagram of the feature extraction layer structure.

[0031] Figure 7 Diagram of a multi-view image feature fusion model;

[0032] Figure 8 A flowchart of an automatic grading method for meningiomas based on multi-view image feature fusion provided by the present invention;

[0033] Figure 9 ROC curve;

[0034] Figure 10 This is a confusion matrix diagram. Detailed Implementation

[0035] The present invention will be further described below with reference to specific embodiments and accompanying drawings to help understand the content of the present invention.

[0036] This invention provides an automatic grading method for meningiomas based on multi-view image feature fusion, comprising the following steps:

[0037] S1: First, for the model input, the lesion images in DICOM format are converted into PNG data. Then, images from different directions are input into the multi-view image feature fusion model as three pathways. PNG is lossless compression, which can preserve the features of the original format to the greatest extent. Examples of LGM T1WI enhanced images and HGM T1WI enhanced images are shown below. Figure 1 As shown.

[0038] S21: The branch network for the cross-sectional input uses the shallow network ResNet 18. ResNet not only offers a variety of choices in network depth but is also used as a baseline in most classification tasks, making it a recognized classic network in the field of deep learning. ResNet 18 uses BasicBlock, and the BasicBlock structure diagram is shown below. Figure 2 As shown, the fully connected layer is removed at the end of the network.

[0039] S22: In the coronal and sagittal input pathways, a deep network ResNet 101 is used. ResNet 101 uses the Bottleneck Block, and the Bottleneck Block structure diagram is shown below. Figure 3 As shown, the Bottleneck structure replaces two 3x3 convolutional layers with a single 1x1 convolutional layer, and then adds another 1x1 convolutional layer to the existing 3x3 convolutional layer. It cleverly reduces or expands the feature map dimension through the 1x1 convolution. The middle 3x3 convolutional layer first reduces computation under a dimensionality-reducing 1x1 convolutional layer, and then restores the original dimension under another 1x1 convolutional layer, maintaining model accuracy while reducing network parameters and computational cost, thus saving computation time. Furthermore, NAM channel attention is added after each layer to suppress insignificant features in their respective channels. Since ResNet generates a large number of channels (e.g., ResNet101 can have up to 2048 channels), the addition of channel attention is crucial. The dataset contains many non-lesion regions, which can effectively suppress insignificant features in the channels. The NAM channel attention structure diagram is shown below. Figure 4 As shown. Then, at the end of the network, the fully connected layer is removed. The relevant expression for NAM channel attention is as follows:

[0040] M c =sigmoid(W γ (BN(F)))

[0041]

[0042]

[0043] m B and σ B These are the mean and standard deviation of the mini-batch, respectively; γ and β are trainable affine transformation parameters (scaling and shifting). Where γ is the scaling factor for each channel, and W... γ M represents the weight in the weight channel. c This represents the output feature map.

[0044] S23: After extracting features in the coronal and sagittal planes, a transformation layer is used to change the channel size of the image for subsequent add operations, changing the output shape from [2048×16×16] to [512×16×16]. The transformation layer consists of 7×7 convolutional layers, with added batch normalization layers and ReLU activation functions. The transformation layer structure diagram is shown below. Figure 5 As shown, convolutional layers are used to reduce the dimensionality of features from the upper layers, normalization layers are used to enhance the network's backpropagation ability, and activation functions are added to better improve the nonlinearity of the neural network.

[0045] S31: After fusing the feature information extracted from each view, features are extracted again. Two different feature fusion methods, `cancat` and `add`, are used for the feature information extracted from the three views. The `concat` method concatenates the feature information from the three views without changing any individual features of each view; the number of feature channels after fusion is three times that of a single view. The `add` method adds the corresponding elements of the feature information from the three views, resulting in a feature channel count equal to that of a single view. The expressions for convolution after both fusion methods are as follows:

[0046]

[0047]

[0048] In the formula: Fc represents the output of the concat method, Fa represents the output of the add method, Xi, Yi and Zi are the inputs of each parallel path, K represents the convolution kernel, and c represents the number of channels of the input feature.

[0049] S32: Further filter out interfering information. After concat and add, design feature re-extraction layer A and feature re-extraction layer B respectively. The feature re-extraction layer structure diagram is shown below. Figure 6 As shown, each feature re-extraction layer consists of a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in series. The number of convolutional kernels in layer A is 1024, 1024, and 512, respectively; the number of convolutional kernels in layer B is 256, 256, and 512, respectively. The function of the feature re-extraction layer is to filter the fused high-level features again to avoid the influence of interfering features.

[0050] S33: After passing through different feature re-extraction layers A and B, the re-extracted features are concatenated using the concat method. Then, two fully connected layers are used for hierarchical detection. The multi-view image feature fusion model diagram is shown below. Figure 7 As shown.

[0051] like Figure 8 The diagram shown is a flowchart of an automatic grading method for meningiomas based on multi-view image feature fusion provided by the present invention. The working principle is as follows:

[0052] For T1WI enhanced images, the format conversion operation in S1 is performed first, and then the image is input into a three-parallel channel branch. Figure 7The multi-view image feature fusion model shown is used for training and hierarchical prediction of image data. Improvements include parallel input of three views, the introduction of transfer learning and NAM channel attention, the design of transformation layers and feature re-extraction layers, and the design of multi-view image feature fusion methods, covering the content of S2 and S3. Therefore, hierarchical classification models for LGM and HGM of T1WI enhanced images can be obtained to assist clinicians in preoperative diagnosis using MRI images.

[0053] To further illustrate the technical effects of this invention, experimental results are presented below: The framework used in the experiment was PyTorch, and 100 epoches were trained on an Nvidia Tesla V100 GPU. Furthermore, the Adamax optimizer was used for optimization, with an initial learning rate set to 10. -3 It adopts the StepLR mechanism, which sets the learning rate decay according to the epoch, that is, the learning rate is adjusted at equal intervals, with an adjustment factor of 0.1 and an adjustment epoch interval of 10.

[0054] Following the experimental procedure, three-fold cross-validation was performed on the MRI findings of 180 meningioma patients, including 122 cases of LGM and 58 cases of HGM. The average grading accuracy was 0.9559, sensitivity was 0.9924, specificity was 0.8803, and AUC was 0.9763. The specific experimental evaluation indicators for each fold are shown in Table 1. The ROC curve and confusion matrix of this experiment are shown in the figure. Figure 9 and Figure 10 As shown.

[0055] Table 1 Experimental Evaluation Indicators

[0056]

[0057] Therefore, it can be seen that the T1WI enhanced images of the present invention have high diagnostic and grading accuracy for meningiomas, and have achieved relatively ideal results in small sample experiments. The present invention can greatly help doctors in the preoperative imaging diagnosis of meningiomas and has significant clinical significance.

[0058] This invention can achieve the integration of meningioma feature information from three views without image enhancement or data amplification, and clearly distinguish between low-grade meningioma (LGM) and high-grade meningioma (HGM) on T1WI enhanced images. The grading accuracy is high, which can assist doctors in the preoperative pathological diagnosis of patients.

[0059] This article uses specific examples to illustrate the inventive concept in detail. The description of the above embodiments is only for the purpose of helping to understand the core idea of ​​the present invention. It should be noted that any obvious modifications, equivalent substitutions or other improvements made by those skilled in the art without departing from the inventive concept should be included within the protection scope of the present invention.

Claims

1. An automatic grading method for meningiomas based on multi-view image feature fusion, characterized in that, Includes the following steps: S1: For the model input, the lesion image in Dicom format is converted into PNG data without any image enhancement or data amplification processing, and then used as the input of the three channels into the multi-view image feature fusion model. S2: Remove the fully connected layers and max pooling of ResNet and use it as the base network for feature extraction of each view. Introduce transfer learning and add a normalized attention module to the model to suppress insignificant features of related views in order to extract the best feature information for each view. Different depths of networks are used for feature extraction in different directions, and a NAM channel attention mechanism module is added for two of the directions. The specific steps are as follows: S21: The branch network of the cross-sectional input adopts the shallow network ResNet 18, and the fully connected layer is removed at the end of the network; S22: In the two input pathways of coronal and sagittal planes, a deep network ResNet 101 is used, and a NAM channel attention mechanism module is added after each layer to suppress insignificant feature information in each pathway. The fully connected layer is removed at the end of the network. S23: After extracting features in the coronal and sagittal planes, a transformation layer consisting of 7×7 convolutional layers is used, along with a batch normalization layer and a ReLU activation function. The convolutional layers are used to reduce the dimensionality of the upper-layer features, the normalization layers are used to enhance the network's backpropagation ability, and the activation function is added to better improve the nonlinearity of the neural network. S3: After fusing the feature information extracted from each view, feature re-extraction is performed to further filter out interference information; a combination of two different feature fusion methods is used, and different feature re-extraction layers are designed after the two fusions. The specific steps are as follows: S31: The feature information extracted from the three viewpoint images is fused using two different feature fusion methods: concat and add. The concat method concatenates the feature information from the three views without changing any individual features of each view. The number of feature channels after fusion is three times that of a single view. The add method adds the corresponding elements of the feature information from the three views, resulting in a feature channel count equal to that of a single view. The expressions after convolution for both fusion methods are as follows: ； ； In the formula: Fc This represents the output of the concat method. Fa This represents the output of the add method. Xi , Yi and Zi These are the inputs for each parallel path, K Represents the convolution kernel. c This represents the number of channels for the input feature; S32: After concat and add, design feature re-extraction layer A and feature re-extraction layer B respectively. The feature re-extraction layers are composed of 1×1 convolutional layer, 3×3 convolutional layer and 1×1 convolutional layer in series. The number of convolutional kernels of A are 1024, 1024 and 512 respectively, and the number of convolutional kernels of B are 256, 256 and 512 respectively. S33: After different feature extraction layers A and B, the extracted features are concatenated using the concat method, and then hierarchical detection is performed through two fully connected layers.

2. The automatic grading method for meningiomas based on multi-view image feature fusion according to claim 1, characterized in that, In S1, a new model input method is adopted, in which NMR data from different directions are input into the network in parallel with different input terminals, and their respective feature information is extracted at the same time.

Images