Method for establishing brain glioma grading hybrid network based on multi-view feature fusion

Abstract

The application discloses a method for establishing a brain glioma grading hybrid network based on multi-view feature fusion, which comprises four steps of data preprocessing, feature extraction, multi-view feature fusion and grading processing of the obtained features through a classifier; the method can balance global receptive field and induction bias, combines global and local multi-scale and multi-granularity features, balances global receptive field and induction bias at the same time, and enhances feature extraction capability; by constructing different view data sources (interlayer view, scanning plane view and modality view) and designing a novel feature fusion module, the network can fully and effectively utilize key information provided by different scanning planes and different modalities of MRI, including interlayer, spatial and multi-modality information, and realizes comprehensive cognition of brain glioma.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging technology, specifically, it relates to a method for constructing a hybrid network for glioma grading based on multi-view feature fusion. Background Technology

[0002] Traditional imaging diagnostic methods rely heavily on the doctor's personal expertise and clinical experience, which cannot fully leverage the advantages of magnetic resonance imaging (MRI). They are time-consuming, labor-intensive, highly subjective, and prone to diagnostic bias and misdiagnosis.

[0003] Existing convolutional neural network (CNN) methods based on single-modal MRI analysis only utilize single-modal MRI for feature extraction and modeling, which makes it impossible to fully capture the heterogeneity and complexity within tumors and to gain a comprehensive understanding of tumors. Furthermore, because CNNs are not good at extracting global features, they cannot establish long-range dependencies during feature extraction, resulting in the loss of some valuable information and limiting the accuracy and reliability of classification results.

[0004] Existing attention mechanism methods based on multimodal MRI fusion utilize multimodal MRI, but are limited to a single scanning plane. Most of them only use the axial axis and ignore the sagittal and coronal axes, resulting in the loss of key tumor information in different scanning planes. Furthermore, because the attention mechanism is not good at extracting local features, the fusion effect is poor and the grading performance is limited. Summary of the Invention

[0005] To address the above deficiencies, this invention provides a method for constructing a hybrid network for glioma grading based on multi-view feature fusion, specifically including the following steps:

[0006] S1. Data Preprocessing:

[0007] The MRI data were standardized and then normalized using the Min-Max method. Slices containing tumors in the axial, sagittal, and coronal planes were selected based on tumor segmentation labels and then centrally cropped to obtain three perspective data sources: inter-slice perspective data source, spatial perspective data source, and modal perspective data source. These were then divided into training and validation sets, and the training set was augmented.

[0008] S2, Feature Extraction:

[0009] This includes a hybrid extraction module for global, local, and multi-scale feature extraction of preprocessed data, obtained by combining the attention reduction FOSRA module that focuses on overlapping space and the adaptive dilated convolution SADConv.

[0010] S3, Multi-view Feature Fusion:

[0011] Includes channel I for capturing and enhancing detailed features:

[0012] The three input feature maps X, Y, Z∈R C×H×W The X and Y values ​​in the image are initially fused using an adder to generate feature map O1. Feature map O1 is then processed in parallel through local and global attention channels. Within the local attention channel, point-wise convolution reduces the number of channels to [number missing]. Next, the number of channels is restored to parameter C by using the ReLU activation function and then by point-wise convolution.

[0013] Includes Channel II for enhancing global feature representation:

[0014] Global average pooling is used to transform the feature map from C×H×W to C×1×1, and then point-wise convolution is used to reduce the number of channels. Then, the ReLU activation function is applied, and finally, the number of channels is restored to parameter C through point-wise convolution.

[0015] The weights generated by channel I and channel II are added together and applied to X and Y to obtain the fused feature map O2. Then O2 and Z are fused again and processed by the same attention mechanism to finally generate the fused feature map 0.

[0016] S4. The obtained features are then processed into hierarchical levels using a classifier.

[0017] Furthermore, the interlayer perspective data source, spatial perspective data source, and modal perspective data source are divided into training set and validation set in a 4:1 ratio, and the enhancement operations include random flipping and rotation operations.

[0018] Furthermore, the working steps of the hybrid extraction module are as follows:

[0019] Input X∈R to the hybrid extraction module C×H×W Then, the hybrid extraction module first splits it into channels. Next, X1 and X2 are processed by the Focused Overlap Spatial Reduction Attention (FOSRA) module and the Adaptive Dilated Convolution (SADConv) module, respectively, and finally, after feature merging, a C×H×W feature map is obtained.

[0020] Furthermore, the hierarchical network structure in the multi-scale convolutional MConv module is a five-layer network structure, with five multi-scale feature extractors of different scales stacked sequentially. The first layer has a scale of... The second layer is The third layer is The fourth layer is The fifth floor is

[0021] The multi-scale convolution MConv module uses a regular depthwise separable convolution (1×1) and three reparameterized refocusing depthwise separable convolutions (3×3, 5×5, 7×7) to process a quarter channel of the output features in parallel.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] It can balance the global receptive field and inductive bias, combining global and local multi-scale and multi-granular features, while balancing the global receptive field and inductive bias, thus enhancing feature extraction capabilities. By constructing data sources from different perspectives (inter-slice perspective, scanning plane perspective, and modal perspective) and designing novel feature fusion modules, the network can fully and effectively utilize the key information provided by MRI from different scanning planes and different modalities, including inter-slice, spatial, and multimodal information, to achieve a comprehensive understanding of gliomas. Attached Figure Description

[0024] Figure 1 This is a diagram of the overall architecture of the present invention.

[0025] Figure 2 This is a flowchart of data preprocessing in this invention.

[0026] Figure 3 This is a visual diagram illustrating the data preprocessing process in this invention.

[0027] Figure 4 This is an overall architecture diagram of the feature extraction step in this invention.

[0028] Figure 5 This is an architecture diagram of the hybrid extraction module in this invention.

[0029] Figure 6 This is an architecture diagram of the Focused Overlap Space Reduction Attention FOSRA module in this invention.

[0030] Figure 7 This is an architecture diagram of the adaptive dilated convolution SADConv module in this invention.

[0031] Figure 8 This is an architecture diagram of the multi-scale convolution MConv module in this invention.

[0032] Figure 9 This is an architecture diagram of the multi-view feature fusion (MVFF) module in this invention. Detailed Implementation

[0033] To facilitate understanding of the present invention, the apparatus of the present invention will now be described more fully with reference to the accompanying drawings. Embodiments of the apparatus are shown in the drawings. However, the apparatus can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to make the disclosure of the present invention more thorough and complete.

[0034] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "configuration" should be interpreted broadly. For example, they can refer to a fixed connection or configuration, a detachable connection or configuration, or an integral connection or configuration. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] Example

[0036] like Figure 1 As shown in the figure, this embodiment provides a method for constructing a hybrid network for glioma grading based on multi-view feature fusion, including four stages: data preprocessing, feature extraction, multi-view feature fusion, and grading processing, as detailed below:

[0037] The data preprocessing steps are as follows:

[0038] Standardize the data from magnetic resonance imaging (MRI) (e.g.) Figure 2 As shown), this process includes image transformation, which makes the image coordinate axes the closest standard orthogonal axes, and then normalization is performed using the Min-Max method.

[0039] Secondly, based on tumor segmentation labels, slices containing tumors in the axial, sagittal, and coronal planes are identified (slices without tumors will not participate in network training). Slices containing tumors in the axial, sagittal, and coronal planes are selected based on tumor segmentation labels and then centrally cropped at a 224×224 ratio to obtain three view data sources: inter-layer view data source (View1), spatial view data source (View2), and modal view data source (View3). Specifically, the inter-layer view data source is based on the same... The modal perspective data source is obtained by channel overlay of four slices from the same scan plane and modality, aiming to simultaneously focus on four slices at different locations and extract more comprehensive interlayer features. The spatial perspective data source is obtained by channel overlay of four slices (including two slices of the same modality and the same scan plane) from three scan planes based on the same modality, aiming to simultaneously focus on slices from three sections at the same location and extract more comprehensive spatial features. The modal perspective data source is obtained by channel overlay of four slices from four different modalities based on the same scan plane, aiming to simultaneously focus on slices from four different modalities at the same location and extract more comprehensive modal features.

[0040] The three data sources were then divided into training and validation sets in a 4:1 ratio, and the training set was augmented with data operations such as random flipping and rotation.

[0041] In practical use, such as Figure 3 As shown, each patient's MRI needs to be processed, including three dimensions: modality, scan plane, and inter-slice. Figure 3 In the diagram, the data represented by 1-12 belongs to View1; the data in View2 is formed by combining the same numbered slices (1, 5, and 7) (or 2, 6, and 10; 3, 7, and 11; 4, 8, and 12), and each data must contain 3 scan planes. The last channel is filled by taking one of the remaining slices; the four-channel data in View3 is formed by combining the same numbered slices (1, 5, and 7) into three four-channel data. Figure 1 , 2 The MRI scans of the same number in 3 and 4 (or 5, 6, 7 and 8; 9, 10, 11 and 12) are combined. Based on the above data preprocessing steps, 12 data points can be generated for each patient's MRI, namely View1, View2 and View3.

[0042] Feature extraction steps Figure 4 As shown, the details are as follows:

[0043] 1) such as Figure 5 As shown, the steps for extracting global and local features are as follows:

[0044] Input X∈R to the hybrid extraction module C×H×W Then, the hybrid extraction module first splits it into channels. Next, X1 and X2 are processed by the Focusing Overlap Spatial Reduction Attention F0SRA module and the Adaptive Dilated Convolution SADConv module, respectively. Finally, feature merging is performed to obtain a C×H×W feature map, which combines global and local features well, while taking into account the global receptive field and inductive bias.

[0045] Existing techniques like Overlapping Spatial Reduction Attention (OSRA) utilize sparse labeled region relationships to effectively extract global information and introduce Overlapping Spatial Reduction (OSR) constraints to better represent spatial structures near boundaries using larger and overlapping regions. However, this approach is challenging due to the computational complexity of secondary self-attention. Therefore, combining the idea of ​​Focused Linear Attention (FLA), an FLA module was designed to reduce computational complexity while maintaining expressive power. The FLA module simulates the distribution of the original Softmax attention by designing a novel mapping function and uses ordinary Depthwise Separable Convolution (DSC) to restore feature diversity. Furthermore, to enhance global feature extraction, the two methods are combined to obtain the Focused Overlapping Spatial Reduction Attention (FOSRA) module.

[0046] like Figure 6 As shown, the given input shape is The tensor is processed by the OSR on the left channel and then by a linear layer. Its output is split into two tensors, V and K. The tensor on the right channel is directly processed by a linear layer to generate the Q tensor. Then, the Q, K and V tensors are fed into the FLA module to finally obtain a tensor with the same shape as the input.

[0047] Right now The announcement is as follows:

[0048] Y = OSR(X1);

[0049] Q = Linear(X1);

[0050] K, V = Split(Linear(Y));

[0051] O = FLA(Q, K, V);

[0052] like Figure 7 As shown, the workflow of the adaptive dilated convolution SADConv module is as follows:

[0053] Input a feature map Adaptive average pooling is used to aggregate the spatial context, compressing the spatial dimension to K. 2 Then, it is fed into a 1×1 convolution for dimensionality reduction, followed by a spatial channel reconstruction convolution (SCRConv) to reduce redundant features and thus save costs; subsequently, a 1×1 convolution is used to obtain the feature map. Where G represents the number of attention groups, and then F is reshaped into... Use the Softmax function to generate attention weights A∈ Then A is compared with the learnable parameters Multiplying them gives the final weight W. Finally, dilated convolution is performed on X and W together, which can be represented as:

[0054]

[0055] A = Softmax(Reshape(F));

[0056]

[0057] O = DConv(X2, W);

[0058] In addition, the non-1×1 standard convolutional layers in the network were reparameterized and refocused, which further enhanced the ability to extract local features.

[0059] 2) Multi-scale feature extraction:

[0060] It includes a hierarchical network structure for multi-scale feature extraction and a multi-scale convolutional MConv module for enhancing feature extraction performance;

[0061] The hierarchical network structure is a five-layer network structure, with five multi-scale feature extractors of different scales stacked sequentially to achieve a first-layer scale of [missing information]. The second layer is The third layer is The fourth layer is The fifth floor is The first layer of image embedding uses a 7×7 convolutional layer (stride=4) and Batch Normalization (BN) to implement the image embedding, while the remaining layers use a 3×3 convolutional layer (stride=2) and BN.

[0062] The multi-scale convolutional MConv module is placed within the multi-scale feature extractor, fully extracting and transforming multi-scale information to improve the model's representational capabilities. For example... Figure 8 As shown, the multi-scale convolution MConv module uses a regular depthwise separable convolution (1×1) and three reparameterized refocusing depthwise separable convolutions (3×3, 5×5, 7×7) in parallel to process one-quarter channels of the output features respectively, in order to effectively capture multi-scale features. The 1×1 depthwise separable convolution is used to weight and scale the channels to better fuse multi-scale information.

[0063] Multi-view feature fusion is achieved through the Multi-view Feature Fusion (MVFF) module:

[0064] like Figure 9 As shown, firstly, the three input feature maps X, Y, Z ∈ R C×H×WX and Y are first initially fused using an adder to generate feature map O1. Then, feature O1 is processed in parallel through local and global attention channels. In the local attention channels, point-wise convolution is used to reduce the number of channels. Next, the ReLU activation function is applied, and then point-wise convolution is used to restore the number of channels to C, which focuses on capturing and enhancing detailed features;

[0065] In the global attention channel, global average pooling is first used to transform the feature map from C×H×W to C×1×1, and then point-wise convolution is used to reduce the number of channels. Next, the ReLU activation function is applied, followed by point-wise convolution to restore the number of channels to C. This channel is used to enhance the expression of global features.

[0066] The weights generated by the two attention channels are added together and applied to X and Y to obtain the fused feature map O2. Then, O2 and Z are fused again and processed by the same attention mechanism to finally generate the fused feature map 0.

[0067] The Multi-View Feature Fusion (MVFF) module combines local and global attention mechanisms to achieve multi-level capture and efficient utilization of features, enhancing the expressive power of features. Through multiple fusion operations, it gradually integrates features from different perspectives, avoiding information loss and ultimately achieving efficient integration of multi-view features.

[0068] The grading process involves feeding the features obtained above into a classifier to achieve accurate and reliable grading of gliomas.

[0069] It should be noted that the structure described in this invention can be implemented in many different forms and is not limited to the embodiments described. Any equivalent transformations made by those skilled in the art based on the description and drawings of this invention, or direct or indirect applications in other related technical fields, such as the loading and unloading of other items, are included within the protection scope of this invention.

Claims

1. A method for establishing a brain glioma grading hybrid network based on multi-view feature fusion, characterized in that: Specifically comprising the following steps: S1, data preprocessing: The data of magnetic resonance imaging (MRI) is standardized and normalized using the Min-Max method. The slices with tumors in the axial, sagittal and coronal planes are selected according to the tumor segmentation label, and center cropping is performed to obtain three perspective data sources, namely the interlayer perspective data source, the spatial perspective data source and the modal perspective data source. Then, the data is divided into a training set and a validation set, and the training set is subjected to data augmentation; The interlayer perspective data source is obtained by channel stacking of four slices based on the same modality and the same scanning plane, aiming to simultaneously focus on four slices at different positions and extract more comprehensive interlayer features; The spatial perspective data source is obtained by channel stacking of four slices based on the same modality and the same scanning plane, aiming to simultaneously focus on three slices at the same position and extract more comprehensive spatial features. The four slices include two slices containing the same modality and the same scanning plane; The modal perspective data source is obtained by channel stacking of four slices based on four modalities of the same scanning plane, aiming to simultaneously focus on four modalities of the same position and extract more comprehensive modal features; S2, feature extraction: It includes a hybrid extraction module for global and local feature extraction, a hierarchical network structure for multi-scale feature extraction, and a multi-scale convolution (MConv) module for enhancing feature extraction effect; S3, multi-perspective feature fusion: It includes channel I for capturing and enhancing detailed features: The input three feature maps X, Y in the adder are preliminarily fused to generate a feature map O1 The feature O1 is processed in parallel through a local attention channel and a global attention channel, in the local attention channel, the channel number is reduced to by Point-wise convolution, then the channel number is restored to the parameter C through ReLU activation function and Point-wise convolution. It includes channel II for enhancing global feature expression: The feature map is converted from to by using global average pooling, and then the number of channels is reduced to by Point-wise convolution, followed by a ReLU activation function, and finally the number of channels is restored to the parameter C by Point-wise convolution. The weights generated by channel I and channel II are added and applied to X and Y to obtain the fusion feature map O2. Then, O2 and Z are fused again and subjected to the same attention mechanism processing to finally generate the fusion feature map O. S4, the obtained features are subjected to hierarchical processing by a classifier.

2. The method of claim 1, wherein the method comprises: The interlayer perspective data source, spatial perspective data source and modal perspective data source are divided into a training set and a validation set in a ratio of 4:

1. The enhancement operation includes random flipping and rotation operation.

3. The method of claim 1, wherein the method comprises: The working steps of the hybrid extraction module are: input to the mixed extraction module After that, the mixed extraction module is firstly split into along the channel dimension, then processed by the focused overlapped spatial reduction attention (FOSRA) module and the self-adaptive dilated convolution (SADConv) module respectively, and finally the feature map of is obtained through feature merging.

4. The method of claim 1, wherein the method comprises: The hierarchical network structure in the multi-scale convolution MConv module is a five-layer network structure, five multi-scale feature extractors with different scales are stacked in sequence, the first layer scale is , the second layer is , the third layer is , the fourth layer is , and the fifth layer is ; The multi-scale convolution MConv module uses one ordinary depthwise separable convolution and three reparameterized refocusing depthwise separable convolutions to process the quarter channels of the output features in parallel, respectively.

Images