Method for establishing brain glioma hierarchical hybrid network based on multi-view feature fusion

Abstract

The invention discloses a method for establishing a brain glioma hierarchical hybrid network based on multi-view feature fusion. The method comprises four steps of data preprocessing, feature extraction, multi-view feature fusion and hierarchical processing of obtained features through a classifier. The global receptive field and the inductive bias can be balanced, the global and local multi-scale and multi-granularity features are combined, the global receptive field and the inductive bias are balanced, and the feature extraction capability is enhanced; by constructing different view angle data sources (an interlayer view angle, a scanning plane view angle and a modal view angle) and designing a novel feature fusion module, a network can fully and effectively utilize key information, including interlayer, space and multi-modal information, provided by MRI (Magnetic Resonance Imaging) of different scanning planes and different modals, so that comprehensive cognition of brain glioma is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image recognition, and in particular, relates to a method for constructing a hybrid network for grading brain gliomas based on multi-view feature fusion. Background Art

[0002] Traditional imaging diagnostic methods are highly dependent on the doctor's personal expertise and clinical experience, and are unable to fully utilize the advantages of magnetic resonance imaging (MRI). They are time-consuming, labor-intensive, and highly subjective, and are prone to diagnostic bias and misdiagnosis.

[0003] The existing convolutional neural network (CNN) method based on single-modality MRI analysis only uses single-modality MRI for feature extraction and modeling, which makes it unable to fully capture the internal heterogeneity and complexity of the tumor and unable to fully understand the tumor. In addition, because CNN is not good at extracting global features, it cannot establish long-range dependencies when extracting features, and thus loses some valuable information, limiting the accuracy and reliability of the classification results.

[0004] Although the existing attention mechanism methods based on multimodal MRI fusion utilize multimodal MRI, they are limited to a single scanning plane. Most of them only use the axial axis and ignore the sagittal and coronal axes, resulting in the loss of key tumor information in different scanning planes. In addition, because the attention mechanism is not good at extracting local features, the fusion effect is poor and the grading performance is limited. Summary of the invention

[0005] In view of the above defects, the present invention provides a method for establishing a hybrid network for grading gliomas based on multi-view feature fusion, which specifically includes the following steps:

[0006] S1. Data preprocessing:

[0007] The MRI data was standardized and then normalized using the Min-Max method. Slices with tumors in the axial, sagittal, and coronal planes were selected according to the tumor segmentation labels, and central cropping was performed to obtain three perspective data sources, namely, the inter-layer perspective data source, the spatial perspective data source, and the modal perspective data source. The data were then divided into a training set and a validation set, and data enhancement was performed on the training set.

[0008] S2. Feature extraction:

[0009] It includes a hybrid extraction module for global, local and multi-scale feature extraction of pre-processed data by combining the Focused Overlapping Spatial Reduced Attention (FOSRA) module and the Adaptive Dilated Convolution (SADConv);

[0010] S3, multi-view feature fusion:

[0011] Includes channel I for capturing and enhancing detailed features:

[0012] The three input feature maps X, Y, Z∈R C×H×W X and Y in are initially fused by the adder to generate feature map O1, which is processed in parallel by the local attention channel and the global attention channel. In the local attention channel, the number of channels is reduced to Then, the ReLU activation function is used, and then the number of channels is restored to the parameter C through Point-wise convolution;

[0013] Including channel II for enhancing global feature expression:

[0014] The feature map is converted from C×H×W to C×1×1 using global average pooling, and then the number of channels is reduced to Then, the ReLU activation function is used, and finally the number of channels is restored to the parameter C through Point-wise convolution;

[0015] The weights generated by channel I and channel II are added and applied to X and Y to obtain the fused feature map O2, and then O2 and Z are fused again and the same attention mechanism is performed to finally generate the fused feature map 0;

[0016] S4. The obtained features are processed by classifier.

[0017] Furthermore, the inter-layer perspective data source, the spatial perspective data source and the modal perspective data source are divided into a training set and a validation set in a ratio of 4:1, and the enhancement operation includes random flipping and rotation operations.

[0018] Furthermore, the working steps of the mixed extraction module are:

[0019] Input X∈R to the hybrid extraction module C×H×W After that, the hybrid extraction module is first split into Then X1 and X2 are processed by the Focused Overlap Spatial Reduced Attention (FOSRA) module and the Adaptive Dilated Convolution (SADConv) module respectively, and finally a C×H×W feature map is obtained by feature merging.

[0020] Furthermore, the hierarchical network structure in the multi-scale convolution MConv module is a five-layer network structure, and five multi-scale feature extractors with different scales are stacked in sequence. The second layer is The third layer is The fourth floor is The fifth floor is

[0021] The multi-scale convolution MConv module uses a normal depthwise separable convolution (1×1) and three re-parameterized refocused depthwise separable convolutions (3×3, 5×5, 7×7) to process one quarter of the channels of the output features in parallel.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] It can balance the global receptive field and inductive bias, combine global and local multi-scale and multi-granularity features, balance the global receptive field and inductive bias, and enhance the feature extraction capability; by constructing data sources with different perspectives (inter-layer perspective, scanning plane perspective, modal perspective) and designing novel feature fusion modules, the network can fully and effectively utilize the key information provided by MRI of different scanning planes and different modalities, including inter-layer, spatial and multi-modal information, to achieve a comprehensive understanding of brain gliomas. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall architecture diagram of the present invention.

[0025] Figure 2 The figure is a flow chart of data preprocessing in the present invention.

[0026] Figure 3 It is a visualization diagram of data preprocessing in the present invention.

[0027] Figure 4 This is the overall architecture diagram of the feature extraction step in the present invention.

[0028] Figure 5 This is an architectural diagram of the hybrid extraction module in the present invention.

[0029] Figure 6 This is the architecture diagram of the F0SRA module for focusing on overlapping space and reducing attention in the present invention.

[0030] Figure 7 This is the architecture diagram of the adaptive dilated convolution SADConv module in the present invention.

[0031] Figure 8 This is the architecture diagram of the multi-scale convolution MConv module in the present invention.

[0032] Fig. 9 This is the architecture diagram of the multi-view feature fusion MVFF module in the present invention. DETAILED DESCRIPTION

[0033] In order to facilitate the understanding of the present invention, the device of the present invention will be described more fully below with reference to the relevant drawings. Embodiments of the device are given in the drawings. However, the device can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0034] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "set" should be understood in a broad sense, for example, they can be fixedly connected or set, or detachably connected or set, or integrally connected or set. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0035] Example

[0036] like Figure 1 As shown, this embodiment provides a method for constructing a hybrid network for grading glioma based on multi-view feature fusion, including four stages: data preprocessing, feature extraction, multi-view feature fusion, and grading processing, as follows:

[0037] The data preprocessing steps are as follows:

[0038] Standardize MRI data (such as Figure 2 As shown), this process includes image transformation, that is, the image coordinate axis becomes the closest standard orthogonal axis, and then the Min-Max method is used for normalization;

[0039] Secondly, the slices with tumors in the axial, sagittal and coronal planes are determined according to the tumor segmentation labels (the slices without tumors will not participate in network training). The slices with tumors in the axial, sagittal and coronal planes are selected according to the tumor segmentation labels, and they are centrally cropped at a ratio of 224×224 to obtain three perspective data sources, namely, the inter-layer perspective data source (View1), the spatial perspective data source (View2) and the modal perspective data source (View3). In detail, the inter-layer perspective data source is based on the same The data source of spatial perspective is obtained by channel superposition of four slices of three scanning planes based on the same modality (including two slices of the same modality and the same scanning plane), aiming to focus on the slices of three sections at the same position at the same time and extract more comprehensive spatial features; and the data source of modal perspective is obtained by channel superposition of four slices of four modes based on the same scanning plane, aiming to focus on the slices of four modes at the same position at the same time and extract more comprehensive modal features.

[0040] The three data sources are then divided into training set and validation set in a ratio of 4:1, and the training set is enhanced, including random flipping and rotation.

[0041] In actual use, Figure 3 As shown, each patient's MRI needs to be processed, including the three dimensions of modality, scanning plane and slice. Figure 3 In the above diagram, the data represented by 1-12 belongs to View1; the slices with the same number in 1, 5 and 7 (or 2, 6 and 10; 3, 7 and 11; 4, 8 and 12) are combined into three four-channel data, which are the data in View2, and each data must contain 3 scanning planes, and the last channel is filled with one of the remaining slices; the four-channel data in View3 is composed of the slices in View2. Figure 1 , 2 , 3 and 4 (or 5, 6, 7 and 8; 9, 10, 11 and 12). According to the above data preprocessing steps, the MRI of each patient can generate 12 data for View1, View2 and View3 respectively.

[0042] Feature extraction steps Figure 4 As shown, the details are as follows:

[0043] 1) If Figure 5 As shown, the steps for extracting global and local features are:

[0044] Input X∈R to the hybrid extraction module C×H×W After that, the hybrid extraction module is first split into Then X1 and X2 are processed by the focused overlapping spatial reduced attention F0SRA module and the adaptive dilated convolution SADConv module respectively, and finally a C×H×W feature map is obtained after feature merging to well combine global and local features while taking into account the global receptive field and inductive bias.

[0045] The existing Overlapping Spatial Reduction Attention (OSRA) technology uses sparse labeled region relationships to effectively extract global information and introduces Overlapping Spatial Reduction Attention (OSR) to better represent the spatial structure near the boundary by using larger and overlapping regions. However, it is difficult for the quadratic computational complexity of self-attention. Therefore, the idea of ​​Focused Linear Attention (FLA) is combined to design the FLA module, which reduces the computational complexity while maintaining the expressive power. The FLA module simulates the distribution of the original Softmax attention by designing a novel mapping function and uses ordinary depthwise separable convolution (DSC) to restore feature diversity. In order to enhance the extraction of global features, the above two methods are combined to obtain the Focused Overlapping Spatial Reduction Attention FOSRA module.

[0046] like Figure 6 As shown, given the input shape is The left channel first passes through OSR and then through a linear layer, and its output is split into two tensors V and K. The right channel directly passes through a linear layer to generate a Q tensor. Subsequently, the Q, K, and V tensors are fed into the FLA module, and finally a tensor with the same shape as the input is obtained;

[0047] Right now The announcement is as follows:

[0048] Y = OSR (X1);

[0049] Q = Linear(X1);

[0050] K, V = Split(Linear(Y));

[0051] O = FLA (Q, K, V);

[0052] like Figure 7 As shown in the figure, the workflow of the adaptive dilated convolution SADConv module is as follows:

[0053] Input a feature map Adaptive average pooling is used to aggregate the spatial context and compress the spatial dimension to K. 2 , and then send it to 1×1 convolution for dimensionality reduction, and then perform spatial channel reconstruction convolution (SCRConv) to reduce redundant features and save costs; then use 1×1 convolution to get the feature map Where G represents the number of attention groups. Then, F is reshaped into And use the Softmax function to generate the attention weight A∈ Then A and the learnable parameter Multiply them together to get the final weight W, and finally perform a dilated convolution on X and W, which can be expressed as:

[0054]

[0055] A = Softmax(Reshape(F));

[0056]

[0057] O = DConv(X2, W);

[0058] In addition, all non-1×1 standard convolutional layers in the network are reparameterized and refocused to further enhance the ability to extract local features.

[0059] 2) Multi-scale feature extraction:

[0060] It includes a hierarchical network structure for extracting multi-scale features and a multi-scale convolution MConv module for enhancing the feature extraction effect;

[0061] The hierarchical network structure is a five-layer network structure, and five multi-scale feature extractors with different scales are stacked in sequence to achieve the first layer scale of The second layer is The third layer is The fourth floor is The fifth floor is The image embedding of the first layer is implemented using a 7×7 convolutional layer (stride=4) and batch normalization (BN), and the image embedding of the remaining layers is implemented using a 3×3 convolutional layer (stride=2) and BN.

[0062] The multi-scale convolution MConv module is placed in the multi-scale feature extractor to fully extract and transform multi-scale information to improve the model representation capability. Figure 8 As shown in the figure, the multi-scale convolution MConv module uses a normal depth-separable convolution (1×1) and three re-parameterized refocused depth-separable convolutions (3×3, 5×5, 7×7) to process a quarter of the output feature channels in parallel to effectively capture multi-scale features. The 1×1 depth-separable convolution is used to perform weighted scaling on the channel to better fuse multi-scale information.

[0063] Multi-view feature fusion is achieved through the multi-view feature fusion MVFF module:

[0064] like Fig. 9 As shown, first, the three feature maps X, Y, Z ∈ R C×H×W, X, Y are first fused by the adder to generate feature map O1. Then, feature O1 is processed in parallel through the local attention channel and the global attention channel. In the local attention channel, the number of channels is reduced to Then, the ReLU activation function is used, and then the Point-wise convolution is used to restore the number of channels to C. This channel focuses on capturing and enhancing detailed features.

[0065] In the global attention pass, the feature map is first converted from C×H×W to C×1×1 using global average pooling, and then the number of channels is reduced to Then, the ReLU activation function is used, and then the Point-wise convolution is used to restore the number of channels to C. This channel is used to enhance the expression of global features.

[0066] The weights generated by the two attention channels are added and applied to X and Y to obtain the fused feature map O2. Then, O2 and Z are fused again and processed by the same attention mechanism to finally generate the fused feature map 0.

[0067] The multi-view feature fusion MVFF module achieves multi-level capture and efficient utilization of features through the combination of local and global attention mechanisms, enhancing the expressiveness of features; through multiple fusion operations, it gradually integrates features from different viewpoints to avoid information loss, and ultimately achieves efficient integration of multi-view features.

[0068] The classification process is to send the above-obtained features into a classifier to achieve accurate and reliable classification of brain gliomas.

[0069] It should be noted that the structure described in the present invention can be implemented in a variety of different forms and is not limited to the described embodiments. Any equivalent transformations made by ordinary technicians in this field using the contents of the present invention description and drawings, or directly or indirectly applied to other related technical fields, such as the loading and unloading of other items, are included in the protection scope of the present invention.

Claims

1. A method for constructing a hybrid network for grading gliomas based on multi-view feature fusion, characterized by: The specific steps include: S1. Data preprocessing: The MRI data was standardized and then normalized using the Min-Max method. Slices with tumors in the axial, sagittal, and coronal planes were selected according to the tumor segmentation labels, and central cropping was performed to obtain three perspective data sources, namely, the inter-layer perspective data source, the spatial perspective data source, and the modal perspective data source. The data were then divided into a training set and a validation set, and data enhancement was performed on the training set. S2. Feature extraction: It includes a hybrid extraction module for global and local feature extraction, a hierarchical network structure for multi-scale feature extraction, and a multi-scale convolution MConv module for enhancing feature extraction effects; S3, multi-view feature fusion: Includes channel I for capturing and enhancing detailed features: The three input feature maps X, Y, Z∈R C×H×W X and Y in are initially fused by the adder to generate feature map O1, which is processed in parallel by the local attention channel and the global attention channel. In the local attention channel, the number of channels is reduced to Then, the ReLU activation function is used, and then the number of channels is restored to the parameter C through Point-wise convolution; Including channel II for enhancing global feature expression: The feature map is converted from C×H×W to C×1×1 using global average pooling, and then the number of channels is reduced to Then, the ReLU activation function is used, and finally the number of channels is restored to the parameter C through Point-wise convolution; The weights generated by channel I and channel II are added and applied to X and Y to obtain the fused feature map O2, and then O2 and Z are fused again and the same attention mechanism is performed to finally generate the fused feature map 0; S4. The obtained features are processed by classifier.

2. The method for constructing a hybrid network for grading glioma based on multi-view feature fusion according to claim 1, characterized in that: The inter-layer perspective data source, the spatial perspective data source and the modal perspective data source are divided into a training set and a validation set in a ratio of 4:1, and the enhancement operation includes random flipping and rotation operations.

3. The method for establishing a hybrid network for grading glioma based on multi-view feature fusion according to claim 1, characterized in that: The working steps of the mixed extraction module are: Input X∈R to the hybrid extraction module C×H×W After that, the hybrid extraction module is first split into X1 according to the channel dimension, Then X1 and X2 are processed by the focused overlapping spatial reduced attention F0SRA module and the adaptive dilated convolution SADConv module respectively, and finally a C×H×W feature map is obtained after feature merging.

4. The method for constructing a hybrid network for grading glioma based on multi-view feature fusion according to claim 1, characterized in that: The hierarchical network structure in the multi-scale convolution MConv module is a five-layer network structure, in which five multi-scale feature extractors of different scales are stacked in sequence. The scale of the first layer is The second layer is The third layer is The fourth floor is The fifth floor is The multi-scale convolution MConv module uses a 1×1 ordinary depth-wise separable convolution and three 3×3, 5×5, and 7×7 re-parameterized refocused depth-wise separable convolutions to process one quarter of the output feature channels in parallel.

Images