Multi-modal fusion lightweight segmentation network and segmentation method for brain MRI images

Abstract

The application discloses a kind of multimodal fusion light-weight segmentation network and segmentation method for brain MRI image, mainly including three parts: coding part, containing four independent encoders, to carry out feature extraction to four modal original drawings respectively, while different modalities adopt different attention strategies, wherein each encoder contains three convolution modules, to carry out down-sampling by convolution and pooling layer;Feature fusion part carries out feature fusion to four modalities at feature level, when fusion, with different combinations at different feature layers, join light-weight modal attention, spatial attention and channel attention, to improve the segmentation accuracy of model;And decoding part, using convolution and up-sampling, to restore the original resolution of feature map, the up-sampling is realized using transpose convolution.Compared with prior art, the network architecture designed in the application has higher segmentation accuracy while keeping the model lightweight.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of multimodal fusion light weight segmentation network and segmentation method for brain MRI image, belong to medical image segmentation field. BACKGROUND

[0002] Brain tumor is a disease formed by cancer cells in the brain, and is a life-threatening medical condition. The purpose of brain tumor segmentation is to identify and segment tumor regions and healthy tissues with medical imaging technology. Computed tomography (CT), positron emission computed tomography (PET) and magnetic resonance imaging (MRI) are the three most commonly used imaging methods for diagnosing brain tumors. Among them, MRI is widely used because of its high resolution, strong soft tissue contrast and non-invasiveness. Manual tumor contour segmentation based on brain tumor images is both labor-intensive and prone to errors. With the development of deep learning, brain tumor segmentation neural network models based on automatic image features have become the mainstream. Generally, the method is to simply stack the four modalities of MRI, without taking full advantage of the characteristics of different modalities, and the segmentation accuracy is not very high. And 3D medical segmentation has large amount of calculation, and the training time required is long. Under the premise of ensuring the accuracy to meet the application standard, it is also important to reduce the model parameter quantity.

[0003] U-Net consists of an encoder that extracts features from the input image and a decoder that generates a segmentation mask. The encoder and decoder are connected by a "bridge" to pass high-level features to the decoder. Because U-Net has the characteristics of small parameter quantity and high accuracy, it has been widely used in various segmentation tasks. However, most of the current mainstream models are based on U-Net variants, but mainly stack the four modalities directly at the pixel level, and then encode them by an encoder, so different strategies cannot be taken for different modalities. SUMMARY

[0004] The purpose of the present application is to provide a multimodal fusion light weight segmentation network and segmentation method for brain MRI image, to improve the model segmentation accuracy while keeping the model light weight.

[0005] To achieve the above purpose, the present application provides a multimodal fusion light weight segmentation network for brain MRI image, comprising:

[0006] The encoding part contains four independent encoders to extract features from the original images of the four modalities, and different attention strategies are taken for different modalities, wherein each encoder contains three convolution modules to downsample through convolution and pooling layers;

[0007] The feature fusion part fuses the features of the four modalities at the feature level, and adds lightweight modal attention, spatial attention and channel attention at different feature layers in different combinations, so as to improve the segmentation accuracy of the model.

[0008] The decoding part recovers the original resolution of the feature map by convolution and upsampling, and the upsampling is realized by transposed convolution.

[0009] As a further improvement of the application, before feature fusion, the channel attention mechanism is added to each layer of the encoder of the enhanced sequence to focus on the weights of different channels of the enhanced sequence.

[0010] As a further improvement of the application, in the feature fusion part, the same layer features of the four encoders are connected by a jump connection.

[0011] As a further improvement of the application, the core component of the feature fusion part is a lightweight attention module ACSMB based on modal, channel and space, which uses channel attention CA, spatial attention SA and modal attention MA in sequence to calculate the weights of channels, spaces and modalities.

[0012] As a further improvement of the application, the second layer features of the four encoders contain detailed information and high-level semantics, and the second layer features use the attention mechanism based on modal, channel and space during fusion, and the remaining two layer features only use the attention mechanism based on channel and space during fusion.

[0013] As a further improvement of the application, in the decoding part, the fused multi-modal features are decoded and output by a pool layer through three layers of upsampling to obtain the segmentation result.

[0014] To achieve the above purpose, the application provides a segmentation method of a multi-modal fusion lightweight segmentation network, which is applied to the aforementioned segmentation network and mainly includes the following steps:

[0015] S1: two consecutive 3*3*3 convolution operations with a step size of 1 are performed on the input original images T1, T2, T1ce and Flair sequence to generate a first feature map with a resolution equal to that of the original image and a channel number of 16;

[0016] S2: the first feature map of the T1, T2, T1ce and Flair sequence is first subjected to maximum pooling with a step size of 2, and then subjected to two consecutive 3*3*3 convolution operations with a step size of 1 to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32;

[0017] S3: maximum pooling with a step size of 2 is performed on the second feature maps of the T1, T2, T1ce, and Flair sequences, and then two consecutive 3x3x3 convolutions with a step size of 1 are performed to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0018] S4: two consecutive 3x3x3 convolutions with a step size of 1 are performed on the third feature maps of the T1, T2, T1ce, and Flair sequences, respectively, to generate fourth feature maps with a resolution of 1 / 8 of the original image and a channel number of 64;

[0019] S5: after the first feature maps of the four sequences are spliced in the channel dimension, a feature map with a channel number of 64 is generated, and then a channel-spatial joint attention module is used, followed by two consecutive 3x3x3 convolutions with a step size of 1 to generate a fifth feature map with a resolution equal to that of the original image and a channel number of 16;

[0020] S6: after the second feature maps of the four sequences are spliced in the channel dimension, a feature map with a channel number of 128 is generated, and then a lightweight self-attention module is used, followed by two consecutive 3x3x3 convolutions with a step size of 1 to generate a sixth feature map with a resolution of 1 / 2 of the original image and a channel number of 32;

[0021] S7: after the third feature maps of the four sequences are spliced in the channel dimension, a feature map with a channel number of 256 is generated, and then a lightweight self-attention module is used, followed by two consecutive 3x3x3 convolutions with a step size of 1 to generate a seventh feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0022] S8: after the fourth feature maps of the four sequences are spliced in the channel dimension, a feature map with a channel number of 256 is generated, and then two consecutive 3x3x3 convolutions with a step size of 1 are performed to generate an eighth feature map with a resolution of 1 / 8 of the original image and a channel number of 128;

[0023] S9: after the eighth feature map is subjected to a 3x3x3 transposed convolution with a step size of 1, a ninth feature map with a resolution of 1 / 4 of the original image and a channel number of 64 is generated;

[0024] S10: after the ninth feature map and the seventh feature map are spliced in the channel dimension, a feature map with a channel number of 128 is generated, and then two consecutive 3x3x3 convolutions with a step size of 1 are performed to generate a tenth feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0025] S11: after the tenth feature map is subjected to a 3x3x3 transposed convolution with a step size of 1, an eleventh feature map with a resolution of 1 / 2 of the original image and a channel number of 32 is generated.

[0026] S12: after the eleventh feature map and the sixth feature map are spliced in the channel dimension, a feature map with a channel number of 64 is generated, and then two consecutive 3*3*3 convolution operations with a step of 1 are performed to generate a twelfth feature map with a resolution of 1 / 2 of the original image and a channel number of 32;

[0027] S13: after the twelfth feature map is subjected to a 3*3*3 transposed convolution operation with a step of 1, a thirteenth feature map with a resolution equal to that of the original image and a channel number of 16 is generated;

[0028] S14: after the thirteenth feature map and the fifth feature map are spliced in the channel dimension, a feature map with a channel number of 32 is generated, and then two consecutive 3*3*3 convolution operations with a step of 1 are performed to generate a fourteenth feature map with a resolution equal to that of the original image and a channel number of 16, which is a segmentation result map.

[0029] As a further improvement of the application, the step S1 specifically comprises:

[0030] S11: two consecutive 3*3*3 convolution operations with a step of 1 are performed on the input original image T1 sequence to generate a first feature map with a resolution equal to that of the original image and a channel number of 16;

[0031] S12: two consecutive 3*3*3 convolution operations with a step of 1 are performed on the input original image T2 sequence to generate a first feature map with a resolution equal to that of the original image and a channel number of 16;

[0032] S13: two consecutive 3*3*3 convolution operations with a step of 1 are performed on the input original image T1ce sequence, and then a channel attention module is used to generate a first feature map with a resolution equal to that of the original image and a channel number of 16;

[0033] S14: two consecutive 3*3*3 convolution operations with a step of 1 are performed on the input original image Flair sequence to generate a first feature map with a resolution equal to that of the original image and a channel number of 16.

[0034] As a further improvement of the application, the step S2 specifically comprises:

[0035] S21: the first feature map of the T1 sequence is subjected to a maximum value pooling operation with a step of 2, and then two consecutive 3*3*3 convolution operations with a step of 1 are performed to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32;

[0036] S22: the first feature map of the T2 sequence is subjected to maximum value pooling with a step of 2, then two consecutive 3*3*3 convolution operations with a step of 1 are performed, so that a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32 is obtained;

[0037] S23: the first feature map of the T1ce sequence is subjected to maximum value pooling with a step of 2, then two consecutive 3*3*3 convolution operations with a step of 1 are performed, and then a channel attention module is used, so that a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32 is obtained;

[0038] S24: the first feature map of the Flair sequence is subjected to maximum value pooling with a step of 2, then two consecutive 3*3*3 convolution operations with a step of 1 are performed, so that a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32 is obtained.

[0039] As a further improvement of the application, the step S3 specifically comprises:

[0040] S31: the second feature map of the T1 sequence is subjected to maximum value pooling with a step of 2, then two consecutive 3*3*3 convolution operations with a step of 1 are performed, so that a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64 is obtained;

[0041] S32: the second feature map of the T2 sequence is subjected to maximum value pooling with a step of 2, then two consecutive 3*3*3 convolution operations with a step of 1 are performed, so that a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64 is obtained;

[0042] S33: the second feature map of the T1ce sequence is subjected to maximum value pooling with a step of 2, then two consecutive 3*3*3 convolution operations with a step of 1 are performed, and then a channel attention module is used, so that a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64 is obtained;

[0043] S34: the second feature map of the Flair sequence is subjected to maximum value pooling with a step of 2, then two consecutive 3*3*3 convolution operations with a step of 1 are performed, so that a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64 is obtained.

[0044] Compared with the prior art, the technical effect of the application is that the application has significantly improved the model precision compared with the existing model, and the experimental results show that the network architecture designed in the application has achieved high segmentation precision while maintaining the lightweight of the model. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The figure is a schematic diagram of the architecture of the multi-modal fusion lightweight segmentation network for brain MRI images.

[0046] Figure 2 Channel attention, spatial attention, modal attention and 3DConvBlock for the present application.

[0047] Figure 3 Architecture diagram of the modal channel and spatial joint attention module (ACSMB) and channel spatial joint attention module (ACSB) of the present application.

[0048] Figure 4 Segmentation result diagram of the network of the present application on the BRATS2021 dataset.

[0049] Figure 5 Experimental result diagram of the present application compared with other models. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in detail below in combination with the drawings and specific embodiments.

[0051] Here, it should be noted that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the scheme of the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0052] In the Brats2021 dataset, the labels are divided into obviously enhanced tumor regions (ET), gross tumor regions also known as "tumor core" (TC), and complete tumor regions (WT). Compared with the conventional T1, T1ce shows high signal in the ET region. The TC region contains most of the tumor, which is the typical area that needs to be resected. Compared with T1, the appearance of necrosis (NCR) and non-enhanced (NET) tumors in T1ce usually shows obvious low signal. The whole tumor describes the complete degree of the disease, because it includes the tumor core region and peritumoral edema (ED), which is usually described by high intensity signal in FLAIR. According to different tumor biological characteristics, the present application adds different attention mechanisms to different modalities in the encoding part, and also takes corresponding strategies when fusing at the feature level.

[0053] The overall network framework proposed by the present application is a multi-modal fusion lightweight segmentation network based on modal attention. After the four modal sequences pass through the separate encoder, the feature-level multi-modal fusion is first performed, and then the fused features are decoded to obtain the segmentation image.

[0054] As Figure 1As shown, the multi-modal fusion lightweight segmentation network for brain MRI images provided by the application is composed of three basic parts, including an encoding part, a feature fusion part and a decoding part. The encoding part contains four independent encoders to respectively extract features of the original images of the four modalities, so as to take different attention strategies for each modality according to different tumor biological characteristics before feature fusion. Each encoder contains three convolutional modules to down-sample through convolution and pooling layers. The application pays more attention to the differences between modalities, and different processing is performed on different modalities before early fusion of the modalities. According to the relevant knowledge in the medical field, the T1ce enhanced sequence has more rich feature information of the tumor part, so the channel attention mechanism (CA, Channel-Attention) is added to each layer of the encoder of the T1ce enhanced sequence before the modal feature fusion, so as to pay attention to the weight of different channels of the enhanced sequence, and fully utilize the feature enhancement characteristics of the sequence.

[0055] The feature fusion part is used to fuse the features of the four modalities at the feature level, and light-weight modal attention, spatial attention and channel attention are added at different feature layers in different combinations to improve the segmentation accuracy of the model. Specifically, in the feature fusion part, the same layer features of the four encoders are connected through a jump connection. Considering that the second layer features of the four encoders contain both detailed information and high-level semantics, only the second layer features adopt the modal, channel and spatial joint attention (ACSM) mechanism during fusion, and the other two layer features only adopt the channel and spatial joint attention (ACS) mechanism during fusion, which not only improves the accuracy of the model, but also maintains the lightweight of the model.

[0056] That is, the core component of the feature fusion part is a novel lightweight attention module based on modal, channel and spatial joint attention (ACSM, Attention Based On Channel Spatial and Modal), which successively adopts lightweight channel attention (CA, Channel-Attention), spatial attention (SA, Spatial-Attention) and modal attention (MA, Modal-Attention) to respectively calculate the weights of the channel, space and modal, so that the modal can make full use of the respective features of each modality during fusion, thereby improving the segmentation accuracy. The lightweight attention module used at the same time can also effectively reduce the computational complexity of the network, so that the network can maintain very few model parameters and improve the feature extraction effect, ensuring that the network can achieve the best trade-off between accuracy and efficiency.

[0057] The decoding part adopts convolution and up-sampling to restore the original resolution of the feature map. The up-sampling is implemented by using a transposed convolution. Specifically, the fused multi-modal feature is decoded by the same decoder through three up-sampled pooling layers to output a segmentation result.

[0058] According to Figure 1 The overall network framework can efficiently extract the modal information and detail information of the nuclear magnetic data and can be trained end-to-end. Compared with the recent mainstream segmentation network, the designed network architecture realizes higher detection accuracy and a lightweight model structure.

[0059] In combination Figure 1 The segmentation method of the overall network structure designed by the application is described, mainly including the following steps:

[0060] S1: Two consecutive 3*3*3 convolutions with a step size of 1 are performed on the input original images T1, T2, T1ce and Flair sequences to generate a first feature map with a resolution equal to that of the original image and a channel number of 16. Specifically, it includes:

[0061] S11: Two consecutive 3*3*3 convolutions with a step size of 1 are performed on the input original image T1 sequence to generate a first feature map with a resolution equal to that of the original image and a channel number of 16;

[0062] S12: Two consecutive 3*3*3 convolutions with a step size of 1 are performed on the input original image T2 sequence to generate a first feature map with a resolution equal to that of the original image and a channel number of 16;

[0063] S13: Two consecutive 3*3*3 convolutions with a step size of 1 are performed on the input original image T1ce sequence, and then a channel attention module is used to generate a first feature map with a resolution equal to that of the original image and a channel number of 16;

[0064] S14: Two consecutive 3*3*3 convolutions with a step size of 1 are performed on the input original image Flair sequence to generate a first feature map with a resolution equal to that of the original image and a channel number of 16.

[0065] S2: The first feature maps of the T1, T2, T1ce and Flair sequences are first subjected to maximum value pooling with a step size of 2, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32. Specifically, it includes:

[0066] S21: The first feature map of the T1 sequence is first subjected to maximum value pooling with a step size of 2, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32;

[0067] S22: performing maximum value pooling with a step of 2 on the first feature map of the T2 sequence, and then performing two consecutive 3*3*3 convolution with a step of 1 to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32;

[0068] S23: performing maximum value pooling with a step of 2 on the first feature map of the T1ce sequence, and then performing two consecutive 3*3*3 convolution with a step of 1, and then passing through a channel attention module to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32;

[0069] S24: performing maximum value pooling with a step of 2 on the first feature map of the Flair sequence, and then performing two consecutive 3*3*3 convolution with a step of 1 to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32.

[0070] S3: performing maximum value pooling with a step of 2 on the second feature map of the T1, T2, T1ce and Flair sequences, and then performing two consecutive 3*3*3 convolution with a step of 1 to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64. Specifically, it includes:

[0071] S31: performing maximum value pooling with a step of 2 on the second feature map of the T1 sequence, and then performing two consecutive 3*3*3 convolution with a step of 1 to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0072] S32: performing maximum value pooling with a step of 2 on the second feature map of the T2 sequence, and then performing two consecutive 3*3*3 convolution with a step of 1 to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0073] S33: performing maximum value pooling with a step of 2 on the second feature map of the T1ce sequence, and then performing two consecutive 3*3*3 convolution with a step of 1, and then passing through a channel attention module to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0074] S34: performing maximum value pooling with a step of 2 on the second feature map of the Flair sequence, and then performing two consecutive 3*3*3 convolution with a step of 1 to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64.

[0075] S4: performing two consecutive 3*3*3 convolution with a step of 1 (3D ConvBlock) on the third feature map of the T1, T2, T1ce and Flair sequences respectively to generate a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 64.

[0076] S5: The first feature maps of the four sequences are concatenated in the channel dimension to generate a feature map with a channel number of 64, and then a channel spatial joint attention module (ACSB) is used, followed by two consecutive 3x3x3 convolutions (3D ConvBlock) with a step size of 1 to generate a fifth feature map with a resolution equal to that of the original image and a channel number of 16.

[0077] S6: The second feature maps of the four sequences are concatenated in the channel dimension to generate a feature map with a channel number of 128, and then an ACSMB module is used, followed by two consecutive 3x3x3 convolutions (3D ConvBlock) with a step size of 1 to generate a sixth feature map with a resolution of 1 / 2 of the original image and a channel number of 32.

[0078] S7: The third feature maps of the four sequences are concatenated in the channel dimension to generate a feature map with a channel number of 256, and then a lightweight self-attention module (ACSMB) is used, followed by two consecutive 3x3x3 convolutions (3D ConvBlock) with a step size of 1 to generate a seventh feature map with a resolution of 1 / 4 of the original image and a channel number of 64.

[0079] S8: The fourth feature maps of the four sequences are concatenated in the channel dimension to generate a feature map with a channel number of 256, and then two consecutive 3x3x3 convolutions (3D ConvBlock) with a step size of 1 are used to generate an eighth feature map with a resolution of 1 / 8 of the original image and a channel number of 128.

[0080] S9: The eighth feature map is passed through a 3x3x3 transposed convolution with a step size of 1 to generate a ninth feature map with a resolution of 1 / 4 of the original image and a channel number of 64.

[0081] S10: The ninth feature map and the seventh feature map are concatenated in the channel dimension to generate a feature map with a channel number of 128, and then two consecutive 3x3x3 convolutions (3D ConvBlock) with a step size of 1 are used to generate a tenth feature map with a resolution of 1 / 4 of the original image and a channel number of 64.

[0082] S11: The tenth feature map is passed through a 3x3x3 transposed convolution with a step size of 1 to generate an eleventh feature map with a resolution of 1 / 2 of the original image and a channel number of 32.

[0083] S12: After the eleventh feature map and the sixth feature map are spliced in the channel dimension, a feature map with a channel number of 64 is generated, and then two consecutive 3x3x3 convolution layers with a step of 1 (3D ConvBlock) are used to generate a twelfth feature map with a resolution of 1 / 2 of the original image and a channel number of 32.

[0084] S13: After the twelfth feature map is spliced in the channel dimension, a feature map with a channel number of 32 is generated, and then two consecutive 3x3x3 convolution layers with a step of 1 (3D ConvBlock) are used to generate a thirteenth feature map with a resolution equal to the original image and a channel number of 16.

[0085] S14: After the thirteenth feature map and the fifth feature map are spliced in the channel dimension, a feature map with a channel number of 32 is generated, and then two consecutive 3x3x3 convolution layers with a step of 1 (3D ConvBlock) are used to generate a fourteenth feature map with a resolution equal to the original image and a channel number of 16, which is the segmentation result map.

[0086] Special note: The above steps S1-S4 represent the output of four modalities of three layers, respectively, and are combined with steps S5-S14 Figure 1 to be described, the steps and Figure 1 may be mutually verified.

[0087] Figure 2 CA, SA, MA and 3D ConvBlock are the core modules of the entire network, which are described in detail as follows:

[0088] Figure 2 (a) in the above is a spatial attention module, which performs channel dimension maximum pooling and average pooling on the given input feature map respectively to generate two 3D spatial attention maps and Then, after adding the two feature maps obtained, a 7x7x7 convolution layer with a step of 1 is used, and finally a Sigmoid activation function is used to compress

[0089] M S (F1) = sigmoid(Conv 7×7×7 (AvgPool(F1) + MaxPool(F1))) (1).

[0090] Figure 2 (b) in the above is a 3D ConvBlock module, which is composed of two 3x3x3 convolution layers, batch processing and Relu layer in series.

[0091] Figure 2 (c) in the above is a channel attention module, which performs channel dimension maximum pooling and average pooling on the given input feature map Adaptive average pooling and adaptive max pooling are performed separately to generate two 1D channel attention maps. and Then the two attention maps M CM and M CA The inputs are fed into a multilayer perceptron (MLP) module and then summed. The sum is then compressed using a sigmoid activation function to obtain the final product.

[0092] M C (F2)=sigmoid(MLP(AvgPool(F2))+MLP(MaxPool(F2))) (2).

[0093] Figure 2 In this paper, (d) represents the modal attention module MA designed in this invention, given the input. First, transform its dimensions as The feature maps of the four modalities are then compressed using average pooling to obtain the spatial dimension. Then, the channel dimension is compressed using average pooling to obtain... The attention feature map is then multiplied back into the original feature map by the weight scalars of the four modalities:

[0094] M M (F3)=Reshape(AvgPool(AvgPool(Reshape(F3)))×F3') (3).

[0095] Figure 3 Figure (a) shows the architecture of the lightweight self-attention module (ACSMB) designed in this invention. Given input First, it is passed through a channel attention module, and the output is a... The channel attention weight matrix is ​​then multiplied by the original feature map to obtain... Then output by a spatial attention module Spatial attention weights are multiplied by the input to obtain Then, a modal attention module is used to focus on feature information at different locations in 3D space, and the output is then compared with the input. Adding them together produces one Feature map:

[0096]

[0097]

[0098] F CSM =M M (F CS )+F CS (6).

[0099] Figure 3 (b) is an architecture diagram of the ACSB module, and Figure 3 Similar to (a) in the figure, the ACSMB is based on the removal of a modal attention module.

[0100] Figure 4 is the segmentation verification result of the segmentation network of the application on the Brats2021 dataset. In order to verify the accuracy and implementation efficiency of the network designed by the application, the model is trained, evaluated and predicted on the widely used Brats2021 dataset. The Brats2021 training set / validation set / test set respectively contains 1251 / 219 / 570 images.

[0101] Figure 5 is the experimental result compared with other models. The average Dice coefficient of the four-encoder model is improved by 0.009 compared with the basic single-encoder model, and after adding the SA module to the T1ce sequence, the feature information of the sequence is better utilized, so it can be found that compared with the ordinary four-encoder model structure, the Dice coefficients of ET and CT are improved by 0.013 and 0.010 respectively. For the feature splicing part of multi-modal fusion, ACS modules are added in the first and third layers, and an ACSM module is added in the second layer, which strengthens the weight of each part in the modal fusion process and the attention degree to different modalities. The average DICE of the final model reaches the best 0.891, in which the DICE coefficient, ET is improved by 0.037, TC is improved by 0.038, and WT is improved by 0.015.

[0102] In summary, the application uses a novel lightweight attention module ACSM based on modal, channel and space, which uses lightweight channel attention CA, spatial attention SA and modal attention MA in turn to calculate the weights of channels, spaces and modalities, so that the modalities can make full use of the characteristics of each modality during fusion, thereby improving the segmentation accuracy. The application has significantly improved the accuracy compared with existing models, and the experimental results show that the network architecture designed by the application has achieved high segmentation accuracy while keeping the model lightweight.

[0103] The above embodiments are only used to illustrate the technical solutions of the application and not to limit it. Although the application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the application.

Claims

1. A multi-modal fusion lightweight segmentation network for brain MRI images, characterized in that, Comprise: The encoding part contains four independent encoders to extract features from the original images of four modalities respectively, and different attention strategies are adopted for different modalities, wherein each encoder contains three convolution modules to down-sample through convolution and pooling layers; The feature fusion part fuses the features of the four modalities at the feature level, and different combinations are used at different feature layers, and light modal attention, spatial attention and channel attention are added, wherein the second layer features of the four encoders contain detailed information and high-level semantics, and the second layer features adopt a modal, channel and spatial joint attention mechanism during fusion, and the remaining two layer features only adopt a channel and spatial joint attention mechanism during fusion, so as to improve the segmentation accuracy of the model; And The decoding part uses convolution and up-sampling to restore the original resolution of the feature map, and the up-sampling is realized by transposed convolution.

2. The multi-modal fusion lightweight segmentation network according to claim 1, characterized in that: Before feature fusion, a channel attention mechanism is added to each layer of the encoder of the enhanced sequence to focus on the weights of different channels of the enhanced sequence.

3. The multi-modal fusion lightweight segmentation network according to claim 1, characterized in that: In the feature fusion part, the same layer features of the four encoders are connected by a jump connection.

4. The multi-modal fusion lightweight segmentation network according to claim 1, characterized in that: The core component of the feature fusion part is a light attention module ACSMB based on modal, channel and spatial joint, which successively adopts channel attention CA, spatial attention SA and modal attention MA to calculate the weights of channels, spaces and modalities.

5. The multi-modal fusion lightweight segmentation network according to claim 1, characterized in that: In the decoding part, the fused multi-modal features are decoded by the same decoder through three up-sampled pooling layers to output the segmentation result.

6. A segmentation method of a multi-modal fusion lightweight segmentation network, applied to the segmentation network of any one of claims 1-5, characterized in that, Mainly includes the following steps: S1: Two consecutive 3x3x3 convolutions with a step size of 1 are performed on the input original images T1, T2, T1ce, Flair sequence to generate a first feature map with a resolution equal to the original image and a channel number of 16; S2: The first feature map of T1, T2, T1ce, Flair sequence is first subjected to maximum pooling with a step size of 2, and then two consecutive 3x3x3 convolutions with a step size of 1 are performed to obtain a second feature map with a resolution of 1 / 2 of the original image and a channel number of 32; S3: The second feature map of T1, T2, T1ce, Flair sequence is subjected to maximum pooling with a step size of 2, and then two consecutive 3x3x3 convolutions with a step size of 1 are performed to obtain a third feature map with a resolution of 1 / 4 of the original image and a channel number of 64; S4: Two consecutive 3x3x3 convolutions with a step size of 1 are performed on the third feature map of T1, T2, T1ce, Flair sequence to generate a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 64; S5: The first feature map of the four sequences is subjected to channel dimension feature splicing to generate a feature map with a channel number of 64, and then subjected to two consecutive 3x3x3 convolutions with a step size of 1 to generate a fifth feature map with a resolution equal to the original image and a channel number of 16; S6: The second feature maps of the four sequences are concatenated in the channel dimension to generate a feature map with a channel number of 128, and then a lightweight self-attention module is used, followed by two consecutive 3*3*3 convolutions with a step of 1 to generate a sixth feature map with a resolution of 1 / 2 of the original image and a channel number of 32; S7: The third feature maps of the four sequences are concatenated in the channel dimension to generate a feature map with a channel number of 256, and then a lightweight self-attention module is used, followed by two consecutive 3*3*3 convolutions with a step of 1 to generate a seventh feature map with a resolution of 1 / 4 of the original image and a channel number of 64; S8: The fourth feature maps of the four sequences are concatenated in the channel dimension to generate a feature map with a channel number of 256, and then two consecutive 3*3*3 convolutions with a step of 1 are used to generate an eighth feature map with a resolution of 1 / 8 of the original image and a channel number of 128; S9: The eighth feature map is passed through a 3*3*3 transposed convolution with a step of 1 to generate a ninth feature map with a resolution of 1 / 4 of the original image and a channel number of 64; S10: The ninth feature map and the seventh feature map are concatenated in the channel dimension to generate a feature map with a channel number of 128, and then two consecutive 3*3*3 convolutions with a step of 1 are used to generate a tenth feature map with a resolution of 1 / 4 of the original image and a channel number of 64; S11: The tenth feature map is passed through a 3*3*3 transposed convolution with a step of 1 to generate an eleventh feature map with a resolution of 1 / 2 of the original image and a channel number of 32; S12: The eleventh feature map and the sixth feature map are concatenated in the channel dimension to generate a feature map with a channel number of 64, and then two consecutive 3*3*3 convolutions with a step of 1 are used to generate a twelfth feature map with a resolution of 1 / 2 of the original image and a channel number of 32; S13: The twelfth feature map is passed through a 3*3*3 transposed convolution with a step of 1 to generate a thirteenth feature map with a resolution equal to the original image and a channel number of 16; S14: The thirteenth feature map and the fifth feature map are concatenated in the channel dimension to generate a feature map with a channel number of 32, and then two consecutive 3*3*3 convolutions with a step of 1 are used to generate a fourteenth feature map with a resolution equal to the original image and a channel number of 16, which is the segmentation result map.

7. The segmentation method of claim 6, wherein, The step S1 specifically comprises: S11: Two consecutive 3*3*3 convolutions with a step of 1 are performed on the input original image T1 sequence to generate a first feature map with a resolution equal to the original image and a channel number of 16; S12: Two consecutive 3*3*3 convolutions with a step of 1 are performed on the input original image T2 sequence to generate a first feature map with a resolution equal to the original image and a channel number of 16; S13: Two consecutive 3*3*3 convolutions with a step of 1 are performed on the input original image T1ce sequence, and then a channel attention module is used to generate a first feature map with a resolution equal to the original image and a channel number of 16; S14: two consecutive 3*3*3 convolutions with a step size of 1 are performed on the input raw image Flair sequence to generate a first feature map with a resolution equal to that of the raw image and a channel number of 16.

8. The segmentation method of claim 6, wherein, The step S2 specifically comprises: S21: maximum value pooling with a step size of 2 is first performed on the first feature map of the T1 sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a second feature map with a resolution of 1 / 2 of the raw image and a channel number of 32; S22: maximum value pooling with a step size of 2 is first performed on the first feature map of the T2 sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a second feature map with a resolution of 1 / 2 of the raw image and a channel number of 32; S23: maximum value pooling with a step size of 2 is first performed on the first feature map of the T1ce sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed, and then a channel attention module is used to obtain a second feature map with a resolution of 1 / 2 of the raw image and a channel number of 32; S24: maximum value pooling with a step size of 2 is first performed on the first feature map of the Flair sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a second feature map with a resolution of 1 / 2 of the raw image and a channel number of 32.

9. The segmentation method of claim 6, wherein, The step S3 specifically comprises: S31: maximum value pooling with a step size of 2 is first performed on the second feature map of the T1 sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a third feature map with a resolution of 1 / 4 of the raw image and a channel number of 64; S32: maximum value pooling with a step size of 2 is first performed on the second feature map of the T2 sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a third feature map with a resolution of 1 / 4 of the raw image and a channel number of 64; S33: maximum value pooling with a step size of 2 is first performed on the second feature map of the T1ce sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed, and then a channel attention module is used to obtain a third feature map with a resolution of 1 / 4 of the raw image and a channel number of 64; S34: maximum value pooling with a step size of 2 is first performed on the second feature map of the Flair sequence, and then two consecutive 3*3*3 convolutions with a step size of 1 are performed to obtain a third feature map with a resolution of 1 / 4 of the raw image and a channel number of 64.

Images