Deep learning based multi-modal medical image segmentation method

Abstract

The application discloses a kind of based on deep learning multi-modal medical image segmentation method, the method includes the following steps: one, obtains multi-modal medical image and standardizes size;Two, modality 1-3 are input into the FCSA fusion module of the channel 2 of segmentation network and obtain the feature map after fusion;Three, leading modality 4 is input into the channel 1 of segmentation network;Four, the channel 1 and channel 2 of segmentation network are all extracted by five layers improved Tok-KAN module and down sampling to different scale feature information;Five, in decoding process, the different scale encoding feature map of each layer output is recovered and decoded by the decoder of dense jump connection;Six, by upsampling and Sigmoid activation, generate segmentation prediction from the output of the last feature extractor of decoder.The application solves the problems of low segmentation accuracy and poor model interpretability of multi-modal medical image segmentation algorithm.

Description

TECHNICAL FIELD

[0001] The application relates to a medical image segmentation method, in particular to a multi-modal medical image segmentation method based on a dual-channel Kolmogorov-Arnold network and multi-scale data fusion. BACKGROUND

[0002] Medical image segmentation is one of the important tasks in medical image analysis, which aims to accurately segment regions with specific anatomical or pathological significance (such as organs, lesions, etc.) in image data. The segmentation results can provide key support for disease diagnosis, treatment planning and prognosis evaluation. With the advancement of medical imaging equipment, multi-modal images (such as multi-parameter MRI, CT and MRI combined imaging, ultrasound and photoacoustic imaging, etc.) have become an important part of modern medical imaging. These imaging modalities can provide rich tissue characteristic information from different physical or physiological perspectives. Fusing multi-modal information for segmentation has been proven to be an important method to improve the accuracy and reliability of medical image segmentation. Traditional medical image segmentation methods mostly rely on regularized image processing techniques and statistical models, such as region growing, level set and graph cuts. These methods have poor adaptability to image noise and modal differences, and are difficult to handle complex anatomical structures and nonlinear relationships between modalities. In recent years, deep learning-based segmentation methods, with convolutional neural networks as the core, have greatly improved the performance of medical image segmentation.

[0003] Convolutional neural networks (CNNs) have shown great performance in medical image segmentation tasks due to their powerful feature extraction ability. Among them, the symmetrical U-shaped encoding-decoding structure of UNet network has become a milestone in segmentation framework. Its variants such as UNet++, VNet, Attention-UNet, etc. further strengthen the network's full use of feature information by redefining cross-link, changing network structure and adding attention mechanism, which significantly improves the segmentation performance. However, they still have deficiencies in long-distance modeling and image detail processing. Therefore, the combination of Transformer, which performs well in long-distance modeling, and CNNs effectively alleviates these contradictions. TransUNet and SwinUNet design special Transformer-based encoders and decoders, but this method has high model complexity and requires a large amount of training data, which is not suitable for diseases such as brain tumors with low incidence and small data sets. Structured state space sequence model (SSM) has been proven to be a more suitable method, and U-Mamba and Seg-Mamba have shown good segmentation performance in multiple modal medical image segmentation tasks. Since these methods still use convolution kernels, Transformers and multi-layer perceptrons (MLP), they lack the ability to model non-linear features, so their segmentation performance is not satisfactory when dealing with medical images containing complex non-linear features. In addition, these methods are generally considered a black box operation when processing segmentation tasks, and the model's explainability in the decision-making process is poor, which is unacceptable for medical image segmentation that requires clinical trust. In addition, multi-modal segmentation methods have been widely used in medical image segmentation by extracting and integrating feature information from different modalities, such as ultrasound-ultrasound elasticity, ultrasound-photoacoustic, multi-parameter MRI, etc. However, current multi-modal segmentation methods often ignore the potential relationship between different modal images, resulting in the generation of redundant information or the omission of key complementary information, which leads to a decrease in segmentation performance. SUMMARY

[0004] In order to solve the problems of low segmentation accuracy and poor model explainability of multi-modal medical image segmentation algorithms, the present application provides a multi-modal medical image segmentation method based on deep learning.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A multi-modal medical image segmentation method based on deep learning, comprising the following steps:

[0007] Step one, obtain multi-modal medical images and standardize the size;

[0008] Step two, input modalities 1-3 into the FCSA fusion module of channel 2 of the segmentation network to obtain the fused feature map, the specific steps are as follows:

[0009] Step two, input modalities 1, 2 and 3 into channel 2 of the segmentation network, and generate a feature map through the convolution module Where H and W are spatial dimensions, and C is the number of channels. The feature maps of the three modalities are concatenated in the channel direction to obtain a multi-modal feature representation

[0010] X = Concat(X1, X2, X3)

[0011] Where Concat(·) represents the concatenation operation.

[0012] Step two, sequentially pass the multi-modal feature representation X through the FCSA fusion module for weighted fusion, and use the multi-dimensional attention mechanism guided fusion method to efficiently extract and fuse the features of different modalities from the frequency-channel dimension and the spatial dimension. The feature X FC and the spatially weighted feature X S are added, and the channel number of the fused feature is changed to the same as the input modality dimension through linear operation, to obtain the final feature map processed by the FCSA fusion module

[0013] X FCSA = Linear(X FC +X S )

[0014] Where Linear(·) is a linear operation for dimension reduction.

[0015] Step three, input the dominant modality 4 into channel 1 of the segmentation network.

[0016] Step four, the channels 1 and 2 of the segmentation network are processed through five improved Tok-KAN modules and down-sampling to extract feature information of different scales, wherein: the size of the feature map output by each stage is halved, and the number of channels is doubled; the improved Tok-KAN module is an encoder based on KAN and attention. In the encoding process, the output of the encoder in channel 2 will be fused with the output of the encoder in channel 1 of the corresponding scale, and input into the next stage of the encoding process in channel 1, to realize the complement of the feature information of modalities 1-3 to the dominant modality 4.

[0017] Step five, in the decoding process, the encoded feature maps of different scales output by each layer are restored and decoded through the dense skip connection decoder.

[0018] Step six, generate segmentation prediction from the output of the last feature extractor of the decoder by upsampling and sigmoid activation.

[0019] Compared with the prior art, the present application has the following advantages:

[0020] 1. In order to effectively utilize the information of multi-modal medical images, the present application designs a dual-channel input strategy, and introduces a FCSA fusion module in the input of channel 2, which uses multi-dimensional attention to guide the efficient fusion of multi-modal information.

[0021] 2. In order to strengthen the model's ability to extract nonlinear features and the model's interpretability, the present application uses an improved Tok-KAN module as the model's encoder, and embeds a lightweight CSA module into Tok-KAN, which not only improves the accuracy of segmentation, but also improves the convergence of the model.

[0022] 3. In order to avoid the loss of detailed features in the down-sampling and up-sampling processes, the present application uses a decoder structure with dense skip connections for multi-scale information fusion and interaction, achieving high-precision segmentation prediction.

[0023] 4. The quantitative and qualitative evaluation results on the open-source multi-parameter MRI brain tumor BraTS19 dataset and BraTS20 dataset show that the method proposed by the present application has better segmentation performance than several traditional segmentation methods and deep learning-based segmentation methods, and more importantly, has good model interpretability. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Flowchart of the multi-modal medical image segmentation method based on deep learning;

[0025] Figure 2 Multi-modal medical image segmentation framework based on deep learning;

[0026] Figure 3 FCSA fusion module framework;

[0027] Figure 4 Structure framework of the lightweight channel spatial attention (CSA) module;

[0028] Figure 5 Segmentation results of the multi-modal medical image segmentation method on the multi-parameter MRI brain tumor dataset. DETAILED DESCRIPTION

[0029] The technical solutions of the present application will be further described below in conjunction with the drawings, but are not limited thereto. Any modification or equivalent replacement to the technical solutions of the present application without departing from the spirit and scope of the present application shall be covered in the protection scope of the present application.

[0030] The application provides a deep learning-based multi-modal medical image segmentation method, the main process of the method is as shown in Figure 1 The overall framework is as shown in Figure 2 The core includes an FCSA fusion module, a Tok-KAN Block and a multi-scale feature fusion structure, and the specific steps are as follows:

[0031] Step one, input modal 1, modal 2 and modal 3 into channel 2 of the segmentation network, and generate a feature map through a convolution module Where H and W are spatial dimensions, and C is the number of channels. The feature maps of the three modalities are spliced in the channel direction to obtain a multi-modal feature representation

[0032] X = Concat (X1, X2, X3) (1)

[0033] Where, Concat(·) represents a splicing operation.

[0034] Step two, sequentially pass the multi-modal feature representation X through the FCSA fusion module for weighted fusion, and use the fusion mode guided by the multi-dimensional attention mechanism to efficiently extract and fuse the features of different modalities from the frequency-channel dimension and the spatial dimension, emphasize the key information and also suppress the generation of redundant information, and improve the feature utilization rate. As shown in Figure 3 The FCSA fusion module includes frequency-channel attention and spatial attention, which are connected in series. Finally, the features X FC and the features X S after frequency-channel weighting are added, and the channel number of the fused features is changed to the same as the input modality dimension through linear operation, to obtain the feature map after processing by the FCSA fusion module As shown in equation (2):

[0035] X FCSA = Linear (X FC + X S ) (2)

[0036] Where, Linear(·) is a linear operation for dimension reduction. This spatial attention based on global self-attention mechanism can effectively capture the dependency relationship between distant pixels, enhance the understanding of the overall structure of the image by the model and the processing ability of complex scenes.

[0037] Step three, input the dominant modality 4 into channel 1 of the segmentation network.

[0038] Step 4: Both Channel 1 and Channel 2 are processed through a five-layer improved Tok-KAN module and downsampling to extract feature information at different scales. The size of the output feature map is halved and the number of channels is doubled in each stage. The improved Tok-KAN module is a KAN-based encoder with attention. Introducing a lightweight Channel Spatial Attention (CSA) module into Tok-KAN effectively improves the encoder's feature extraction capability and, to some extent, improves the network's convergence. During encoding, the output of the encoder in Channel 2 is fused with the output of the encoder in Channel 1 at the corresponding scale and input into the next stage of encoding for Channel 1, thus supplementing the dominant mode 4 with the feature information of modes 1-3. The improved Tok-KAN first divides the input data into small blocks and generates multiple embedding vectors. Next, each embedding vector is tokenized. Then, the tokenized embedding vectors are input into the KAN Layer, where the model performs nonlinear transformations on each token using multiple learnable univariate B-spline basis functions to capture and represent complex features. Furthermore, depthwise convolution (DwConv) performs independent intra-channel convolution operations on each label. To enhance the network's ability to model the importance of features and reduce the training difficulty, a CSA block is added after DwConv. The structure of CSA is as follows: Figure 4 As shown, this includes channel attention and spatial attention. The CSA block first expands in the spatial dimension and uses a fully connected layer to reduce the number of channels, making the attention more lightweight. Next, another fully connected layer restores the number of channels. Then, the channel weights are rearranged to the same dimension as the input feature map, resulting in a channel attention-weighted feature map. Subsequently, the spatial attention weights are obtained by performing two consecutive convolutions and activations on the feature map. Finally, the channel attention-weighted feature map is multiplied by the spatial attention weights to obtain the feature map processed by the CSA block. The Tok-KAN module, based on KAN and attention mechanisms, significantly improves the ability to model nonlinear information and enhances the network training process through attention.

[0039] Step five, in the decoding process, the different scale encoded feature maps output by each layer are recovered and decoded by the decoder with dense skip connections. The decoder is composed of multiple feature extractors and up-sampling modules, which realize the flexible fusion of different scale feature information through dense skip connections, and effectively reduce the semantic difference between down-sampling and up-sampling. Among them, the feature extractor performs convolution operation on the feature maps of different scales, so as to extract deeper semantic information, that is, according to the current scale i and the step j, the feature extractor will perform convolution, batch normalization (Batch Normalization) and ReLU activation function processing on the input feature map for two times in succession, and the extracted information will be passed to the next level, or fused with the high-resolution feature map through up-sampling. The up-sampling recovers the resolution by using bilinear interpolation, and the generated feature map is spliced with the encoder output or the feature map of the previous layer to enhance the integration effect of information, so as to maintain the integrity of spatial details.

[0040] Step six, the segmentation prediction is generated from the output of the last feature extractor of the decoder through up-sampling and Sigmoid activation. The multi-feature fusion retains the deep semantic information through iterative up-sampling and cross-scale feature fusion, and maintains the spatial resolution in the gradual restoration process. This design enables the network to effectively utilize multi-scale feature information, significantly improving the prediction performance and detail restoration ability.

[0041] Embodiment:

[0042] This embodiment uses the multi-modal medical image segmentation method proposed in the present application to perform brain tumor segmentation experiments on the BraTS2019 and BraTS2020 open source datasets. The T1, T1ce, T2 and FLAIR brain multi-parameter MRI images are directly connected to the image input interface of the segmentation method. Through experimental testing, the segmentation results of the enhanced tumor region (ET), tumor core region (TC) and whole tumor region (WT) of the brain tumor and the corresponding labels are as shown in Figure 5 The quantitative indicators of the mainstream method are shown in Table 1. Compared with other mainstream methods, the method of the present application significantly improves the accuracy of segmentation.

[0043] Table 1 Quantitative indicators of different fusion methods

[0044]

[0045]

Claims

1. A multimodal medical image segmentation method based on deep learning, characterized in that... The method includes the following steps: Step 1: Acquire multimodal medical images and standardize their dimensions; Step 2: Input modalities 1-3 into the FCSA fusion module of channel 2 of the segmentation network to obtain the fused feature map. The specific steps are as follows: Step 2: Input mode 1, mode 2, and mode 3 into channel 2 of the segmentation network. The feature map generated by the convolution module is as follows: Where H and W are spatial dimensions, and C is the number of channels, the feature maps of the three modalities are concatenated along the channel direction to obtain a multimodal feature representation. : in, Indicates a splicing operation; Step 22: Represent the multimodal features The features are sequentially weighted and fused through the FCSA fusion module. A multi-dimensional attention mechanism-guided fusion method efficiently extracts and fuses features from different modalities in both the frequency-channel and spatial dimensions. The frequency-channel weighted features are then processed... and spatially weighted features The features are summed and then linearly processed to make the number of channels of the fused features the same as the dimension of the input modalities, resulting in the final feature map processed by the FCSA fusion module. : in, This is a linear operation used to reduce the dimensionality of the number of channels; The FCSA fusion module includes frequency-channel attention and spatial attention, which are connected in series. Step 3: Input the dominant mode 4 into channel 1 of the segmentation network; Step 4: Channels 1 and 2 of the segmentation network are processed through a five-layer improved Tok-KAN module and downsampling to extract feature information at different scales. Specifically, the size of the output feature map is halved and the number of channels is doubled at each stage. The improved Tok-KAN module is a KAN-based encoder with attention. During the encoding process, the output of the encoder in channel 2 is fused with the output of the encoder in channel 1 at the corresponding scale and input into the next stage of encoding of channel 1, so as to supplement the feature information of modes 1 to 3 to the dominant mode 4. The improved Tok-KAN divides the input data into small blocks and generates multiple embedding vectors. Each embedding vector is then tokenized and input into the KAN Layer. Multiple learnable univariate B-spline basis functions are used to perform nonlinear transformations on each token, enabling the capture and representation of complex features. DwConv performs independent intra-channel convolution operations on each token. To enhance the network's ability to model the importance of features and reduce training difficulty, a CSA block is added after DwConv. The CSA block is first expanded in the spatial dimension and its channel count is reduced using a fully connected layer, making the attention more lightweight. Another fully connected layer restores the channel count. Then, the channel weights are rearranged to the same dimension as the input feature map, resulting in a channel attention-weighted feature map. The spatial attention weights are obtained by performing two consecutive convolutions and activations on the feature map. The channel attention-weighted feature map is multiplied by the spatial attention weights to obtain the feature map processed by the CSA block. Step 5: During the decoding process, the encoded feature maps of different scales output by each layer are restored and decoded through a decoder with dense skip connections. Step 6: Generate segmentation predictions from the output of the last feature extractor of the decoder through upsampling and Sigmoid activation.

2. The deep learning-based multimodal medical image segmentation method according to claim 1, characterized in that... The decoder consists of multiple feature extractors and upsampling modules, which achieve flexible fusion of feature information at different scales through dense skip connections.

3. The deep learning-based multimodal medical image segmentation method according to claim 2, characterized in that... The feature extractor performs two consecutive convolutions, batch normalization, and ReLU activation on the input feature map based on the current scale i and stride j. The extracted information will be passed to the next level or fused with the high-resolution feature map through upsampling. Upsampling uses bilinear interpolation to restore resolution. The generated feature map is then stitched together with the encoder output or the feature map from the previous layer to enhance information integration and thus maintain the integrity of spatial details.

Images