Multi-modal medical image segmentation method based on deep learning

Abstract

The invention discloses a multi-modal medical image segmentation method based on deep learning. The method comprises the following steps: 1, obtaining a multi-modal medical image and standardizing the size of the multi-modal medical image; 2, inputting modals 1-3 into an FCSA fusion module of a channel 2 of the segmentation network to obtain a fused feature map; 3, inputting the dominant mode 4 into a channel 1 of the segmentation network; 4, extracting feature information of different scales through five layers of improved Tok-KAN modules and down-sampling of a channel 1 and a channel 2 of the segmentation network; 5, in a decoding process, realizing feature recovery and decoding of the coding feature patterns of different scales output by each layer through a dense jump connection decoder; and 6, through up-sampling and Sigmoid activation, generating segmentation prediction from the output of the last feature extractor of the decoder. The problem that a multi-modal medical image segmentation algorithm is low in segmentation accuracy and poor in model interpretability is solved.

Description

Technical Field

[0001] The present invention relates to a medical image segmentation method, and particularly to a multi-modal medical image segmentation method based on a dual-channel Kolmogorov - Arnold Network and multi-scale data fusion. Background Art

[0002] Medical image segmentation is one of the important tasks in the field of medical image analysis. Its goal is to accurately segment regions with specific anatomical or pathological significance (such as organs, lesions, etc.) in image data. The segmentation results can provide crucial support for disease diagnosis, treatment planning, and prognosis assessment. With the progress of medical imaging equipment, multi-modal images (such as multi-parametric MRI, combined CT and MRI imaging, ultrasound and photoacoustic imaging, etc.) have become an important part of modern medical images. These imaging modalities can provide rich tissue characteristic information from different physical or physiological perspectives. Segmenting by fusing multi-modal information has been proven to be an important method for improving the accuracy and reliability of medical image segmentation. Traditional medical image segmentation methods mostly rely on regularized image processing techniques and statistical models, such as region growing method, level set method, and graph cut segmentation method. These methods have poor adaptability to image noise and modality differences, and are difficult to handle complex anatomical structures and non-linear associations between modalities. In recent years, segmentation methods based on deep learning, with convolutional neural networks as the core, have greatly improved the performance of medical image segmentation.

[0003] Convolutional neural networks (CNNs) have shown excellent performance in medical image segmentation tasks due to their powerful feature extraction capabilities. Among them, the symmetric U-shaped encoder-decoder structure of the UNet network has become a milestone segmentation framework. Its variants such as UNet++, VNet, Attention-UNet, etc. further strengthen the network's full utilization of feature information by redefining skip connections, changing the network structure, and adding attention mechanisms, significantly improving the segmentation performance. However, they still have deficiencies in long-range modeling and image detail processing. Therefore, combining Transformer, which performs well in long-range modeling, with CNNs effectively alleviates these contradictions. TransUNet and SwinUNet designed special Transformer-based encoders and decoders. However, this method has a high model complexity and requires a large amount of training data, which is not very suitable for diseases with a low incidence rate like brain tumors and small datasets. The structured state space sequence model (SSM) has been proven to be a more suitable method, and U-Mamba and Seg-Mamba have demonstrated good segmentation performance in multi-modal medical image segmentation tasks. Since these methods still use convolutional kernels, Transformer, and multi-layer perceptrons (MLPs), resulting in insufficient ability to model feature non-linearity, their segmentation performance is not satisfactory when dealing with medical images containing complex non-linear features. In addition, these methods are generally considered a black box operation when dealing with segmentation tasks, and the interpretability of the model during the decision-making process is poor, which is not acceptable for medical image segmentation that requires clinical trust. In addition, multi-modal segmentation methods have been widely used in the field of medical image segmentation by extracting and integrating feature information from different modalities, such as ultrasound-ultrasound elastography, ultrasound-photoacoustic, multi-parametric MRI, etc. However, current multi-modal segmentation methods often ignore the potential relationships between different modality images, resulting in the generation of redundant information or the omission of key complementary information, thus leading to a decline in segmentation performance. Summary of the Invention

[0004] To solve the problems of low segmentation accuracy and poor model interpretability of multi-modal medical image segmentation algorithms, the present invention provides a multi-modal medical image segmentation method based on deep learning.

[0005] The object of the present invention is achieved through the following technical solutions:

[0006] A multi-modal medical image segmentation method based on deep learning, comprising the following steps:

[0007] Step 1: Obtain multi-modal medical images and standardize their sizes;

[0008] Step 2: Input modalities 1-3 into the FCSA fusion module in channel 2 of the segmentation network to obtain the fused feature map. The specific steps are as follows:

[0009] Step 2-1: Input modality 1, modality 2, and modality 3 into channel 2 of the segmentation network. The feature map generated by passing through the convolutional module is where H and W are the spatial dimensions, and C is the number of channels. Concatenate the feature maps of the three modalities along the channel dimension to obtain the multi-modal feature representation

[0010] X = Concat(X1, X2, X3)

[0011] where Concat(·) represents the concatenation operation;

[0012] Step 2-2: Sequentially pass the multi-modal feature representation X through the FCSA fusion module for weighted fusion. Use the fusion method guided by the multi-dimensional attention mechanism to efficiently extract and fuse the features of different modalities from the frequency-channel dimension and the spatial dimension. For the feature X after frequency-channel weighting FC and the feature X after spatial weighting S add them and perform a linear operation to change the number of channels of the fused feature to the same as the input modality dimension, obtaining the final feature map processed by the FCSA fusion module

[0013] X FCSA = Linear(X FC + X S )

[0014] where Linear(·) is a linear operation used to reduce the number of channels;

[0015] Step 3: Input the dominant modality 4 into channel 1 of the segmentation network;

[0016] Step 4: Both channel 1 and channel 2 of the segmentation network pass through five improved Tok-KAN modules and downsampling to extract feature information of different scales. Among them: the size of the feature map output at each stage is halved, and the number of channels is doubled; the improved Tok-KAN module is an encoder based on KAN and attention. During the encoding process, the output of the encoder in channel 2 will be fused with the output of the encoder in channel 1 at the corresponding scale and input into the encoding process of the next stage of channel 1 to realize the complement of the feature information of modalities 1-3 to the dominant modality 4;

[0017] Step 5: During the decoding process, the encoded feature maps of different scales output by each layer are decoded through the decoder with dense skip connections to realize feature recovery and decoding;

[0018] Step 6: Generate segmentation predictions from the output of the last feature extractor of the decoder through upsampling and Sigmoid activation.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] 1. In order to effectively utilize the information of multimodal medical images, the present invention designs a dual-channel input strategy, and introduces an FCSA fusion module into the input of Channel 2 to utilize multi-dimensional attention to guide the efficient fusion of multimodal information.

[0021] 2. In order to strengthen the model's ability to extract non-linear features and the interpretability of the model, the present invention uses the improved Tok-KAN module as the encoder of the model, and embeds the lightweight CSA module into Tok-KAN, which not only improves the segmentation accuracy, but also can improve the convergence of the model.

[0022] 3. In order to avoid the loss of detailed features during the downsampling and upsampling processes, the present invention adopts a decoder structure with dense skip connections for the fusion and interaction of multi-scale information, and realizes high-precision segmentation predictions.

[0023] 4. The quantitative and qualitative evaluation results on the open-source multi-parameter MRI brain tumor BraTS19 dataset and BraTS20 dataset show that the method proposed by the present invention has better segmentation performance compared with several traditional segmentation methods and deep learning-based segmentation methods. More importantly, it has good model interpretability. Brief Description of the Drawings

[0024] Figure 1 It is a flowchart of a deep learning-based multimodal medical image segmentation method;

[0025] Figure 2 It is a deep learning-based multimodal medical image segmentation framework;

[0026] Figure 3 It is an FCSA fusion module framework;

[0027] Figure 4 It is a structural framework of a lightweight channel spatial attention (CSA) module;

[0028] Figure 5 It is the segmentation result of the multimodal medical image segmentation method on the multi-parameter MRI brain tumor dataset. Detailed Embodiments

[0029] The technical solutions of the present invention will be further described below with reference to the accompanying drawings, but are not limited thereto. Any modifications or equivalent replacements to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention shall be covered within the protection scope of the present invention.

[0030] The present invention provides a multi-modal medical image segmentation method based on deep learning. The main process of the method is as Figure 1 shown, and the overall framework is as Figure 2 shown. Its core includes an FCSA fusion module, a Tok-KAN Block, and a multi-scale feature fusion structure. The specific steps are as follows:

[0031] Step 1: Input modality 1, modality 2, and modality 3 into channel 2 of the segmentation network. The feature maps generated by passing through the convolutional module are where H and W are the spatial dimensions, and C is the number of channels. Concatenate the feature maps of the three modalities along the channel dimension to obtain a multi-modal feature representation

[0032] X = Concat(X1, X2, X3) (1)

[0033] where Concat(·) represents the concatenation operation.

[0034] Step 2: Sequentially pass the multi-modal feature representation X through the FCSA fusion module for weighted fusion. Use the fusion method guided by the multi-dimensional attention mechanism to efficiently extract and fuse the features of different modalities from the frequency-channel dimension and the spatial dimension, emphasizing the key information while suppressing the generation of redundant information and improving the feature utilization rate. As Figure 3 shown, the FCSA fusion module includes frequency-channel attention and spatial attention, which are connected in series. Finally, the feature X FC after frequency-channel weighting and the feature X S after spatial weighting are added and the number of channels of the fused features is changed to the same as the input modality dimension through a linear operation to obtain the final feature map processed by the FCSA fusion module as shown in (2):

[0035] X FCSA = Linear(X FC + X S ) (2)

[0036] where Linear(·) is a linear operation used to reduce the number of channels. This spatial attention based on the global self-attention mechanism can effectively capture the dependence relationships between distant pixels, enhancing the model's understanding of the overall structure of the image and its processing ability for complex scenes.

[0037] Step 3: Input the dominant modality 4 into channel 1 of the segmentation network.

[0038] Step 4: Both Channel 1 and Channel 2 go through five improved Tok-KAN modules and downsampling to extract feature information at different scales. The size of the feature map output at each stage is halved, and the number of channels is doubled. Among them, the improved Tok-KAN module is an encoder based on KAN and attention. By introducing a lightweight channel spatial attention (CSA) module into Tok-KAN, the feature extraction ability of the encoder can be effectively improved, and the convergence of the network can be improved to a certain extent. During the encoding process, the output of the encoder in Channel 2 will be fused with the output of the encoder in Channel 1 at the corresponding scale and input into the encoding process of the next stage of Channel 1 to realize the complement of the feature information of Modes 1-3 to the dominant Mode 4. The improved Tok-KAN first divides the input data into small pieces and generates multiple embedding vectors. Secondly, each embedding vector is tokenized. Then, the tokenized embedding vectors are input into the KAN Layer, and the model performs non-linear transformation on each token through multiple learnable univariate B-spline basis functions to capture and represent complex features. Further, depthwise convolution (DwConv) performs independent in-channel convolution operations on each token. To enhance the network's ability to model the importance of features and at the same time improve the training difficulty of the network, a CSA block is added after DwConv. The structure of CSA is as Figure 4 shown, including channel attention and spatial attention. The CSA block first expands in the spatial dimension and reduces the number of channels with a fully connected layer to make the attention more lightweight. Secondly, another fully connected layer is used to restore the number of channels. Then, the channel weights are rearranged into the same dimension as the input feature map to obtain the feature map weighted by channel attention. Furthermore, the spatial attention weights are obtained by performing convolution and activation on the feature map twice continuously. Finally, the feature map weighted by channel attention is multiplied by the spatial attention weights to obtain the feature map processed by the CSA block. The Tok-KAN module based on KAN and the attention mechanism can significantly improve the ability to model non-linear information and strengthen the training process of the network through attention.

[0039] Step 5. During the decoding process, the encoded feature maps of different scales output by each layer are restored and decoded through a decoder with dense skip connections. The decoder consists of multiple feature extractors and upsampling modules, which flexibly fuse feature information of different scales through dense skip connections, and effectively reduce the semantic difference between downsampling and upsampling. Among them, the feature extractor performs convolution operations on feature maps of different scales to extract deeper semantic information, that is: according to the current scale i and stride j, the feature extractor will perform convolution, batch normalization (Batch Normalization), and ReLU activation function processing on the input feature map twice continuously. The extracted information will be passed to the next level or fused with the high-resolution feature map through upsampling. The upsampling restores the resolution using bilinear interpolation, and the generated feature map will be concatenated with the output of the encoder or the feature map of the previous layer to enhance the information integration effect, thereby maintaining the integrity of spatial details.

[0040] Step 6. Through upsampling and Sigmoid activation, segmentation predictions are generated from the output of the last feature extractor of the decoder. The multi-feature fusion retains deep semantic information through iterative upsampling and cross-scale feature fusion, and maintains the spatial resolution during the gradual restoration process. This design enables the network to effectively utilize multi-scale feature information and significantly improve the prediction performance and detail restoration ability.

[0041] Example:

[0042] In this example, the brain tumor segmentation experiment was carried out on the BraTS2019 and BraTS2020 open-source datasets using the multi-modal medical image segmentation method proposed in the present invention. The T1, T1ce, T2, and FLAIR brain multi-parameter MRI images were directly connected to the image input interface of the segmentation method. After experimental testing, the segmentation results of the enhanced tumor region (ET), tumor core region (TC), and whole tumor region (WT) of the brain tumor and the corresponding labels are as Figure 5 shown, and the quantitative indicators of the mainstream methods are shown in Table 1. Compared with other mainstream methods, the method of the present invention significantly improves the segmentation accuracy.

[0043] Table 1 Quantitative indicators of different fusion methods

[0044]

[0045]

Claims

1. A multi-modal medical image segmentation method based on deep learning, characterized in that The method includes the following steps: Step 1: Obtain multimodal medical images and standardize their sizes; Step 2: Input modalities 1 to 3 into the FCSA fusion module in channel 2 of the segmentation network to obtain the fused feature map; Step 3: Input the dominant modality 4 into channel 1 of the segmentation network; Step 4: Both channel 1 and channel 2 of the segmentation network pass through five improved Tok-KAN modules and downsampling to extract feature information at different scales, where: the size of the feature map output at each stage is halved and the number of channels is doubled; the improved Tok-KAN module is an encoder based on KAN and attention. During the encoding process, the output of the encoder in channel 2 will be fused with the output of the encoder in channel 1 at the corresponding scale and input into the encoding process of the next stage of channel 1, realizing the supplementation of the feature information of modalities 1 to 3 to the dominant modality 4; Step 5: During the decoding process, the encoded feature maps at different scales output by each layer are decoded through the decoder with dense skip connections to recover the features; Step 6: Through upsampling and Sigmoid activation, generate the segmentation prediction from the output of the last feature extractor of the decoder.

2. The multi-modal medical image segmentation method based on deep learning according to claim 1, characterized in that The specific steps of step 2 are as follows: Step 2-1: Input modality 1, modality 2, and modality 3 into channel 2 of the segmentation network. The feature map generated by the convolutional module is where H and W are the spatial dimensions and C is the number of channels. Concatenate the feature maps of the three modalities along the channel direction to obtain the multi-modal feature representation X = Concat(X1, X2, X3) Among them, C onca t ( · ) represents a splicing operation; Step 22: Multimodal feature representation X Successively pass through the FCSA fusion module for weighted fusion. Use the fusion method guided by the multi-dimensional attention mechanism to efficiently extract and fuse features of different modalities from the frequency-channel dimension and the spatial dimension. The features after frequency-channel weighting XFC And the features after spatial weighting XS Are added together and the number of channels of the fused features is changed to the same as the input modality dimension through a linear operation to obtain the final feature map processed by the FCSA fusion module X FCSA = Linear(X FC + X S ) Among them, Linear(·) is a linear operation for reducing the dimensionality of the number of channels.

3. The multimodal medical image segmentation method based on deep learning according to claim 1, wherein The FCSA fusion module includes frequency-channel attention and spatial attention, which are connected in series.

4. The multi-modal medical image segmentation method based on deep learning according to claim 1, characterized in that The improved Tok-KAN divides the input data into small pieces and generates multiple embedding vectors; each embedding vector is tokenized, and the tokenized embedding vectors are input into the KAN Layer. Nonlinear transformation is performed on each token through multiple learnable univariate B-spline basis functions to capture and represent complex features; DwConv performs independent intra-channel convolution operations on each token; in order to enhance the network's ability to model the importance of features and at the same time improve the training difficulty of the network, a CSA block is added after DwConv. The CSA block first unfolds in the spatial dimension and reduces the number of channels with a fully connected layer to make the attention more lightweight, then restores the number of channels with another fully connected layer, and then rearranges the channel weights into the same dimension as the input feature map to obtain the feature map weighted by channel attention. The spatial attention weight is obtained by performing convolution and activation on the feature map twice continuously. The feature map weighted by channel attention is multiplied by the spatial attention weight to obtain the feature map processed by the CSA block.

5. The multi-modal medical image segmentation method based on deep learning according to claim 1, wherein The decoder consists of multiple feature extractors and upsampling modules, which flexibly fuse feature information at different scales through dense skip connections.

6. The multi-modal medical image segmentation method based on deep learning according to claim 5, wherein The feature extractor performs convolution, batch normalization, and ReLU activation function processing on the input feature map twice according to the current scale i and stride j. The extracted information will be passed to the next level or fused with the high-resolution feature map through upsampling; Upsampling uses bilinear interpolation to restore the resolution, and the generated feature map will be concatenated with the output of the encoder or the feature map of the previous layer to enhance the information integration effect and thus maintain the integrity of spatial details.

Images