Abdomen multi-organ segmentation method and device based on multi-slice attention

Abstract

The invention relates to an abdomen multi-organ segmentation method and device based on multi-slice attention, and the method comprises the steps: constructing an abdomen multi-organ segmentation model which comprises a plurality of two-dimensional convolutional neural networks, a plurality of multi-slice attention modules and a UNet network, constructing a multi-slice attention module based on the mixing of CNN and Transformer, the module can be flexibly inserted into different segmentation network trunks based on convolution to form a new network framework, and the new network framework not only can better model long-distance feature dependence by using a self-attention mechanism of Transform, but also still has the strong capability of modeling local features by convolution operation in an original trunk. Experimental results show that the multi-slice attention module can improve the accuracy of abdomen multi-organ segmentation under the condition that special network design is not needed.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular, to a method and device for abdominal multi-organ segmentation based on multi-slice attention. Background Art

[0002] In recent years, with the development of deep learning, deep learning models represented by Convolutional Neural Networks (CNNs) use a large number of annotated training images to automatically extract image features without complex non-rigid registration, greatly improving the accuracy of medical image segmentation. Inspired by the fact that clinical radiologists need multiple adjacent CT slices to locate organs on a single CT slice, most existing methods take multiple adjacent two-dimensional CT slices as the input of a two-dimensional network architecture, or directly adopt a three-dimensional network design with three-dimensional volumes as the input to extract more three-dimensional context information. Although both 2D and 3D methods have achieved good segmentation performance, 2D segmentation methods based on multi-slice input are more popular than pure 3D methods because 3D networks face huge computational and storage costs, and publicly available 3D medical datasets are not sufficient for robust 3D pre-training.

[0003] Although 2D methods based on multi-slice input have shown good performance in different applications, they still have inherent limitations. 1) Weak global context modeling within slices: Most methods use convolutional neural networks as the backbone to extract image features as powerful image representations. Although CNNs are powerful in local feature extraction, they are limited in dealing with global context. 2) Weak context modeling between slices: When processing multi-slice input, most methods perform feature extraction on each slice using independent two-dimensional convolutions respectively, and then use convolutional operations for feature fusion. This convolution-based fusion method can handle local features well, but is still poor in dealing with global features between different slices. To solve this problem, some methods propose to reshuffle the feature channels between different slices. Although this approach alleviates the problem to some extent, due to relying on pure convolutional operations, they still have obvious deficiencies in capturing rich global context information. Summary of the Invention

[0004] To overcome at least one deficiency in the prior art, this application provides a method and device for abdominal multi-organ segmentation based on multi-slice attention.

[0005] In a first aspect, a method for abdominal multi-organ segmentation based on multi-slice attention is provided, including:

[0006] Obtain a model training dataset. The samples in the model training dataset are abdominal organ image slices, and the abdominal organ image slices include key slices and multiple adjacent slices of the key slices;

[0007] Train an abdominal multi-organ segmentation model based on the model training dataset to obtain a trained abdominal multi-organ segmentation model; The abdominal multi-organ segmentation model includes multiple two-dimensional convolutional neural networks, multiple multi-slice attention modules, and a UNet network. The samples are sequentially subjected to feature extraction through multiple two-dimensional convolutional neural networks, and the features output by each two-dimensional convolutional neural network Figure 1 are correspondingly input into multiple multi-slice attention modules for global information extraction and local information extraction, and the extracted global information and local information are fused to output fused features; All the fused features output by multiple multi-slice attention modules are input into the UNet network to obtain an organ segmentation map corresponding to the sample;

[0008] Input the abdominal organ image slices to be segmented into the trained abdominal multi-organ segmentation model to obtain an organ segmentation map corresponding to the abdominal organ image slices to be segmented.

[0009] In one embodiment, the multi-slice attention module includes a local information extraction block and a global information extraction block;

[0010] The feature map is input into the local information extraction block to output local information;

[0011] The feature map is input into the global information extraction block to output global information;

[0012] After the local information and global information are fused and subjected to a convolution operation, fused features are output.

[0013] In one embodiment, the local information extraction block includes a channel shuffle module and a convolutional layer. The channel shuffle module divides the feature map along the channel dimension into multiple groups, and performs depthwise separable convolution on each group to extract local features; The local features of each group are concatenated along the channel dimension to obtain the concatenated features;

[0014] After the concatenated features pass through the convolutional layer, local information is obtained.

[0015] In one embodiment, the global information extraction block includes a patch embedding extraction block and a slice feature attention block;

[0016] The patch embedding extraction block includes a single-slice patch embedding module and a full-slice patch embedding module. The single-slice patch embedding module is used to perform single-slice patch embedding on different slice feature maps respectively to obtain the patch embedding of each slice feature map, and the full-slice patch embedding module is used to perform full-patch embedding on the feature maps of all slices to obtain the full-slice patch embedding;

[0017] The patch embedding extraction block is also used to generate query vectors and key vectors based on the patch embeddings of the key slice feature maps and the full-slice patch embeddings; generate value vectors based on the patch embeddings of all adjacent slice feature maps;

[0018] The slice feature attention block includes a multi-head self-attention (MSA) and a multi-layer perceptron (MLP); the query vectors, key vectors, and value vectors are input into the multi-head self-attention (MSA) to obtain self-attention features. After the self-attention features are element-wise added to the value vectors, a first addition result is obtained. The first addition result is input into the multi-layer perceptron (MLP), and the output of the multi-layer perceptron (MLP) is element-wise added to the first addition result to obtain a second addition result; an interpolation operation is performed on the second addition result to output global information.

[0019] In one embodiment, the single-slice patch embedding module is used to perform two-dimensional convolution, average pooling, and shape reshaping on the slice feature map to obtain the patch embedding of the slice feature map.

[0020] In one embodiment, the full-slice patch embedding module is used to perform three-dimensional convolution, dimension compression, and shape reshaping on the feature maps of all slices to obtain the full-slice patch embedding.

[0021] In a second aspect, there is provided an abdominal multi-organ segmentation device based on multi-slice attention, including:

[0022] A dataset acquisition module, configured to acquire a model training dataset, where the samples in the model training dataset are abdominal organ image slice groups, and the abdominal organ image slice groups include key slices and multiple adjacent slices of the key slices;

[0023] A model training module, configured to train an abdominal multi-organ segmentation model based on the model training dataset to obtain a trained abdominal multi-organ segmentation model; the abdominal multi-organ segmentation model includes multiple two-dimensional convolutional neural networks, multiple multi-slice attention modules, and a UNet network. The samples are sequentially subjected to feature extraction by multiple two-dimensional convolutional neural networks, and the features output by each two-dimensional convolutional neural network Figure 1 are correspondingly input into multiple multi-slice attention modules for global information extraction and local information extraction, and the extracted global information and local information are fused to output fused features; all the fused features output by the multiple multi-slice attention modules are input into the UNet network to obtain an organ segmentation map corresponding to the sample;

[0024] A prediction module, configured to input the abdominal organ image slice group to be segmented into the trained abdominal multi-organ segmentation model to obtain an organ segmentation map corresponding to the abdominal organ image slice group to be segmented.

[0025] In one embodiment, the multi-slice attention module includes a local information extraction block and a global information extraction block;

[0026] The feature map is input into the local information extraction block, and local information is output;

[0027] The feature map is input into the global information extraction block, and global information is output;

[0028] After the local information and the global information are fused and then subjected to a convolution operation, the fused feature is output.

[0029] In one embodiment, the local information extraction block includes a channel shuffle module and a convolutional layer. The channel shuffle module divides the feature map into multiple groups along the channel dimension, and performs depthwise separable convolution on each group to extract local features; the local features of each group are concatenated along the channel dimension to obtain the concatenated feature;

[0030] After the concatenated feature passes through the convolutional layer, local information is obtained.

[0031] In one embodiment, the global information extraction block includes a patch embedding extraction block and a sliced feature attention block;

[0032] The patch embedding extraction block includes a single-slice patch embedding module and a full-slice patch embedding module. The single-slice patch embedding module is used to perform single-slice patch embedding on different sliced feature maps respectively to obtain the patch embedding of each sliced feature map. The full-slice patch embedding module is used to perform full-patch embedding on the feature maps of all slices to obtain the full-slice patch embedding;

[0033] The patch embedding extraction block is also used to generate a query vector and a key vector according to the patch embedding of the key sliced feature map and the full-slice patch embedding; generate a value vector according to the patch embeddings of all adjacent sliced feature maps;

[0034] The sliced feature attention block includes a multi-head self-attention (MSA) and a multi-layer perceptron (MLP); the query vector, the key vector and the value vector are input into the multi-head self-attention MSA to obtain the self-attention feature. After the self-attention feature and the value vector are added element-wise, a first addition result is obtained. The first addition result is input into the multi-layer perceptron MLP. The output of the multi-layer perceptron MLP and the first addition result are added element-wise to obtain a second addition result; an interpolation operation is performed on the second addition result to output the global information.

[0035] Compared with the prior art, the present application has the following beneficial effects: The abdominal multi-organ segmentation method and device based on multi-slice attention of the present application constructs a multi-slice attention module based on a hybrid of CNN and Transformer. This module can be flexibly inserted into different convolutional-based segmentation network backbones to form a new network framework. This new network framework can not only utilize the self-attention mechanism of Transformer to better model long-distance feature dependencies, but also still has the powerful ability of the convolutional operations in the original backbone to model local features. Experimental results show that the multi-slice attention module of the present application can improve the accuracy of abdominal multi-organ segmentation without the need for special network design. Brief Description of the Drawings

[0036] The present application can be better understood by referring to the description given below in conjunction with the accompanying drawings. The drawings, together with the following detailed description, are included in this specification and form a part of this specification. In the drawings:

[0037] Figure 1 Shows a schematic diagram of an abdominal multi-organ segmentation model;

[0038] Figure 2 Shows a schematic diagram of the multi-slice attention module;

[0039] Figure 3 Shows a schematic diagram of the global information extraction block;

[0040] Figure 4 Shows two schematic diagrams of patch embedding extraction. Detailed Description of the Embodiments

[0041] Hereinafter, exemplary embodiments of the present application will be described in conjunction with the accompanying drawings. For clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions may be made during the development of any such actual embodiment to achieve the specific goals of the developer, and these decisions may vary with different embodiments.

[0042] Here, it should also be noted that, in order to avoid obscuring the present application with unnecessary details, only the device structures closely related to the solution of the present application are shown in the drawings, and other details less related to the present application are omitted.

[0043] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the drawings. In this document, where feasible, embodiments can be combined with each other, features can be replaced or borrowed between different embodiments, and one or more features can be omitted in one embodiment.

[0044] The embodiments of this application provide a method for abdominal multi-organ segmentation based on multi-slice attention, which mainly includes the following steps:

[0045] Step S1: Obtain a model training dataset. The samples in the model training dataset are abdominal organ image slice groups, and an abdominal organ image slice group includes a key slice and multiple adjacent slices of the key slice.

[0046] The abdominal organ images are obtained through CT imaging. CT imaging scans different parts of the human body with X-rays to generate cross-sectional images. Since different organs and tissues in the human body have different absorption degrees of X-rays, there are certain ranges of differences in their CT values in CT images. In order to highlight the details of the target tissue, it is necessary to adjust the window width and window level of the image. This application aims to perform multi-organ segmentation on abdominal CT images. In order to retain rich information of abdominal organs, the CT image data involved is processed, and by setting the window width and window level, the CT value is limited between -100 and 240.

[0047] Since there is a large amount of background information in CT images, first, the irrelevant parts in the images are removed through image cropping, and only the area containing abdominal organs is retained to reduce the image size. In addition, since an NII format file is a whole scan of the abdomen, several slices corresponding to the start and end of the NII file may not contain organs, so they also need to be deleted, and only the slices containing organs are retained. After windowing the CT images, in order to further improve the image processing effect and the training stability of the model, it is also necessary to perform normalization processing on the images. The purpose of this step is to uniformly adjust the pixel values of the images to between [0, 1]. Through this normalization operation, the pixel values of the images can eliminate the differences under different scanning conditions, thereby improving the accuracy and robustness of the segmentation results. Grouped according to the number of slices, the abdominal organ image slice groups can include 3 slices, 7 slices, or 9 slices, and the middle slice is the key slice.

[0048] Step S2: Train an abdominal multi-organ segmentation model based on the model training dataset to obtain a trained abdominal multi-organ segmentation model. The abdominal multi-organ segmentation model includes multiple two-dimensional convolutional neural networks (2D CNNs), multiple multi-slice attention modules (MSAMs), and a UNet network. The samples are sequentially subjected to feature extraction through multiple two-dimensional convolutional neural networks, and the features output by each two-dimensional convolutional neural network Figure 1 are correspondingly input into multiple multi-slice attention modules for global information extraction and local information extraction, and the extracted global information and local information are fused to output fused features. All the fused features output by multiple multi-slice attention modules are input into the UNet network to obtain the organ segmentation map corresponding to the samples.

[0049] Figure 1shows a schematic diagram of the abdominal multi-organ segmentation model, Figure 1 Take 3 slice input as an example. The key slice is I k Indicates that the same number of upper and lower adjacent slices are represented by I u and I l Therefore, the original input data can be expressed as I = [I l ,I k ,I u ]∈R 3×l×W×H Where W and H represent the width and height of the CT slice, respectively, and l represents the input batch.

[0050] The input data is firstly subjected to feature extraction by four independent two-dimensional convolutional neural networks (2D CNNs), each of which extracts corresponding features from the input data. After this process, the feature map obtained is f = [f l ,f k ,f u ], where f l 、f k and f u Represent the feature maps extracted from low, medium and high CT slices, respectively.

[0051] Then, the features output by the four 2D convolutional neural networks are Figure 1 Each of them is input into four multi-slice attention modules, which can effectively process feature maps, extract global information between and within slices, and finally output fused features, so as to better capture the relative positions and spatial distributions of organs in the abdominal multi-organ segmentation task.

[0052] Then, the four fused features are input into the UNet network. The UNet network includes an encoder and a decoder. The encoder includes four encoding layers, namely EC1, EC2, EC3, and EC4. The decoder includes four decoding layers, namely DC1, DC2, DC3, and DC4. There are jump connections between EC1 and DC1, between EC2 and DC2, between EC3 and DC3, and between EC4 and DC4. The UNet segmentation network includes four main layers, and the number of channels in each layer is 64, 128, 256, and 512, respectively. The encoding layer gradually extracts the high-dimensional features of the image through convolution and pooling operations, compresses the spatial resolution layer by layer, and enhances the feature representation. The decoding layer gradually restores the image resolution through transposed convolution and refines the image segmentation results. The low-level features of the encoder are retained and transferred using jump connections to improve the segmentation accuracy.

[0053] Due to the size differences and class imbalance of organs in abdominal CT images, the segmentation is difficult and there are many false positives. Therefore, the Dice loss and cross-entropy (CE) loss are combined as the loss function of the network to optimize the network performance. The Dice loss improves the segmentation accuracy of small organs and edge regions, while the cross-entropy loss ensures accurate prediction of each class, thereby reducing false positives and improving the overall segmentation effect.

[0054] 100 training epochs are set. After each training epoch, the Dice metric of the validation set is calculated, and only the model weights when the Dice metric on the validation set reaches the maximum value are saved as the optimal model. In the test phase, the optimal model is loaded and used to predict the test set, and finally, metrics such as DSC, HD95, and JSC are used to comprehensively evaluate the performance of the model.

[0055] Step S3: Input the abdominal organ image slice group to be segmented into the trained abdominal multi-organ segmentation model to obtain the organ segmentation map corresponding to the abdominal organ image slice group to be segmented.

[0056] In this embodiment, a multi-slice attention module is constructed. This module can be flexibly inserted into different convolutional-based segmentation network backbones to form a new network framework. This new network framework can not only better model long-distance feature dependencies using the self-attention mechanism of Transformer but also still has the powerful ability of convolutional operations in the original backbone to model local features.

[0057] In one embodiment, Figure 2 shows a schematic diagram of the multi-slice attention module. Refer to Figure 2 , the multi-slice attention module includes a local information extraction block and a global information extraction block;

[0058] The feature map is input into the local information extraction block, and the local information f local ;

[0059] The feature map is input into the global information extraction block, and the global information f global ;

[0060] After the local information and the global information are fused and passed through a convolutional operation, the fused feature f fuse is output.

[0061] The local information extraction block in the multi-slice attention module (MSAM) is mainly responsible for extracting fine-grained local features from the input feature map, enhancing the information interaction and integration between slices; while the global information extraction block focuses on extracting long-distance dependency relationships between and within slices.

[0062] Specifically, referring to Figure 2 , the local information extraction block includes a Channel shuffle module and a Conv layer. The Channel shuffle module divides the feature map along the channel dimension into multiple groups, and performs depthwise separable convolution on each group to extract local features; the local features of each group are concatenated along the channel dimension to obtain the concatenated features;

[0063] After the concatenated features pass through the Conv layer, local information is obtained.

[0064] In this embodiment, a channel shuffle operation is performed on the input feature map to enhance the information integration ability between slices. Channel shuffle is achieved by dividing the input along the channel dimension into multiple groups and applying depthwise separable convolution within each group, so as to effectively rearrange and combine the information of different slices. The convolution operation within each group can extract different scales and local features, enabling the model to better focus on the detailed differences between different abdominal organs. After completing this operation, the local features output by each group are concatenated along the channel dimension to form a new and richer feature tensor. On this basis, further through a 3×1×1 Conv layer, local information f local is extracted. These local information are crucial for the subsequent processing of the model in depicting the details of abdominal organs and capturing edge information. This method not only optimizes the feature transmission between different slices, but also enhances the expression ability of local details, thus improving the overall segmentation accuracy and performance of abdominal organs.

[0065] Specifically, referring to Figure 2 , the global information extraction block includes a Patch Embedding Extraction block and a Slice Feature Attention block;

[0066] Figure 3 shows a schematic diagram of the global information extraction block. Referring to Figure 3 , the Patch Embedding Extraction block includes a Single-slice embedding module and an All-slice embedding module. The Single-slice embedding module is used to perform single-slice patch embedding on different slice feature maps f u , f k , f l respectively, to obtain the patch embeddings E u , E k , E l of each slice feature map. The All-slice embedding module is used to perform all-patch embedding on the feature maps of all slices to obtain the all-slice patch embedding E all;

[0067] The patch embedding extraction block is also used to generate a query vector f k and a full-slice patch embedding E all , and generate a query vector f q and a key vector f k ; According to the patch embeddings E of all adjacent slice feature maps u , E l , generate a value vector f v ; Here, E k and E all are added element-wise and input into the value head H v to obtain the value vector f v , and after fusing E u , E l they are respectively input into the query head H q and the key head H k , and the query vector f q and the key vector f k are respectively obtained.

[0068] The slice feature attention block includes a multi-head self-attention MSA and a multi-layer perceptron MLP; the query vector, the key vector, and the value vector are input into the multi-head self-attention MSA to obtain self-attention features, and after the self-attention features are added element-wise to the value vector f v , the first addition result is obtained, the first addition result is input into the multi-layer perceptron MLP, and the output of the multi-layer perceptron MLP is added element-wise to the first addition result to obtain the second addition result; an interpolation operation is performed on the second addition result to output the global information f global .

[0069] In this embodiment, key slice features are used to enhance the value vector: since using full-slice features as the value vector is likely to have the opposite effect, the key slices are separated and superimposed with the full-slice feature vectors. This method largely preserves the features of the key slices, making the network pay more attention to capturing the global clues between the key slice features and other slice features. Reconstruct the query and key vectors: when using full-slice features to generate the query and key vectors, the network is likely to capture the feature dependencies of the key slice features themselves, but this is contrary to the motivation of introducing multiple adjacent slices to assist multi-organ segmentation. Therefore, the key slice features are removed from the query and key vectors. Although this approach sacrifices the feature dependencies learned by the network in the key slices, it forces the network to learn more inter-slice dependency information, thereby better compensating for the weaknesses brought by the convolutional backbone network.

[0070] Specifically, Figure 4 shows two schematic diagrams of patch embedding extraction, see Figure 4, the single-slice patch embedding module is used to perform two-dimensional convolution (2D Conv), average pooling (Avg pooling), and shape reshaping (reshape) on the slice feature map to obtain the patch embedding of the slice feature map.

[0071] The whole-slice patch embedding module is used to perform three-dimensional convolution (3D Conv), dimension compression (Unsqueeze), and shape reshaping (reshape) on the feature maps of all slices to obtain the whole-slice patch embedding.

[0072] In this embodiment, in the single-slice patch embedding module, the convolution + pooling method is used to extract single-slice patches. The single-slice feature map f l Or (f k And f u ) First, it passes through a 2D convolution with 384 output channels (the output channels are uniformly set to 384 here) and a 1×1 convolution kernel to adjust the channels. Then, average pooling is used for feature compression, and finally, the patch embedding [E l , E k , E u ∈ R 3x256x384 is obtained for each slice feature map. In the whole-slice patch embedding module, three-dimensional convolution is used to fuse the features between different slices, and then the dimension of the convolved feature map is compressed and its shape is reshaped to obtain the whole-slice patch embedding E all ∈ R 256x384 . Therefore, the whole-slice patch embedding and the single-slice patch embedding have the same feature map dimension to adapt to the subsequent transformer blocks.

[0073] Adopting the same inventive concept as the abdominal multi-organ segmentation method based on multi-slice attention, this embodiment also provides a corresponding abdominal multi-organ segmentation device based on multi-slice attention, including:

[0074] A dataset acquisition module, used to acquire the model training dataset. The samples in the model training dataset are abdominal organ image slice groups, and the abdominal organ image slice groups include key slices and multiple adjacent slices of the key slices;

[0075] A model training module, used to train the abdominal multi-organ segmentation model based on the model training dataset to obtain the trained abdominal multi-organ segmentation model; the abdominal multi-organ segmentation model includes multiple two-dimensional convolutional neural networks, multiple multi-slice attention modules, and a UNet network. The samples are sequentially subjected to feature extraction by multiple two-dimensional convolutional neural networks, and the features output by each two-dimensional convolutional neural network Figure 1One corresponding input is fed into multiple multi-slice attention modules for global information extraction and local information extraction, and the extracted global information and local information are fused to output fused features; all the fused features output by the multiple multi-slice attention modules are input into the UNet network to obtain the organ segmentation map corresponding to the sample.

[0076] A prediction module is used to input a group of abdominal organ image slices to be segmented into the trained abdominal multi-organ segmentation model to obtain the organ segmentation map corresponding to the group of abdominal organ image slices to be segmented.

[0077] The abdominal multi-organ segmentation device based on multi-slice attention in this embodiment has the same inventive concept as the above-mentioned abdominal multi-organ segmentation method based on multi-slice attention. Therefore, the specific implementation manner of this device can be seen in the embodiment part of the above-mentioned abdominal multi-organ segmentation method based on multi-slice attention, and its technical effect corresponds to that of the above method, which will not be elaborated here.

[0078] In order to verify the effectiveness of this application, the following experiments were carried out:

[0079] 1. The dataset and experimental settings used in the experiment

[0080] The WORD dataset is a collection of CT scans of 150 patients before radiotherapy at the same medical center. Each three-dimensional CT image in this dataset contains 159 to 330 slices of 512×512 pixels, with an in-plane resolution of 0.976 mm×0.976 mm and a slice spacing of 2.5 mm to 3.0 mm. There are 16 organs to be segmented, including the liver, spleen, kidney (L), kidney (R), stomach, gallbladder, esophagus, pancreas, duodenum, colon, small intestine, adrenal gland, rectum, bladder, femoral head (L), and femoral head (R). The dataset is randomly divided into 100 cases, 20 cases, and 30 cases for training, validation, and testing respectively.

[0081] The model proposed in this application is built based on the Pytorch deep learning framework. In the following experiments, it is trained using the GPU Nvidia GeForce RTX 3090 24GB, and 4-fold cross-validation is used for the final evaluation. For the fairness of comparison, the same parameter settings are used to train the model, and the Adam optimizer is used to optimize the training of the model.

[0082] 2. Train and test the abdominal multi-organ segmentation model based on multi-slice attention

[0083] To evaluate the effectiveness of the proposed MSAM in multi-organ segmentation, comparative experiments were conducted on the WORD dataset with four baseline networks, namely Unet, AttentionUnet, TransUnet, and STUNet. Slicers represent the number of slices input to the network each time. As shown in Table 1, the proposed multi-slice attention module brought promising improvements in segmentation performance to all four baseline networks. When the input slices were 7, the segmentation network achieved the best segmentation results. Under the action of MSAM, the average Dice of Unet, AttentionUnet, TransUnet, and STUNet increased by 1.95%, 1.08%, 0.45%, and 1.03% respectively. Among them, the improvement of MSAM on TransUnet in multi-organ segmentation was the lowest because the Transformer mechanism was introduced in the backbone network of TransUnet to extract global context information in the slices. Therefore, the performance improvement brought by the global information extraction block design in MSAM was relatively small. Nevertheless, the design of enhancing the value vector and reconstructing the query and key vectors in the MSAM global information extraction block still endows TransUnet with a stronger global dependence capture ability to provide significant performance improvement. At the same time, STUNet+MSAM achieved the SOTA result when the input was 7 slicers, with the average Dice, HD95, and JSC increasing by 1.03%, decreasing by 1.06 mm, and increasing by 1.62% respectively. To maintain the fairness of the experiment, subsequent experiments were all conducted with 7 slices input.

[0084] Table 1: Comparative experiments on multi-organ segmentation of MSAM and baseline networks on the WORD dataset.

[0085]

[0086] 3. Design ablation experiments to evaluate the effectiveness of the key designs in MSAM

[0087] Design ablation experiments to evaluate the effectiveness of the key designs in MSAM. As can be seen from Table 2, after adding the global information extraction block and the original Transformer to the baseline network STUNet, the average Dice increased by 0.27% and 0.39% respectively. When both were combined and inserted into STUNet, the average Dice increased by 0.56%. The results show that the combination of global information and local information can indeed improve the segmentation accuracy in the case of multi-slice input. When enhancing the value vector through key slice features, the experimental results improved by 0.34% compared to adding the original Transformer. After further removing the key slice features from the query and key vectors, the best segmentation performance was obtained, and compared with the baseline network STUNet, the average Dice increased by 1.03%. The ablation experiment results show that the proposed MSAM can better capture the context information in multi-slice image data, and all the designs in MSAM can promote the accuracy of multi-organ segmentation.

[0088] Table 2: Ablation experiments on various designs in MSAM on the WORD dataset.

[0089]

[0090] As mentioned above, these are only various implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An abdominal multi-organ segmentation method based on multi-slice attention, characterized in that Including: Obtain a model training data set, where the samples in the model training data set are abdominal organ image slice groups, and the abdominal organ image slice group includes key slices and multiple adjacent slices of the key slices; Train an abdominal multi-organ segmentation model based on the model training data set to obtain a trained abdominal multi-organ segmentation model; the abdominal multi-organ segmentation model includes multiple two-dimensional convolutional neural networks, multiple multi-slice attention modules, and a UNet network. The samples are sequentially subjected to feature extraction by the multiple two-dimensional convolutional neural networks, and the feature maps output by each two-dimensional convolutional neural network are respectively input into the multiple multi-slice attention modules for global information extraction and local information extraction, and the extracted global information and local information are fused to output a fused feature; All the fused features output by the multiple multi-slice attention modules are input into the UNet network to obtain an organ segmentation map corresponding to the sample; Input the abdominal organ image slice group to be segmented into the trained abdominal multi-organ segmentation model to obtain an organ segmentation map corresponding to the abdominal organ image slice group to be segmented.

2. The method according to claim 1, characterized in that, The multi-slice attention module includes a local information extraction block and a global information extraction block; The feature map is input into the local information extraction block to output local information; The feature map is input into the global information extraction block to output global information; After the local information and the global information are fused and subjected to a convolution operation, a fused feature is output.

3. The method according to claim 2, wherein The local information extraction block includes a channel shuffle module and a convolutional layer. The channel shuffle module divides the feature map into multiple groups along the channel dimension and performs depthwise separable convolution on each group to extract local features; The local features of each group are concatenated along the channel dimension to obtain a concatenated feature; After the concatenated feature passes through the convolutional layer, local information is obtained.

4. The method according to claim 2, wherein The global information extraction block includes a patch embedding extraction block and a slice feature attention block; The patch embedding extraction block includes a single-slice patch embedding module and a full-slice patch embedding module. The single-slice patch embedding module is used to perform single-slice patch embedding on different slice feature maps respectively to obtain patch embeddings of each slice feature map, and the full-slice patch embedding module is used to perform full-patch embedding on the feature maps of all slices to obtain a full-slice patch embedding; The patch embedding extraction block is further used to generate a query vector and a key vector according to the patch embedding of the key slice feature map and the full-slice patch embedding; generate a value vector according to the patch embeddings of all adjacent slice feature maps; The slice feature attention block includes a multi-head self-attention (MSA) and a multi-layer perceptron (MLP); the query vector, the key vector, and the value vector are input into the multi-head self-attention MSA to obtain a self-attention feature. After the self-attention feature is added element-wise to the value vector, a first addition result is obtained. The first addition result is input into the multi-layer perceptron MLP, and the output of the multi-layer perceptron MLP is added element-wise to the first addition result to obtain a second addition result; an interpolation operation is performed on the second addition result to output global information.

5. The method according to claim 4, characterized in that The single-slice patch embedding module is used to perform two-dimensional convolution, average pooling, and shape reshaping on the slice feature map to obtain the patch embedding of the slice feature map.

6. The method according to claim 4, wherein The whole-slice patch embedding module is used to perform three-dimensional convolution, dimension compression, and shape reshaping on the feature maps of all slices to obtain the whole-slice patch embedding.

7. An abdominal multi-organ segmentation device based on multi-slice attention, characterized in that, It includes: A dataset acquisition module for acquiring a model training dataset, where the samples in the model training dataset are abdominal organ image slice groups, and the abdominal organ image slice groups include key slices and multiple adjacent slices of the key slices; A model training module for training an abdominal multi-organ segmentation model based on the model training dataset to obtain a trained abdominal multi-organ segmentation model; the abdominal multi-organ segmentation model includes multiple two-dimensional convolutional neural networks, multiple multi-slice attention modules, and a UNet network. The samples are sequentially subjected to feature extraction by the multiple two-dimensional convolutional neural networks, and the feature maps output by each two-dimensional convolutional neural network are respectively input into the multiple multi-slice attention modules for global information extraction and local information extraction, and the extracted global information and local information are fused to output a fused feature; all the fused features output by the multiple multi-slice attention modules are input into the UNet network to obtain the organ segmentation map corresponding to the sample; A prediction module for inputting the abdominal organ image slice group to be segmented into the trained abdominal multi-organ segmentation model to obtain the organ segmentation map corresponding to the abdominal organ image slice group to be segmented.

8. The device according to claim 7, characterized in that, The multi-slice attention module includes a local information extraction block and a global information extraction block; The feature map is input into the local information extraction block to output local information; The feature map is input into the global information extraction block to output global information; After the local information and the global information are fused and subjected to a convolution operation, a fused feature is output.

9. The device according to claim 8, characterized in that, The local information extraction block includes a channel shuffle module and a convolutional layer. The channel shuffle module divides the feature map into multiple groups along the channel dimension and performs depthwise separable convolution on each group to extract local features; The local features of each group are concatenated along the channel dimension to obtain the concatenated feature; After the concatenated feature passes through the convolutional layer, local information is obtained.

10. The device according to claim 8, wherein, The global information extraction block includes a patch embedding extraction block and a slice feature attention block; The patch embedding extraction block includes a single-slice patch embedding module and a whole-slice patch embedding module. The single-slice patch embedding module is used to perform single-slice patch embedding on different slice feature maps respectively to obtain the patch embedding of each slice feature map, and the whole-slice patch embedding module is used to perform whole-patch embedding on the feature maps of all slices to obtain the whole-slice patch embedding; The patch embedding extraction block is further used to generate a query vector and a key vector according to the patch embedding of the key slice feature map and the whole-slice patch embedding; generate a value vector according to the patch embeddings of all adjacent slice feature maps; The slice feature attention block includes a multi-head self-attention (MSA) and a multi-layer perceptron (MLP). The query vector, the key vector, and the value vector are input into the multi-head self-attention MSA to obtain self-attention features. After the self-attention features are element-wise added to the value vector, a first addition result is obtained. The first addition result is input into the multi-layer perceptron MLP, and the output of the multi-layer perceptron MLP is element-wise added to the first addition result to obtain a second addition result. An interpolation operation is performed on the second addition result to output global information.

Images