A medical image segmentation method based on Transformer-like network
By introducing Transformer-like module and U-shaped network structure in medical image segmentation, combining cross-current attention and dense multi-layer perceptron module, the problem of insufficient long-range dependence and receptive field in medical image segmentation is solved, and better global context information capture and feature extraction are achieved, improving segmentation effect.
Patent Information
- Application Number
- CN202310591183.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2025-08-08
- Estimated Expiration
- 2043-05-24
AI Technical Summary
The existing convolutional neural networks are difficult to establish effective long-range dependencies and insufficient receptive fields in medical image segmentation, resulting in the loss of global context information and affecting the segmentation effect.
Using Transformer-like module and U-shaped network structure, combining the cross-current attention module and dense multi-layer perceptron module, the long-term dependence relationship and multi-scale feature fusion between image blocks are enhanced, small-sized image blocks are processed through the image slice module, and jump connections are made between the encoder and the decoder to compensate for information loss.
It improves the accuracy of medical image segmentation and the capture ability of global context information, enhances the extraction of feature information and multi-scale feature representation, reduces the problem of gradient dissipation, and improves the segmentation performance.
Smart Images
Figure CN116580040B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to medical image segmentation technology in the field of image analysis, and in particular to a medical image segmentation method based on a Transformer-like network. Background Art
[0002] Medical image segmentation is gaining increasing clinical importance in the field of medical image analysis, becoming a hot topic in areas such as image-guided surgery and radiotherapy. Current clinical applications include lesion segmentation, cell segmentation, and tissue and organ segmentation. Its purpose is to segment components with similar characteristics or extract regions of interest within medical images, which plays a crucial role in enabling doctors to make clinical diagnoses.
[0003] With the development of convolutional neural networks (CNNs) in computer vision, CNNs have been widely and rapidly applied to medical image segmentation. As a typical CNN variant, the U-Net is a neural network based on an encoder-decoder architecture. It extracts features through continuous downsampling of the encoder and upsampling the output features of the decoder through skip connections. This architecture has been shown to have excellent segmentation performance. In recent years, many methods based on the U-Net architecture have been proposed to address the problem of medical image segmentation, and these methods have demonstrated excellent performance across multiple datasets. However, due to the inherent inductive bias of these methods, the convolution kernels focus only on local image patches, which easily loses global context and fails to model long-range dependencies. Although some researchers have proposed attention-based methods to attempt to model long-term dependencies, these methods are not specifically designed for medical image segmentation tasks, have certain limitations in modeling global context, and rely heavily on large-scale pre-trained models. Therefore, there is considerable potential for improvement in this area.
[0004] Recently, the Transformer has achieved tremendous success in the field of natural language processing (NLP). Because its multi-head self-attention mechanism can effectively establish global connections between labeled sequences and has the ability to model long-range dependencies, it can be used to improve pure CNN models. Therefore, researchers have introduced the Transformer into the field of computer vision to achieve excellent results in visual tasks. In the field of medical image segmentation, some work has achieved satisfactory results. Some researchers have proposed a compatible framework that can be used for multiple visual tasks and has certain potential as a visual backbone. However, this method still relies on pre-trained weights obtained by training on large image datasets. Other researchers have used the Transformer as a powerful encoder for medical image segmentation, combining it with the U-Net structure to recover local spatial information for detail enhancement. However, this method focuses on using the self-attention mechanism at a single scale to model context, while ignoring cross-scale dependencies.
[0005] In summary, by enhancing spatial information and modeling long-term dependencies, accurate medical image segmentation can be achieved. Reducing the loss of feature information, fusing multi-scale features, and achieving good performance on smaller datasets are key issues that need to be addressed in medical image segmentation. Summary of the Invention
[0006] To address the above issues, the present invention aims to provide a medical image segmentation method based on a Transformer-like network. This method uses Transformer-like modules to model long-term dependencies between image blocks and employs a U-shaped network structure to achieve multi-scale information fusion between modules. The technical solution is as follows:
[0007] A medical image segmentation method based on a Transformer-like network includes the following steps:
[0008] Step 1: Select a public medical image segmentation dataset and preprocess the training set in the dataset;
[0009] Step 2: Build an image slicing module to input small-sized image blocks into the image segmentation network;
[0010] Step 3: Construct a cross-stream attention module and a dense multi-layer perceptron module to capture multi-scale features;
[0011] Step 4: Build a Transformer-like module by using a cross-stream attention module and a dense multi-layer perceptron module to capture important global contextual information between image patches and extract rich local information and spatial texture information;
[0012] Step 5: Use the Transformer-like module as a component of the encoder and decoder, and combine it with the bridge to build a U-shaped segmentation network framework.
[0013] Furthermore, the medical image segmentation dataset in step 1 is: GLAnd Segmentation; during preprocessing, all images are adjusted to images with a size of 224×224 pixels.
[0014] Furthermore, the specific process of constructing the image slicing module in step 2 is as follows:
[0015] Step 2.1: For each channel image of the input image I, take a value for every other pixel, so that each channel obtains 4 images of half size, expressed as follows:
[0016] I={I1,I2,...,I C}
[0017]
[0018] Among them, F(I n ) represents the image slicing operation, p (i，j) Represents pixel points, C is the number of image channels, n represents the nth channel of the image, and n≤C; H and W are the height and width of the image respectively. Indicates cascade operation;
[0019] Step 2.2: The information of W and H dimensions is concentrated into the channel space. The input channel is expanded by 4 times, and the output feature map is The image block size is H, W, and the number of channels is C.
[0020] Furthermore, the design process of the cross-stream attention module and dense multi-layer perceptron module in step 3 is as follows:
[0021] Step 3.1: Slice the output feature map of the image module As the input feature of the cross-stream attention module, the cross-stream attention module consists of three branches: large core attention branch, spatial attention branch and channel attention branch;
[0022] 1) The large core attention branch includes three sub-operations DW-Conv, DW-D-Conv and 1×1Conv, which respectively obtain X DW-Conv 、X DW-D-Conv and X Conv Three feature maps, the expression formula of the output feature map of the large core attention branch is as follows;
[0023] X sl =X l-1 ⊙(Conv 1×1(DW-D-Conv(DW-Conv(X l-1 ))))
[0024] Among them, ⊙ is a pixel-by-pixel multiplication operation, DW-Conv represents a depth-wise separable convolution operation, DW-D-Conv represents a depth-wise separable dilated convolution operation, and 1×1Conv represents a convolution operation with a convolution kernel of 1.
[0025] 2) The spatial attention branch first transforms the feature map X l-1 and X DW-Conv Generate feature maps through channel mean and reshape operations and N=H×W;the feature map X′ l and X′ DW-Conv Perform matrix multiplication to obtain the feature map X′ sn , then the feature map X′ sn The feature map is obtained through softmax operation The final feature map X″ sn and X′ l Matrix multiplication obtains the feature map output by the spatial attention branch
[0026] 3) The channel attention branch takes the feature map X l-1 and X DW-Conv Generate feature maps through global pooling and reshape operations and Then the feature map X″ l and X″ DW-Conv Perform matrix multiplication to generate feature map X′ c , and the feature map X′ c Perform softmax operation to generate channel attention map X″ c , and finally the feature map X″ l and channel attention map X″ c Perform matrix multiplication and reshaping operations to generate the feature map output by the channel attention branch
[0027] The cross-stream attention module finally outputs the feature map The expression is:
[0028]
[0029] Step 3.2: The input of the dense multilayer perceptron module is the output feature map of the cross-stream attention module First, the input is reshaped to obtain Then Input into a densely connected multilayer perceptron, the process is expressed as follows:
[0030]
[0031]
[0032] Here, K represents that the dense multilayer perceptron module has K layers, and all subsequent layers of the dense multilayer perceptron module are related to the previous layers; represents the feature map of the k-1th layer, G is the growth rate of the dense multilayer perceptron module, X k represents the output of a densely connected multilayer perceptron, represents the feature map of the kth layer, and MLP represents multi-layer perceptron.
[0033] Furthermore, the Transformer-like module in step 4 consists of three parts: a normalization layer, a cross-stream attention module, and a dense multilayer perceptron module, which are expressed as follows:
[0034]
[0035]
[0036] Among them, X l-1 represents the feature map of the input network, represents the output of the cross-stream attention module, X l Represents the output of the Transformer-like module, Norm represents the normalization operation, SAA represents cross-stream attention, and DenseMLP represents dense multi-layer perceptron.
[0037] Furthermore, step 5 uses the Transformer-like module to construct a U-shaped segmentation network framework, which includes an encoder, a decoder and a skip connection; the encoder is used to extract fine image features, and the image semantic information is obtained from the shallow layer to the deep layer through the Transformer-like module and downsampling in the encoding stage; the decoder is used to achieve pixel-level segmentation, and the spatial resolution of the image is restored through the Transformer-like module and upsampling in the decoding stage; the encoder and the decoder are connected by a skip connection to compensate for the spatial information loss caused by downsampling.
[0038] The beneficial effects of the present invention are:
[0039] 1) The present invention designs a medical image segmentation method based on a Transformer-like network, which can better supplement global context information and encode high-level semantic information.
[0040] 2) This invention combines the cross-stream attention module (SAA) to learn a larger receptive field, and compensates for the shortcomings of the self-attention module in losing information in spatial and channel learning by combining spatial attention and channel attention. The SAA attention module extracts image features from three dimensions, and the three branches use jump connections to transmit information, achieve global and local connections, and better extract feature information.
[0041] 3) The present invention constructs a DenseMLP module through dense connections, which can better learn multi-scale information in the channel dimension compared to the traditional Transformer method. The use of dense connections can alleviate the problem of gradient dissipation during training to a certain extent, and has the ability to extract multi-scale features. With each MLP unit module, the feature dimension of the next layer will increase, so the information circulating in the network is also greater, which can better capture global context information and encode high-level semantic information. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is the cross-stream attention module of the present invention.
[0043] Figure 2 This is the densely connected sensor module of the present invention.
[0044] Figure 3 It is the Transformer-like network module of the present invention.
[0045] Figure 4 This is a flowchart of the medical image segmentation method based on a Transformer-like network of the present invention. DETAILED DESCRIPTION
[0046] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0047] This paper designs a medical image segmentation method based on a Transformer-like network. First, a publicly available medical image segmentation dataset, GLAnd Segmentation, is selected and grouped into training and test sets, followed by data preprocessing. Next, an image slicing module is constructed to feed small image blocks into the image segmentation network's encoder-decoder structure (its core Transformer-like modules include the SAA attention module and the DenseMLP module). Finally, a convolutional layer restores the resolution of the feature map to the target resolution, and a loss is calculated based on the ground truth segmentation value to predict the segmentation result.
[0048] Step 1: Select a public medical image segmentation dataset and preprocess the training set in the dataset.
[0049] The public dataset is: GLAnd Segmentation. The datasets are trained using the 5-fold cross-validation method, and the training sets in the datasets are preprocessed.
[0050] The GLAnd Segmentation dataset contains microscopic images of hematoxylin-eosin-stained sections and ground-truth segmentation images provided by pathologists. The dataset contains 165 images with inconsistent resolutions, with the smallest resolution being 433×574 and the largest being 775×522. In the experiments, the resolution of all images was adjusted to 224×224.
[0051] Step 2: Construct an image slicing module for small-sized image block input to the image segmentation network.
[0052] The specific process of inputting small-size image blocks into the image segmentation network is as follows:
[0053] 1) First, for each channel image of the input image I, a value is taken for every other pixel, similar to neighboring downsampling. As a result, four images of half the size are obtained for each channel. The four images are complementary, but no information is lost. The expression formula is as follows:
[0054] I={I1,I2,...,I C}(n≤C)
[0055]
[0056] Among them, p (i，j) Represents pixel points, C is the number of image channels, n represents the nth channel of the image, H and W are the height and width of the image respectively, Indicates a cascade operation.
[0057] 2) In this way, the information of W and H dimensions can be concentrated into the channel space, the input channel is expanded by 4 times, and the output feature map is The expression formula is as follows:
[0058]
[0059] Among them, F(I n ) represents the image slicing operation, I represents the input image, Represents the image block size;
[0060] A convolution operation is performed on the obtained new image block, and finally a downsampled feature image block with no information loss is obtained, and the feature image block is used as input, which well preserves the global semantic information in the original image.
[0061] Step 3: Design a cross-stream attention module (SAA) and a dense multi-layer perceptron module (DenseMLP) to capture multi-scale feature blocks. The design process of SAA and DenseMLP is as follows:
[0062] 1) Output features of the image slicing module The image is used as the input of the attention module. The attention module (SAA) consists of three branches: large core attention, spatial attention, and channel attention. The cross-stream attention module of the present invention is as follows: Figure 1 As shown, the expression of this module is:
[0063]
[0064] in, represents the output of the cross-stream attention module, X l-1 is the input feature, X sl Output feature map for large kernel attention, X sn Spatial attention output feature map, X c Channel attention output feature map.
[0065] ① The large core attention branch is divided into three sub-operations: DW-Conv DW-D-Conv and 1×1Conv, respectively obtaining X DW-Conv 、X DW-D-Conv and X Conv Three feature maps. Replacing the large-size convolution with three small-size convolutions makes the equivalent convolution kernel larger, thereby increasing the receptive field and obtaining more global features while avoiding excessive computation. The large-core attention branch is expressed as follows;
[0066] X sl =X l-1 ⊙(Conv 1×1 (DW-D-Conv(DW-Conv(C l-1 ))))
[0067] Among them, X l-1 is the input feature, ⊙ is the pixel-by-pixel multiplication operation; DW-Conv represents the depth-wise separable convolution operation, DW-D-Conv represents the depth-wise separable dilated convolution operation, and 1×1Conv represents the convolution operation with a convolution kernel of 1.
[0068] ②The spatial attention branch first transforms the feature map X l-1 and X DW-Conv Generated by channel averaging and reshaping operations and X′ l and X′ DW-Conv Perform matrix multiplication to obtain X′ sn , then X′sn The feature map is obtained through softmax operation Last X″ sn and X′ l Matrix multiplication to obtain the output of the spatial attention branch This branch encodes broader semantic information into the local receptive field to enhance the feature map representation capability and can mine the similarity relationship between each pixel.
[0069] ③The channel attention branch takes the feature map X l-1 and X DW-Conv Generated by global pooling and reshaping operations and Then add X″ l and X″ DW-Conv Perform matrix multiplication to generate X′ c , and for X′ c Perform softmax operation to generate channel attention map X″ c , and finally X″ l and X″ c Perform matrix multiplication and reshaping operations to generate channel feature maps In this branch, the channel attention features can be effectively calculated and the spatial dimensions of the feature map can be compressed, and each image feature map can be mined.
[0070] 2) The input of the dense multilayer perceptron (DenseMLP) module is the output feature map of the SAA module First, the input is reshaped to obtain Then Input into a densely connected multilayer perceptron (MLP), refer to Figure 2 For the densely connected perceptron module of the present invention, the process is expressed as follows:
[0071]
[0072]
[0073] Here, K represents that the DenseMLP module has K layers, and all subsequent layers of the DenseMLP module are related to the previous layers. Represents the feature map of the k-1th layer, the growth rate G of the DenseMLP module is set to 32, Indicates a cascade operation.
[0074] Using dense connections can mitigate the problem of gradient vanishing during training to a certain extent and enable multi-scale feature extraction. With each MLP module, the feature dimension of the next layer increases G-fold, increasing the amount of information flowing through the network and, correspondingly, the network's capabilities. By learning feature representations through this module, we can better capture global context and encode high-level semantic information.
[0075] Step 4: Use the module obtained in step 3 to build a Transformer-like module to capture important global context information between image patches, extract rich local information and spatial texture information, and maintain the ability to model long-distance dependencies. The Transformer-like module consists of three parts: normalization layer, cross-stream attention module (SAA) and dense multi-layer perceptron module (DenseMLP), refer to Figure 3 It is the Transformer-like network module of the present invention, which is represented as follows:
[0076]
[0077]
[0078] Among them, X l-1 represents the feature map of the input network, represents the output of the cross-stream attention module, X l Represents the output of the Transformer-like module, and Norm represents the normalization operation.
[0079] Step 5: Use the Transformer-like module obtained in step 4 as the important component of the encoder and decoder, and combine it with the bridge to build a U-shaped segmentation network framework. Figure 4 This is a flowchart of the medical image segmentation method based on a Transformer-like network of the present invention.
[0080] The basic unit of this network is a Transformer-like module, where the encoder extracts fine-grained image features and the decoder performs pixel-level segmentation. Skip connections are used between the encoder and decoder to compensate for spatial information loss caused by downsampling. During the encoding phase, image semantic information is acquired from shallow to deep layers through Transformer-like modules and downsampling. During the decoding phase, spatial resolution is restored through Transformer-like modules and upsampling. Furthermore, skip connections are used between each block in the encoder and each block in the decoder to avoid feature information loss caused by downsampling and convolution.
Claims
1. A medical image segmentation method based on a Transformer-like network, characterized in that: The steps include: Step 1: Select a public medical image segmentation dataset and preprocess the training set in the dataset; Step 2: Build an image slicing module to input small-sized image blocks into the image segmentation network; Step 3: Construct a cross-stream attention module and a dense multi-layer perceptron module to capture multi-scale features; Step 4: Build a Transformer-like module by using a cross-stream attention module and a dense multi-layer perceptron module to capture important global contextual information between image patches and extract rich local information and spatial texture information; Step 5: Use the Transformer-like module as a component of the encoder and decoder, and combine it with the bridge to build a U-shaped segmentation network framework; The design process of the cross-stream attention module and dense multi-layer perceptron module in step 3 is as follows: Step 3.1: Slice the output feature map of the image module As the input feature of the cross-stream attention module, the cross-stream attention module consists of three branches: large core attention branch, spatial attention branch and channel attention branch; The image block size is H, W, and the number of channels is C; 1) The large core attention branch includes three sub-operations DW-Conv, DW-D-Conv and 1×1Conv, which respectively obtain X DW-Conv 、X DW-D-Conv and X Conv Three feature maps, the expression formula of the output feature map of the large core attention branch is as follows; X sl =X l-1 ☉(Conv 1×1 (DW-D-Conv(DW-Conv(X l-1 )))) Among them, ⊙ is a pixel-by-pixel multiplication operation, DW-Conv represents a depth-wise separable convolution operation, DW-D-Conv represents a depth-wise separable dilated convolution operation, and 1×1Conv represents a convolution operation with a convolution kernel of 1. 2) The spatial attention branch first transforms the feature map X l-1 and X DW-Conv Generate feature maps through channel mean and reshape operations and N=H×W;the feature map X′ l and X′ DW-Conv Perform matrix multiplication to obtain the feature map X′ sn , then the feature map X′ sn The feature map is obtained through softmax operation The final feature map X″ sn and X′ l Matrix multiplication obtains the feature map output by the spatial attention branch 3) The channel attention branch takes the feature map X l-1 and X DW-Conv Generate feature maps through global pooling and reshape operations and Then the feature map X″ l and X″ DW-Conv Perform matrix multiplication to generate feature map X′ c , and the feature map X′ c Perform softmax operation to generate channel attention map X″ c , and finally the feature map X″ l and channel attention map X″ c Perform matrix multiplication and reshaping operations to generate the feature map output by the channel attention branch The cross-stream attention module finally outputs the feature map The expression is: Step 3.2: The input of the dense multilayer perceptron module is the output feature map of the cross-stream attention module First, the input is reshaped to obtain Then The input is fed into a densely connected multilayer perceptron. The program expression is as follows: Here, K represents that the dense multilayer perceptron module has K layers, and all subsequent layers of the dense multilayer perceptron module are related to the previous layers; represents the feature map of the k-1th layer, G is the growth rate of the dense multilayer perceptron module, X k represents the output of a densely connected multilayer perceptron, represents the feature map of the kth layer, and MLP represents multi-layer perceptron; The Transformer-like module in step 4 consists of three parts: a normalization layer, a cross-stream attention module, and a dense multilayer perceptron module, which are represented as follows: Among them, X l-1 represents the feature map of the input network, Represents the output of the cross-stream attention module, Xl represents the output of the Transformer-like module, Norm represents the normalization operation, SAA represents cross-stream attention, and DenseMLP represents dense multi-layer perceptron.
2. The medical image segmentation method based on a Transformer-like network according to claim 1, characterized in that: The medical image segmentation dataset in step 1 is: GLAnd Segmentation; during preprocessing, all images are adjusted to images with a size of 224×224 pixels.
3. The medical image segmentation method based on a Transformer-like network according to claim 1, characterized in that: The specific process of constructing the image slicing module in step 2 is as follows: Step 2.1: For each channel image of the input image I, take a value for every other pixel, so that each channel obtains 4 images of half size, expressed as follows: I={I1,I2,...,I C } Among them, F(I n ) represents the image slicing operation, p (i，j) Represents pixel points, C is the number of image channels, n represents the th channel of the image, and n≤C; H and W are the height and width of the image respectively. Indicates a cascade operation; Step 2.2: The information of W and H dimensions is concentrated into the channel space. The input channel is expanded by 4 times, and the output feature map is The image block size is H, W, and the number of channels is C.
4. The medical image segmentation method based on a Transformer-like network according to claim 1, characterized in that: The step 5 uses the Transformer-like module to construct a U-shaped segmentation network framework, which includes an encoder, a decoder and a skip connection; the encoder is used to extract fine image features, and image semantic information is obtained from shallow to deep layers through the Transformer-like module and downsampling in the encoding stage; the decoder is used to achieve pixel-level segmentation, and the spatial resolution of the image is restored through the Transformer-like module and upsampling in the decoding stage; the encoder and decoder are connected through a skip connection to compensate for the spatial information loss caused by downsampling.