Medical image segmentation method fusing Transform and residual network

Abstract

The invention provides a medical image segmentation method fusing Transform and a residual network, and belongs to the field of medical image segmentation. Firstly, data enhancement is performed to enlarge a training set; the enhanced data set serves as input of model training and is respectively input into a Transform branch and a residual network branch which are parallel to extract global semantic features and local detail features of the image; a deep residual attention module is adopted at the jump joint of each branch to help to predict a target boundary; a step-by-step up-sampling feature fusion mode is adopted in the decoding stage, so that semantic difference and feature information loss are reduced; and finally, carrying out channel splicing on the full-size tensors output by the two branches, and fusing the spliced tensors by using an efficient prediction head to obtain a segmented image. Through verification, the performance of the method is better than that of many existing mainstream methods, and the problems of difficulty in segmentation of complex target areas of human organs, tissues or lesions, loss of edge detail information, insufficient generalization ability and the like in medical image segmentation of the existing method can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a medical image segmentation method integrating a Transformer and a residual network. Background Art

[0002] In medical images, the various organs, tissues and lesion areas of the human body have complex shapes, different sizes, diverse textures, and very vague boundaries between areas, which greatly affects the doctor's correct diagnosis of the patient's condition. Traditional medical image segmentation methods are mainly based on the shape, texture and color of specific areas, and use algorithms based on watershed, grayscale threshold, clustering, etc. to achieve image segmentation. This type of method that uses fixed information to artificially design and extract features can only extract some typical areas, but it is difficult to accurately segment complex and diverse targets.

[0003] In recent years, deep learning-based methods have developed rapidly in the field of medical images. Such methods can automatically learn the features of medical images, quickly lock key areas, and segment target images. They can help doctors find lesion areas in clinical practice, which is beneficial to the prevention and treatment of patients' diseases. Early deep learning-based medical image segmentation methods used convolutional neural networks as the backbone. They used the spatial inductive bias of convolution operations to fully capture the local detail information of the image. However, due to the limited receptive field, convolutional neural networks lack efficiency in capturing global feature information. In addition, as the number of convolutional layers in the network increases, low-level features of the image will be lost. Therefore, convolutional neural network-based methods still face challenges in medical image segmentation. Recently, many Transformer-based network models have been applied to visual tasks and have achieved good results. Transformer has an efficient global receptive field and can effectively capture global dependencies in images. However, it lacks spatial inductive bias in local information modeling, resulting in poor performance in extracting local detail information of images. This deficiency is particularly evident in medical image segmentation. In addition, tissues, lesions and other regions themselves have variability in size, shape and texture, which leads to low segmentation accuracy and poor generalization ability in existing medical segmentation methods. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, improve the accuracy of medical image segmentation and the generalization performance of the model, and help doctors accurately diagnose patients' conditions, the present invention provides a medical image segmentation method that integrates Transformer and residual network.

[0005] In an embodiment of the present invention, the medical image segmentation method integrating Transformer and residual network includes the following technical solutions, and the specific steps are as follows:

[0006] 1. Perform data enhancement on the training set of the model input. The specific methods adopted are: Gaussian blur, color transformation, horizontal and vertical flipping, and affine transformation.

[0007] 2. In order to make the model have the advantages of both Transformer and convolutional neural networks, the present invention uses parallel Transformer network branches and residual network branches to gradually extract the global semantic features and local detail features of the image respectively;

[0008] 3. In order to reduce the information loss in the decoding process and the semantic difference between features at different stages, the feature information of each stage is fused by gradually upsampling features in the decoding stage;

[0009] 4. Create a deep residual attention module as a skip connection structure between the encoder and decoder. This module is composed of a deep residual module and a residual attention module in parallel. The deep residual module can perform deeper feature extraction and improve exploration efficiency; the residual attention module models global and local features, highlights key areas, and improves segmentation accuracy;

[0010] 5. An efficient prediction head is used at the end of the decoder to fuse the coarse-grained and fine-grained features output by the two parallel branches to generate a segmentation map.

[0011] Compared with the prior art, the beneficial effects of the present invention are: it can take into account both the high-level semantic features and local detail features of the image, has advanced learning and generalization capabilities, and its segmentation performance is better than many existing mainstream methods. It can effectively improve the problems of existing models in segmentation, such as difficulty in segmenting complex areas of organs, tissues or lesions, loss of edge detail information, and insufficient generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is the overall flow chart of training and testing of medical image segmentation method integrating Transformer and residual network;

[0013] Figure 2 It is the process of each branch structure processing information, (a): the specific process of the residual network branch processing medical images; (b): the specific process of the Transformer network branch processing medical images;

[0014] Figure 3 The following are the specific details of each module: (a): deep residual attention module; (b): prediction head; (c): residual module; (d): residual attention mechanism in the Transformer branch; (e): residual attention mechanism in the residual network branch. DETAILED DESCRIPTION

[0015] The specific embodiments of the present invention are now described, and the examples of the exemplary embodiments are shown in the accompanying drawings, which are helpful for further understanding by those skilled in the art. It is worth pointing out that the following embodiments are only used to illustrate the technical solution of the present invention, but not to limit it. The protection scope of the present invention is not limited thereto, and any ordinary technician familiar with the technical field can easily think of changes or substitutions without departing from the scope of the present invention, which belong to the protection scope of the present invention.

[0016] See also Figures 1 to 3 The present invention provides a medical image segmentation method integrating Transformer and residual network, comprising the following steps:

[0017] Step 1: Data enhancement, which is only for the training set and is used to expand the data volume. It mainly includes the following operations: Gaussian blur; color transformation; horizontal and vertical flip; affine transformation. Among them, Gaussian blur and color transformation operations are only applied to images, and the other enhancement operations are all applied to images and their corresponding labels.

[0018] Step 2: Use the image encoding module to extract features. The initial input image size is 352×352×3. Figure 1 As shown in the figure, the encoder mainly consists of two parallel branch encoding blocks: the residual network encoder and the Transformer network encoder, which respectively extract the global semantic features and local detail features of the image.

[0019] The specific details of each branch encoder are as follows Figure 2 As shown in the figure, the residual network branch uses the more mature network model Resnet34 as the encoder of the branch, and gradually downsamples through five stages to increase the network's receptive field from local to global. The image sizes output at each stage are 176×176×64, 88×88×64, 44×44×128, 22×22×256, and 11×11×512, respectively.

[0020] The Transformer network branch uses Pyramid Vision Transformer v2 (PVTv2) as the encoder and uses the spatial reduction attention mechanism to model the global dependencies of the image. This branch has four image processing stages to gradually extract the high-level semantic features of the image. The image sizes output by each stage are 88×88×64, 44×44×128, 22×22×256, and 11×11×512 respectively.

[0021] Step 3: If Figure 2 As shown in FIG. 1 , the present invention creates a deep residual attention module between the encoder and decoder of each branch to act as a skip connection structure to process the output of each encoding stage. The details of this module are as follows Figure 3As shown in (a), it consists of a deep residual module and a residual attention module in parallel, and the number of output channels of the deep residual attention mechanism in each stage is 64.

[0022] The deep residual module consists of two serial residual blocks and one parallel residual block. It has multiple receptive fields, which can improve the exploration efficiency of the target area and accurately predict the boundary in a complex environment. The structure of the residual module is as follows: Figure 3 As shown in (c), it consists of a stacking layer and an identity map, and the stacking layer consists of group normalization, activation function and convolutional layer.

[0023] The residual attention module is as follows Figure 3 (d) Figure 3 (e) Figure 3 The module in (d) is used in the jump connection of the Transformer branch. This module comes from the SE-ResNet architecture generated by applying the squeeze and excitation (SE) channel attention mechanism in the residual network. The information processing process can be expressed as:

[0024] (1)

[0025] (2)

[0026] (3)

[0027] In the formula, F i Represents a feature map; Residual represents the stacking layer in the residual module; S represents the Simoid activation function; ReLU activation function. It indicates a convolution with a kernel size of 3×3; represents global average pooling; Represents tensor multiplication; Represents tensor addition. The input feature map first passes through the stacking layer of the residual module for feature extraction. Then, global average pooling is used in formula (2) to compress the size of the output feature map from H×W×C to 1×1×C, focusing attention on each channel. Then, the excitation mechanism consisting of double-layer convolution and activation function learns the channel features of the compressed feature map and models the interdependence between channels. Finally, the final output is obtained through formula (3). The multiplication operation assigns the learned channel attention weight to each channel, allowing the network to pay more attention to important channel features and suppress less useful features. The feature map of the initial input is introduced by the addition operation, which can play the role of the residual structure and reduce the errors caused by the stacking layer and attention mechanism.

[0028] by Figure 3In (d), the present invention combines the spatial attention mechanism with the residual module and proposes Figure 3 The residual attention module in (e) is used in the skip connection of the residual network branch. The information processing process can be expressed as:

[0029] (4)

[0030] (5)

[0031] (6)

[0032] In the formula, F i Represents a feature map; Residual represents the stacking layer in the residual module; S represents the Simoid activation function; Represents a convolution with a kernel size of 7×7; Concatenation represents channel concatenation; represents the maximum pooling along the channel axis; represents average pooling along the channel axis; Represents tensor multiplication; Represents tensor addition. The input feature map is first extracted by the stacking layer of the residual module. Then, the maximum pooling and average pooling are used in formula (5) to perform corresponding pooling operations on the input feature map, focusing attention on the spatial area of ​​the feature map. The two sets of feature maps are concatenated and input into the convolution layer, and then the eigenvalues ​​are compressed to between 0 and 1 using the Simoid activation function to obtain the spatial attention weight tensor. Finally, the final output is obtained by formula (6). The multiplication operation assigns the learned spatial attention weight to each pixel, and the addition operation plays the role of the residual structure. The spatial attention mechanism allows the network model to learn the importance of each spatial region in the medical image to enhance local details and suppress irrelevant areas, while filtering out possible noise in low-level features.

[0033] Step 4: Decode using the progressively upsampled feature fusion decoder, as Figure 2 As shown in the figure, the decoding process adopts a bottom-up approach. First, the deepest feature map is upsampled twice, and then the channel is spliced ​​with the adjacent upper feature map. The feature fusion is performed through the residual block, and the feature map obtained after fusion repeats the above fusion process. The number of output channels of each residual block is 64. The above process is shown in formula (7):

[0034] (7)

[0035] In the formula, y i represents the fused i-th layer feature map, x irepresents the feature map of the i-th layer before fusion. In the Transformer branch, i∈[1,4], y4=x4, and in the residual network branch, i∈[1,5], y5=x5; RB represents the residual module; Up represents two-fold upsampling; Concatenation represents channel concatenation. Because the number of stages of the two branches is different, the multiples of upsampling required to restore the original size after feature fusion are also different. The Transformer branch needs to perform four-fold upsampling, and the residual network branch needs to perform two-fold upsampling.

[0036] Step 5: Tensor concatenation: concatenate the tensors output by the residual network branch and the Transformer network branch along the channel dimension;

[0037] Step 6: Use the efficient prediction head to process the concatenated tensor and obtain the segmentation map. The prediction head consists of a residual block and a convolution block, and its structure is as follows: Figure 3 As shown in (b), it consists of two residual blocks and a convolution module. The output of the Transformer branch and the output of the residual network branch are concatenated as the input of the prediction head. The prediction head fuses the global semantic features extracted by the Transformer branch with the local fine-grained features extracted by the residual network branch to finally obtain the image prediction segmentation map. The number of channels output by the two residual blocks of the prediction head are 32 and 16 respectively, and finally a 3×3 convolution is used to output the image segmentation result.

[0038] In summary, in view of the problems of complex and diverse target areas in medical images, different sizes, and blurred boundaries between regions, the present invention relates to a medical image segmentation method that integrates Transformer and residual network, adopts parallel Transformer branches and residual network branches, takes into account the high-level semantic features and local detail features of the image, introduces deep residual attention modules and step-by-step upsampling feature fusion in each branch, the former improves the model's attention to target areas, boundaries, and more detailed information, and suppresses unimportant areas; the latter promotes the effective interaction of feature information at each stage, and avoids information loss caused by information differences and excessive upsampling multiples. Finally, the outputs of each branch are fused using an efficient prediction head to generate a segmentation map.

[0039] The segmentation performance of the method involved in the present invention is better than many existing excellent models, and it has advanced learning and generalization capabilities. It can effectively improve the problems of existing models in medical image segmentation, such as difficulty in segmenting complex areas, difficulty in extracting edge detail information, and insufficient generalization ability.

[0040] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A medical image segmentation method integrating Transformer and residual network, characterized in that: The following steps are involved: Step 1: Data enhancement, only for the training set, to expand the amount of data; Step 2: Use the encoder to implement feature extraction; the encoder includes two parallel branch encoding blocks: residual network encoder and Transformer network encoder; Step 3: Use the deep residual attention module to process the output of each stage of the encoder. This module is located between the encoder and decoder of each branch and acts as a skip connection structure. Step 4: Feature decoding, each branch decodes the feature map in parallel using a step-by-step upsampling feature fusion decoding method; Step 5: Tensor concatenation: concatenate the tensors output by the residual network branch and the Transformer network branch along the channel dimension; Step 6: Use an efficient prediction head consisting of residual blocks and convolutional blocks to process the spliced ​​tensor to promote feature fusion and obtain the image segmentation map.

2. The medical image segmentation method integrating Transformer and residual network according to claim 1, characterized in that: The specific methods used to enhance the training set data are: Gaussian blur, color transformation, horizontal and vertical flipping, and affine transformation. Gaussian blur and color transformation operations are only applied to images, and the remaining operations are consistently applied to images and their corresponding labels.

3. The medical image segmentation method integrating Transformer and residual network according to claim 1, characterized in that: The specific way the encoder extracts features from an image: using a parallel residual network encoder and a Transformer network encoder to simultaneously extract features from the target area; the residual network branch uses the more mature model Resnet34 as an encoder to extract detail features; The Transformer network branch uses Pyramid Vision Transformer v2 (PVTv2) as the encoder to extract global features.

4. The medical image segmentation method integrating Transformer and residual network according to claim 1, characterized in that: The specific way in which the deep residual attention module processes the output of each stage of the encoder: This module is composed of a residual attention module and a deep residual module in parallel, and the two modules process the image feature information output by the encoder in parallel.

5. The medical image segmentation method integrating Transformer and residual network according to claim 1, characterized in that: The specific way in which the decoder fuses the feature maps of each stage is as follows: a bottom-up approach is adopted, where the deepest feature map is first upsampled twice, and then channel-joined with the adjacent upper-layer feature map, and feature fusion is performed through the residual block. The fused feature map continues to repeat the above fusion process.

6. The medical image segmentation method integrating Transformer and residual network according to claim 1, characterized in that: The prediction head processes the concatenated tensor to generate the image segmentation map in the following specific way: the concatenated tensor first passes through two residual blocks to fully fuse the features, and then passes through a convolution module to finally generate the image segmentation map predicted by the model.

Images