A medical image segmentation method fusing multi-scale residual attention

Abstract

The application discloses a medical image segmentation method fusing multi-scale residual attention, and steps are as follows: after pre-processing of collected data sets, training sets, verification sets and test sets are divided; data enhancement is performed; in the training process, data is randomly vertically flipped, randomly horizontally flipped and randomly rotated with a probability of 0.5; a multi-scale residual attention network is constructed; in the encoder part, the network reduces the channel number of the intermediate part of double convolution, and moderately reduces network parameters; in the decoder, a multi-scale residual attention module is designed to capture the detailed features of medical images under different scales, improve the utilization rate of image feature maps, relieve the gradient vanishing problem and degradation problem caused by the network being too deep, and further improve the stability of the network; and the model is tested in the test set to obtain the segmentation result of the model. The application solves the problem that the target region of a medical image is fuzzy with a background boundary, can effectively reduce the parameter amount of the network while improving the image segmentation precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image segmentation and artificial intelligence, in particular to a medical image segmentation method fusing multi-scale residual attention. BACKGROUND

[0002] The purpose of medical image segmentation is to segment the part of interest from the background in the medical image and extract relevant semantic features. Compared with ordinary images, medical images lack simple linear features, have higher complexity, greater differences between different modalities, and are affected by factors such as artifacts, volume effect, non-uniformity of gray scale, proximity of gray scale between different soft tissues, etc., which greatly affect the accurate segmentation of medical images. The existing medical image segmentation method relies on a large amount of labeled data set. Due to the experience and effort of doctors, the efficiency and accuracy of manually segmenting medical images are not satisfactory. Therefore, there is an urgent need for the emergence of an accurate medical image automatic segmentation method.

[0003] With the development of deep learning technology, convolutional neural networks (CNN) are widely used in medical image segmentation field due to its strong feature extraction ability, high automation level and good segmentation effect. In 2015, Jonathan et al. proposed a fully convolutional network (FCN) to solve the problem of segmenting each pixel in the image, realizing semantic-level image segmentation. However, due to the particularity of medical images, the segmentation result obtained by FCN is still not fine enough, and the relationship between each pixel is not fully considered, lacking spatial consistency. Ronneberger et al. proposed a U-shaped network structure U-Net based on FCN. Due to the U-shaped structure of the model, it can combine context information and speed up the training, and it performs well on small sample data sets, meeting the needs of medical image segmentation, and is widely used in medical image segmentation. Although U-Net is an excellent network structure, it is difficult to meet the performance requirements of today's medical image segmentation tasks. Taking the ISIC2018 data set as an example, the current U-Net mDice index is 85.84%, and the mIoU index is 78.47%.

[0004] On the basis of U-Net, researchers have proposed many related improved models. For example, Alom et al. inspired by the residual network and the recurrent network, proposed a deep residual network R2U-Net for medical image segmentation, which effectively increases the network depth and improves the network performance by replacing the encoder and decoder convolutional layers with recurrent residual blocks. Oktay et al. inspired by the attention mechanism, proposed a medical image segmentation network Attention U-Net mainly applied to the liver, which suppresses irrelevant information in the image and highlights local important features by replacing hard attention with soft attention and integrating attention into the skip connection and up-sampling module of U-Net. Beeche et al. inspired by Residual Block and Squeeze and Excitation, proposed a new U-Net variant Super U-Net to improve the segmentation performance of medical images. However, these improved models often bring problems such as dramatic increase in parameter quantity, excessive memory consumption, and difficulty in guaranteeing the generalization ability of the network. SUMMARY

[0005] The purpose of the present application is to overcome the problems and deficiencies of the prior art, and to provide a medical image segmentation method fusing multi-scale residual attention.

[0006] The present application solves the problems of excessive parameter quantity and poor generalization of existing medical image segmentation methods to some extent by expanding the receptive field, improving the perception ability of the target area, and compressing the network channel number. Such design ensures that the network segmentation model pays more attention to the area of interest and can effectively reduce the interference caused by noise. This makes it easier to segment and interpret complex medical images, effectively improves the precision and accuracy of medical image segmentation, and greatly reduces the network parameter quantity. The network parameter quantity of the method of the present application is only one third of that of U-Net. The mDice index of the method of the present application on the ISIC2018 dataset is 91.88%, and the mIoU index is 86.61%, which is 6.34% and 8.14% higher than that of the traditional U-Net, respectively.

[0007] To achieve the above purpose, the technical scheme is adopted as follows.

[0008] A medical image segmentation method fusing multi-scale residual attention, comprising the following steps:

[0009] S1: Collecting the dataset, preprocessing the collected dataset, and dividing the preprocessed dataset into a training set, a validation set and a test set according to the ratio of 8:1:1;

[0010] S2: Data augmentation is performed on the training set divided in S1 to expand the training set;

[0011] S3: input the training set obtained in S2 into a multi-scale residual attention network for iterative training; and after verification by the verification set obtained in S1, an overall objective function model L with optimal training parameters is obtained;

[0012] S4: input the test set image obtained in S1 into the overall objective function model L with optimal training parameters trained in S3, to generate a segmentation result and save the image.

[0013] Preferably, in the step S1, the data set preprocessing comprises: setting the input image size of all data sets to 352*352.

[0014] Preferably, in the step S2, the specific method of data enhancement is: using a probability of 0.5 to perform random vertical flipping, random horizontal flipping and random 90° rotation on the training set image for data enhancement, and using a multi-scale strategy of {0.75, 1, 1.25} to perform training by scaling the image.

[0015] Preferably, in the step S3, the specific content and design method of the multi-scale residual attention network are: the multi-scale residual attention network is composed of an encoder and a decoder, and has a total of 5 layers. In the encoder part, a compressed double convolution is used to extract the features of the image. In the decoder, a multi-scale residual attention module is designed by combining the multi-scale idea, the residual idea and the attention mechanism, so as to focus on the key areas from the channel level, suppress irrelevant features, effectively improve the feature acquisition ability of the network under the condition of controlling the parameter amount, and further improve the segmentation precision. Through the decoder, the optimal segmentation result image is finally generated.

[0016] In the encoder part, the compressed double convolution is an improvement on the double convolution part of the original U-Net network. The method is: first, reducing the dimension of the first 3*3 convolution to 1 / 4 of the original channel number, and then increasing the dimension of the second 3*3 convolution to twice the original channel number. Through the compression of the channel number in the middle of the double convolution, the purpose of reducing the parameter amount is achieved.

[0017] The specific content and method of designing the multi-scale residual attention module at the decoder are: for an input image X, the image size is CxHxW (C represents the number of channels, H represents the height, and W represents the width), to reduce the parameter amount, the input feature is first reduced in dimension through a 1x1 convolution to obtain a C / 8xHxW feature map f1, then f1 passes through a three-branch structure in parallel to obtain feature maps f2, f3 and f4, the three-branch structure respectively includes a 3x3 convolution, two 3x3 convolutions and three 3x3 convolutions; the multi-branch structure is used to extract richer image features using multi-scale information, and the three feature maps are spliced to obtain better feature representation; after fusing the feature maps f2, f3 and f4 obtained by the three branches to obtain a feature map f5, in order to pursue higher segmentation accuracy, a channel attention SE module is used to adaptively adjust the weights of different channels, the SE module gives a greater weight to the channels containing rich useful information, and can more effectively focus on the target area in the feature map; at the same time, the network also reduces the full connection part of the SE module, and reduces the number of channels to C / 6, further reducing the parameter amount of the network; a 3x3 convolution operation is performed to obtain a feature map f6; batch normalization and ReLU function are performed after each convolution operation; finally, the input image X is added to the feature map f6 through the jump connection to obtain the final output. The purpose of using the jump connection is to solve the problem of gradient disappearance and degradation when the network is deep, which can compensate for the information loss in the previous convolution process to some extent and improve the reusability of the features.

[0018] Preferably, in step S3, in order to improve the accuracy of medical image segmentation and reduce the loss between the true value and the expected value, a pixel position-aware loss (PPA loss) is used as a loss function, and the loss function is defined as:

[0019]

[0020] wherein, and respectively represent the weighted IoU loss and the binary cross-entropy (BCE) loss based on global constraints and local (pixel-level) constraints. The weighted IoU objective is to optimize the global structure, treat all pixels equally, and ignore the differences between pixels. Unlike the IoU loss, the weighted IoU highlights the importance of difficult sample pixels by strengthening their weights. A weight α can be assigned to each pixel, and difficult sample pixels correspond to a larger α, while simple sample pixels will be assigned a smaller α.

[0021] The application provides a novel medical image segmentation method fusing multi-scale residual attention, designs a multi-scale residual attention network, significantly reduces the parameter quantity of the network while improving the generalization ability of medical image segmentation, and reduces the parameter quantity to one third of that of U-Net. The network takes U-Net as the main framework, reduces the channel number of the double convolution intermediate part in the encoder part, and moderately reduces the network parameters. In the decoder part, a multi-scale residual attention module is designed to capture the detailed features of medical images at different scales, improve the utilization rate of image feature extraction, alleviate the gradient disappearance problem and degradation problem caused by the network being too deep, and further improve the stability of the network. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The flowchart of the application is shown.

[0023] Figure 2 The model diagram of the application is shown.

[0024] Figure 3 The compressed double convolution module structure diagram of the application is shown.

[0025] Figure 4 The multi-scale residual attention module structure diagram of the application is shown.

[0026] Figure 5 The SE module structure diagram of the application is shown.

[0027] Figure 6 The four dataset visualization segmentation result diagram of the application is shown. DETAILED DESCRIPTION

[0028] In order to clarify the purpose, technical scheme and advantages of the application, the application will be further described in detail below in combination with the drawings of the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0029] Reference Figure 1 The medical image segmentation method fusing multi-scale residual attention shown in the embodiment includes the following steps:

[0030] Step 1: Prepare a private ultrasound fetal cerebellum dataset JSUAH-Cerebellum and three public biomedical datasets (polyp dataset Kvasir-SEG, cell nucleus dataset 2018Data Science Bowl (2018DSB) and skin cancer lesion dataset ISIC 2018), and preprocess all collected datasets, and set the input size of the image to 352*352. Randomly divide 80% of the samples in the preprocessed dataset into training set, 10% into validation set and 10% into test set. Finally, divide all the datasets into training set, validation set and test set.

[0031] Step 2: In order to solve the problem of small amount of biomedical image dataset, data augmentation is performed on the training set divided in step 1 to expand the training set. The experiment adopts a probability of 0.5 to take random vertical flip, random horizontal flip and random rotation (90°) to perform data augmentation, and adopts a multi-scale (i.e. {0.75, 1, 1.25}) strategy to train by scaling the picture;

[0032] Step 3: Input the training set obtained in step 2 into the multi-scale residual attention network (MRAU-Net) for iterative training. After verification by the validation set obtained in S1, the overall objective function model L with optimal training parameters is obtained.

[0033] As shown in Figure 2 , the multi-scale residual attention network (MRAU-Net) is based on the U-shaped network structure, and by improving the core components of the U-Net (such as the encoder and decoder parts), combining the multi-scale idea and the attention mechanism, the detailed spatial information of the image is extracted to obtain the optimal segmentation effect. In the encoder part, the multi-scale residual attention network (MRAU-Net) adopts the compression (Squeeze) strategy to improve the double convolution part of the U-Net encoder. The first layer convolution in the module reduces the number of channels, and the second layer convolution increases the number of channels, reducing the total parameter amount of the network. In the decoder part, the multi-scale residual attention (MRA) module uses the improved Inception module to process complex spatial and detailed features while reducing the parameter amount of the network, and combines the attention mechanism to improve the accuracy of medical image segmentation; the residual error solves the problem of gradient disappearance and degradation, further improving the stability of the network.

[0034] As shown in Figure 3As shown, the Multi-Scale Residual Attention Network (MRAU-Net) improves upon the dual convolutions in the encoder stage of the U-Net network. Specifically, the first 3×3 convolution is reduced to one-quarter of its original channel count, and the second 3×3 convolution is increased to twice its original channel count. Assuming M is the number of channels in the input feature map and N is the number of channels in the output feature map, since the computational cost of a single 3×3 convolution is 3×3×M×N, the number of parameters can be reduced by decreasing M or N. Each channel represents a different feature of the original image; increasing the number of channels allows for the extraction of more features from the original image using multiple convolutional kernels, and a wider network allows each layer to learn richer semantic features.

[0035] like Figure 4 As shown, the specific content and method of designing the multi-scale residual attention (MRA) module in the decoder are as follows: For the input image X, the image size is C×H×W (C represents the number of channels, H represents the height, and W represents the width). First, a 1×1 convolution is used to reduce the dimensionality of the input features, resulting in a C / 8×H×W feature map f1. Then, f1 is processed through a parallel three-branch structure to obtain feature maps f2, f3, and f4. The three branch structures respectively contain one 3×3 convolution, two 3×3 convolutions, and three 3×3 convolutions. The multi-branch structure is used to extract richer image features using multi-scale information, and the three feature maps are concatenated to obtain a better feature representation. After fusing the feature maps f2, f3, and f4 obtained from the three branches to obtain feature map f5, the channel attention (SE) module is used to adaptively adjust the weights of different channels. The fully connected part of the SE module is also reduced, reducing the number of channels to C / 6. Then, a 3×3 convolution operation is used to obtain feature map f6. Batch normalization and the ReLU function are performed after each convolution operation. Finally, we add the input image X to the feature map f6 through skip connections to obtain the final output. The purpose of using skip connections is to solve the gradient explosion and gradient vanishing problems that occur when the network is deep. This can compensate for the information loss in the previous convolution process to a certain extent and improve the reusability of features.

[0036] like Figure 5 As shown, the fully connected portion of the SE module in the network has also been reduced, further decreasing the number of parameters.

[0037] To further improve the accuracy of medical image segmentation, the network uses PPA loss to train the model, and the loss function is defined as follows:

[0038]

[0039] in, and respectively, represent the weighted IoU loss and binary cross-entropy (BCE) loss based on global constraint and local (pixel level) constraint, respectively. The weighted IoU objective is to optimize the global structure, treat all pixels equally, and ignore the differences between pixels. Unlike the IoU loss, the weighted IoU highlights the importance of difficult sample pixels by strengthening the weights of the difficult sample pixels. A weight α can be assigned to each pixel, with difficult sample pixels corresponding to a larger α, and simple sample pixels being assigned a smaller α.

[0040] Step 4: The test set image obtained in step 1 is input into the overall objective function model L with the optimal training parameters trained in step 3, to generate a segmentation result and save the image.

[0041] The effects under the present application can be further illustrated by the following experiments.

[0042] The experiments use the open-source PyTorch framework, and the training and testing run on an Ubuntu 20.04 64-bit computer operating system, with an AMD 5900x processor and an NVIDIA GeForce RTX 3090 graphics card with 24GB of video memory. All segmentation tasks are trained for 360 cycles, with the learning rate dynamically adjusted by a cosine annealing strategy, and the maximum number of iterations set to 40. The network parameters are optimized by the Adam optimizer, with an initial learning rate of 1e-4.

[0043] The proposed MRAU-Net is evaluated using the following indicators at multiple angles, including the average Dice coefficient (mDice), the average intersection over union (mean IoU, mIoU), the recall (Rec), the precision (Pre), and the F1-Score. The mDice is used to measure the similarity between two groups of samples. The mIoU is an indicator for measuring the accuracy of image segmentation, with the intersection of the predicted region and the actual region divided by the union of the predicted region and the actual region. The recall rate represents the ratio of positive samples that are correctly predicted. The precision is the proportion of correctly segmented samples among the predicted correct samples. The F1 score is an evaluation indicator that combines precision and recall, and is used to reflect the overall situation.

[0044] The following three public biomedical datasets (Kvasir-SEG, 2018 Data Science Bowl, and ISIC 2018) and one self-made fetal cerebellum ultrasound dataset (JSUAH-Cerebellum) are used to verify the experimental results.

[0045] Table 1 Comparison of experimental results on the JSUAH-Cerebellum dataset

[0046] Method mDice mIoU Rec Pre F1 Parameters U-Net 0.8520 0.8365 0.9173 0.9014 0.9020 3.1 x 10 7 ]]> FCN8s 0.8814 0.8373 0.9234 0.8970 0.9043 1.8 x 10 7 ]]> Att U-Net 0.9019 0.8409 0.9280 0.9080 0.9137 3.1 x 10 7 ]] U-Net++ 0.8482 0.8515 0.9405 0.8927 0.9132 0.9 x 10 7 ]] LCU-Net 0.8724 0.8452 0.9304 0.8915 0.9055 0.6 x 10 7 ]] DC-UNet 0.8737 0.8516 0.9398 0.8904 0.9092 2.4 x 10 7 ]] ECAU-Net 0.9135 0.8601 0.9404 0.9042 0.9198 3.1 x 10 7 ]]> Ours 0.9281 0.8710 0.9413 0.9230 0.9295 1.0 x 10 7 ]]

[0047] Table 2. Experimental comparison results on Kvasir-SEG dataset

[0048] Method mDice mIoU Rec Pre U-Net 0.8116 0.7217 0.7949 0.8726 U-Net++ 0.821 0.743 n / a n / a ResUNet++ 0.8133 0.7927 0.7064 0.8774 SFA 0.723 0.611 n / a n / a Ours 0.8864 0.8360 0.8996 0.8994

[0049] Table 3. Experimental comparison results on ISIC 2018 dataset

[0050]

[0051]

[0052] Table 4. Experimental comparison results on 2018 Data Science Bowl dataset

[0053] Method mDice mIoU Rec Pre U-Net 0.9080 0.8314 0.9029 0.9130 U-Net++ 0.7705 0.5265 0.7159 0.6657 ResUNet++ 0.9098 0.8370 0.9169 0.9057 Deeplabv3+(Xception) 0.8857 0.8367 0.9141 0.9081 DoubleUnet 0.9109 0.8429 0.9278 0.9020 PraNet 0.8751 0.7868 0.9182 0.8438 MSRF-Net 0.9244 0.8534 0.9402 0.9022 Ours 0.9314 0.8813 0.9472 0.9276

[0054] In combination Figure 6 In combination with Tables 1-4, the method not only has obvious advantages in various performance indicators over other models, but also achieves good segmentation results in the following cases: different scales of target regions, target regions with complex background occlusion, low contrast between target regions and surrounding background, and fuzzy boundaries between target regions and background.

[0055] The above only describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the scope of claims of the present application.

Claims

1. A medical image segmentation method of fusing multi-scale residual attention, characterized in that, Comprise: S1: collect a data set, pre-process the collected data set, and divide the pre-processed data set into a training set, a validation set and a test set according to an 8:1:1 ratio; input the test set image obtained in S1 into the overall objective function model L with the optimal training parameters trained in S3, generate a segmentation result and save the image; S2: data augmentation is performed on the training set divided in S1 to expand the training set. The specific method of data augmentation is: using a probability of 0.5 to randomly perform vertical flipping, horizontal flipping and 90° rotation on the training set image for data augmentation, and using a multi-scale strategy to perform training by scaling the image, wherein the multi-scale includes: 0.75, 1 and 1.

25. S3: input the training set obtained in S2 into a multi-scale residual attention network for iterative training; after verification by the verification set obtained in S1, an overall target function model L with optimal training parameters is obtained, and specific contents and design methods of the multi-scale residual attention network are as follows: the multi-scale residual attention network is composed of an encoder and a decoder, and has a total of 5 layers; in the encoder part, a compressed double convolution is used to extract features of an image; a multi-scale residual attention module is designed in the decoder by combining a multi-scale idea, a residual idea and an attention mechanism, so as to pay attention to key areas from a channel level, suppress irrelevant features, effectively improve feature acquisition capability of the network under the condition of controlling parameter quantity, and further improve segmentation precision; finally, the decoder generates an optimal segmentation result image; wherein the encoder part, the compressed double convolution is an improvement on the double convolution part of the original U-Net network; the method is as follows: firstly, the first 3*3 convolution is reduced to 1 / 4 of the original channel number, and secondly, the second 3*3 convolution is increased to twice the original channel number; the purpose of reducing parameter quantity is achieved by compressing the channel number in the middle of the double convolution; specific contents and methods of designing the multi-scale residual attention module in the decoder are as follows: for an input image X, the image size is C*H*W, C represents a channel number, H represents a height, and W represents a width; in order to reduce parameter quantity, a 1*1 convolution is used to reduce the input feature to obtain a C / 8*H*W feature map f1, then f1 passes through a three-branch structure to obtain feature maps f2, f3 and f4, the three-branch structure respectively includes one 3*3 convolution, two 3*3 convolutions and three 3*3 convolutions; the multi-branch structure is used to extract more abundant image features by using multi-scale information, and the three feature maps are spliced to obtain better feature representation; after fusing the feature maps f2, f3 and f4 obtained by the three branches to obtain a feature map f5, in order to pursue higher segmentation precision, a channel attention SE module is used to adaptively adjust weights of different channels, the SE module gives a larger weight to a channel containing rich useful information, and the target area in the feature map can be more effectively paid attention to; meanwhile, the full connection part of the network is also reduced, and the channel number is reduced to C / 6, so as to further reduce parameter quantity of the network; a 3*3 convolution operation is used to obtain a feature map f6; batch normalization and a ReLU function are executed after each convolution operation; finally, the input image X is added to the feature map f6 through a jump connection to obtain a final output; in order to improve accuracy of medical image segmentation and reduce loss between a true value and an expected value, a pixel position perception loss (PPA loss) is used as a loss function, and the loss function is defined as: wherein, and respectively represent the weighted IoU loss and the binary cross-entropy (BCE) loss based on global constraint and local constraint, the weighted IoU objective is to optimize the global structure, treat all pixels equally, and ignore the differences between pixels, unlike the IoU loss, the weighted IoU highlights the importance of difficult sample pixels by strengthening the weights of the difficult sample pixels, A weight α can be assigned to each pixel, difficult sample pixels correspond to a larger α, while simple sample pixels will be assigned a smaller α.

2. The medical image segmentation method of claim 1, wherein, In step S1, the data set is at least one private ultrasound fetal cerebellum data set and three public biomedical data sets, i.e. a polyp data set, a cell nucleus data set and a skin cancer lesion data set.

3. The medical image segmentation method of claim 1, wherein, In step S1, the pre-processing of the collected data set is to set the input size of all data sets to 352*352.

Images