Colon polyp image segmentation method based on improved U-Net network
By introducing DUCK Block, MLFC jump connection and Coordinate attention mechanism modules into the U-Net network, the shortcomings of medical image segmentation model in colon polyp recognition accuracy and training samples are solved, and high accuracy and generalization capabilities are achieved The Dice index was significantly improved by segmentation of colon polyps.
Patent Information
- Application Number
- CN202411090558.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
AI Technical Summary
The existing medical image segmentation model has shortcomings in accurately positioning and identifying colon polyps details and edges, and has high requirements for the number of training samples, making it difficult to achieve high-precision segmentation on medical devices.
By introducing DUCK Block module, MLFC jump connection module and Coordinate attention mechanism module in the U-Net network, feature extraction and information transmission are enhanced, information loss is reduced, and original images are added to the encoder input part, and residual connection is used to process the image information.
The accuracy and generalization ability of colon polyp image segmentation is significantly improved, and polyps of different shapes and sizes can be accurately identified under few training samples and found their boundaries. The Dice index has increased from 85.50% to 95.37%.
Smart Images

Figure CN119006410A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a colon polyp image segmentation method based on an improved U-Net network. Background Art
[0002] In recent years, colorectal cancer has risen to the third most common type of cancer, and its mortality rate ranks second among all cancers. Most colon and rectal cancers start from polyps on the inner wall of the colon or rectum. Optical endoscopes play a vital role in the early prediction and timely treatment of colorectal cancer, so people will prevent this disease by performing regular colon and proctoscopy. However, the diagnosis of colon and proctoscopy depends on the medical experience of clinicians, who will inevitably experience visual fatigue and mental errors, which will lead to missed diagnoses in colon and proctoscopy, especially in chronic ulcerative colitis and small target cell polyps with signs of cancer.
[0003] The polyp segmentation system based on computer-aided diagnosis is conducive to improving the accuracy of colonoscopy examinations for patients, while reducing the morbidity and mortality of colorectal cancer. For the medical image segmentation technology for polyps, the segmentation method used previously relied on manual feature creation, such as the geometric features, volume features, intensity distribution, etc. of polyps. This method can produce acceptable polyp segmentation results in some cases. However, due to the irregular appearance of polyp images and the complex environment in the intestine, such methods still have a high missed detection rate.
[0004] The main models of medical image segmentation can be divided into two categories. Among the two models, the convolutional neural network, with U-Net as the structure developed for biomedical images, is widely used in various medical image segmentation tasks. The basic architecture of U-Net is an encoder-decoder structure, which has the characteristics of relatively simple and efficient model, and still has relatively good results in various medical image segmentation tasks. However, in many medical image segmentation tasks, diverse and complex input images emerge in an endless stream, which also leads to the use of other U-Net variants when dealing with partial segmentation tasks. PraNet is a U-type convolutional neural network specifically used for automatic polyp segmentation in colon endoscopy. It proposes to use a parallel reverse attention network to accurately segment polyps. The network first uses a balanced partial decoder (PPD) to extract and aggregate high-level features in the image, thereby obtaining a global map for subsequent processing steps, and then uses the reverse attention mechanism to mine boundary information to establish the relationship between different target areas and their boundaries. In addition, ParNet also adopts a cyclic cooperation mechanism to correct predictions that are inconsistent with the target results, effectively improving the segmentation accuracy. The evaluation results show that ParNet significantly improves the segmentation accuracy of polyp images, and has an advantage in real-time image processing efficiency, with an inference speed of about 50 frames per second.
[0005] Another type of model, Transformer, has been widely used recently in the fields of computer vision (CV) and natural language processing (NLP). It has a very good effect in preserving the global context information of the current subject area. The visual transformer developed based on these two uses the visual attention mechanism to aggregate context information and extract features from large image blocks. Since ViTs performs well in the field of computer vision, it has also made it more widely used in the field of medical image segmentation. Although the above-mentioned convolutional neural network has performed well in the task of polyp segmentation, it still has a gap compared with other methods that use transformer as encoder. These models usually use the visual transformer that has been pre-trained on a large dataset as the encoder of their model to extract features from the input image. After that, the extracted features will be transmitted to the decoder. The decoder uses different modules to fuse the received multi-scale features and finally integrate them into a single final output. A representative method in this series is FCN-transformer. FCN-transformer combines the advantages of traditional CNN and Transformer to construct a dual-branch feature extraction. The full convolution branch is used to extract fine features and the Transformer is used to extract coarse features. The features extracted by the dual branches are upsampled to obtain a global mapping to generate the final segmentation map. The advantage of this model is that it uses two main feature extraction methods to obtain results that are better than single-class methods, and the model structure is simple and effective.
[0006] Convolutional neural networks and transformer models are both important components of deep learning medical image segmentation. However, as a field such as medical image segmentation that pursues precision, it is obvious that substantial improvement in segmentation accuracy indicators is an important indicator to measure whether the method is advanced. How to improve the accuracy of segmentation results is the focus and difficulty of the current medical image segmentation field. To this end, the present invention proposes a colon polyp image segmentation method based on an improved U-Net network. Summary of the invention
[0007] The technical problem to be solved by the present invention is: how to enable the polyp segmentation model to accurately locate the details of each polyp on the segmented image and correctly find their edges; at the same time, it can obtain better segmentation results with fewer training samples and higher segmentation accuracy, so as to facilitate use in medical equipment, and provide a colon polyp image segmentation method based on an improved U-Net network.
[0008] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:
[0009] Follow these steps:
[0010] S1: Image preprocessing
[0011] Obtain a public dataset of colon polyp images and preprocess the images in the dataset according to the segmentation task requirements;
[0012] S2: Building a colon polyp image segmentation network
[0013] Based on the original U-Net network, the basic convolution part in the original U-Net network is replaced with the DUCK Block module, the jump connection part is replaced with the MLFC jump connection module and the residual block is added, the Coordinate attention mechanism module is added to the encoder and decoder, and the original image of the corresponding level is added to the encoder, and the residual connection is added to the bottom layer of the encoder, thereby obtaining the improved U-Net network, namely the colon polyp image segmentation network;
[0014] S3: Model training and validation
[0015] The colon polyp image segmentation network constructed in step S2 is trained using the training set, and a final colon polyp image segmentation model is selected based on the segmentation effect on the validation set;
[0016] S4: Model testing
[0017] The segmentation effect of the colon polyp image segmentation model is tested through an independent test set, and the segmentation mask map is output at the same time.
[0018] Furthermore, in step S1, the preprocessing process is as follows:
[0019] S11: Use the Albumenations library to perform data augmentation and data expansion on the images in the dataset;
[0020] S12: Divide the colon polyp images into training set, validation set and test set according to the set ratio;
[0021] S13: The images in the training set and the validation set are uniformly adjusted to a set size.
[0022] Furthermore, in step S2, the specific processing process is as follows:
[0023] S21: Select the original U-Net network as the base network, and use the DUCK Block module to replace the 3×3 convolutional layer except the last downsampling part;
[0024] S22: Use the MLFC jump connection module to replace the jump connection part in the original U-Net network, add residual blocks corresponding to the number of layers in the encoder and stack the MLFC jump connection modules;
[0025] S23: Add the Coordinate attention mechanism module after the convolutional blocks in the encoder and decoder;
[0026] S24: The corresponding size of the original image is added to each layer of the input part of the encoder, and the residual connection is used in the last part of the downsampling structure to capture and process the image information.
[0027] Furthermore, in the step S21, the DUCK Block module includes a first residual convolution block, a second residual convolution block, a third residual convolution block, a first dilated convolution block, a second dilated convolution block, and a separable convolution block connected in parallel; wherein the first residual convolution block includes one residual convolution block, the second residual convolution block includes two residual convolution blocks connected in series, the third residual convolution block includes three residual convolution blocks connected in series, the kernel sizes of the first residual convolution block, the second residual convolution block, and the third residual convolution block are 5×5, 9×9, and 13×13, respectively, the first dilated convolution block includes one dilated convolution block, the second dilated convolution block includes two dilated convolution blocks connected in series, the kernel sizes of the first dilated convolution block and the second dilated convolution block are 7×7 and 15×15, respectively, and the separable convolution block uses 1×N and N×1 convolution layers to combine to produce an N×N convolution effect.
[0028] Furthermore, in step S22, the MFLC jump connection module obtains convolutional feature maps of different levels and converts them into the same size and then connects them with the current level features, thereby enriching the current features through multi-level features. For the five different levels of features x in the network, 1 、x 2 、x 3 、x 4 、x 5 , the specific processing formula is as follows:
[0029] x comb,i =PConv ctot→ci (resize i (x 1 )∥resize i (x 2 )∥resize i (x 3 )∥resize i (x 4 )∥resize i (x 5 )
[0030] x i =PConv 2ci→ci (x comb,i ∥x i ),i=1,2,3,4,5
[0031] Among them, resize i (x j ) is to convert x j Resize to x i The size of C tot =C 1 +C 2 +C 3 +C 4 +C 5 .
[0032] Furthermore, in the step S23, the Coordinate attention mechanism module encodes the channel relationship and the long-distance dependency relationship by embedding the coordinate information and generating the coordinate attention. First, the horizontal and vertical directions are averagely pooled to obtain two 1-dimensional vectors. Next, the channel is compressed in the spatial dimension by using the Concat operation plus a 1x1 convolution layer. Then, the vertical and horizontal spatial information is encoded by the BN layer and the nonlinear activation function. Next, after the split, each of them is passed through a 1x1 convolution layer to obtain the input feature. Figure 1 The same number of channels is used, and finally the weighted fusion is normalized by the sigmoid activation function to obtain the final output.
[0033] Furthermore, the specific processing process of the Coordinate attention mechanism module is as follows:
[0034] S231: Given an input x, use two pooling kernels (H, 1) and (1, W) to encode the channels in the horizontal and vertical directions. The outputs of the kth channel with height h and width w are:
[0035]
[0036] S232: Connect the above formula and input it into function F 1 In the function F 1 is a 1×1 convolution transformation function, and the formula is as follows:
[0037] f=δ(F 1 ([z h ,z w ]))
[0038] Among them, δ is a nonlinear activation function, [.,.] represents the concatenation operation in the spatial dimension, and f is the feature map that encodes the spatial information in the horizontal and vertical directions;
[0039] S233: Split f into two tensors f along the horizontal and vertical directions h ∈R C / r×H and f W ∈R C / r×W , r is the scaling ratio of the control module size; then two 1×1 convolution transformation functions F h and F w f h and f W Convert them to tensors with the same size as the input x respectively. The expressions of the two tensors are as follows:
[0040] v h =σ(F h (f h ))
[0041] v w =σ(F w (f w ))
[0042] Among them, σ is the sigmoid activation function, v h and v w are the attention weights respectively;
[0043] S234: The output of the Coordinate attention mechanism module is:
[0044]
[0045] Furthermore, in step S3, the loss function Dice is used to train the colon polyp image segmentation network.
[0046] Furthermore, the loss function Dice is defined as follows:
[0047]
[0048] Among them, X and Y represent positive samples and negative samples respectively.
[0049] Compared with the prior art, the present invention has the following advantages: the colon polyp image segmentation method based on the improved U-Net network enhances the ability of the network to extract important features of polyps by adding a DUCK Block module to the original U-Net network, and at the same time, strengthens the attention to the detailed features of polyps; and proposes an MLFC jump connection module, and adds a residual block corresponding to the number of decoder layers before the module, and significantly reduces the information loss of the network in the encoder-decoder information transmission process through the jump connection, so that the feature transmission of each layer can obtain global features, increase the information exchange between various features, and the convolution block is used to reduce the semantic gap; the Coordinate attention mechanism module is added to the original U-Net network to enhance the long-distance dependency and position information sensitivity of the model, so as to more accurately identify the polyp position to extract more effective features; the corresponding size of the original image is added to each layer of the encoder input part, and the residual connection method is used in the bottom layer of the downsampling structure to capture and process the image information to reduce information loss; thereby, the colon polyp image segmentation module has excellent generalization ability, and can accurately identify and segment polyps of different shapes and sizes and find their boundaries even when the amount of training data is not sufficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is a flow chart of a method for segmenting colon polyps based on multi-module fusion of U-Net in an embodiment of the present invention;
[0051] Figure 2 (1) is an example diagram of a colon polyp image and a mask sample in the Kvasir-SEG dataset in an embodiment of the present invention;
[0052] Figure 2 (2) is an example diagram of colon polyp images and mask samples in the CVC-ClinDB dataset in an embodiment of the present invention;
[0053] Figure 2 (3) is an example diagram of colon polyp images and mask samples in the CVC-ColonDB dataset in an embodiment of the present invention;
[0054] Figure 2 (4) is an example diagram of colon polyp images and mask samples in the ETIS-LaribPolypDB dataset in an embodiment of the present invention;
[0055] Figure 3 (1) is a schematic diagram of the structure of the DUCK Block module in the improved U-Net network in an embodiment of the present invention;
[0056] Figure 3 (2) is a schematic diagram of the structure of Residual 1 in the DUCK Block module in an embodiment of the present invention;
[0057] Figure 3 (3) is a schematic diagram of the structure of Dilated 1 in the DUCK Block module in an embodiment of the present invention;
[0058] Figure 3 (4) is a schematic diagram of the structure of Dilated 2 in the DUCK Block module in an embodiment of the present invention;
[0059] Figure 3 (5) is a schematic diagram of the structure of Separated in the DUCK Block module in an embodiment of the present invention;
[0060] Figure 4 (1) is a schematic diagram of the structure of the residual block Residual 2 added to the encoder part of the improved U-Net network in an embodiment of the present invention;
[0061] Figure 4 (2) is a schematic diagram of the structure of the MLFC jump connection module in an embodiment of the present invention;
[0062] Figure 5 Schematic diagram of the structure of the Coordinate attention mechanism module in an embodiment of the present invention;
[0063] Figure 6 Schematic diagram of the overall structure of the improved U-Net network in the embodiment of the present invention;
[0064] Figure 7 It is a rendering of the comparison between the segmentation mask map generated by the improved U-Net network and the segmentation mask map generated by other advanced networks in an embodiment of the present invention;
[0065] Figure 8 (1) is a comparison diagram of the segmentation effect of colon polyp images in the Kvasir-SEG dataset in an embodiment of the present invention, wherein a is the original image of colon polyps, b is the segmentation mask image annotated by experts, and c is the actual segmentation mask image of the colon polyp image segmentation model of the present invention;
[0066] Figure 8 (2) is a comparison diagram of the segmentation effect of colon polyp images in the CVC-ClinDB dataset in an embodiment of the present invention, wherein a is the original image of colon polyps, b is the segmentation mask image annotated by experts, and c is the actual segmentation mask image of the colon polyp image segmentation model of the present invention;
[0067] Figure 8(3) is a comparison diagram of the segmentation effect of colon polyp images in the CVC-ColonDB dataset in an embodiment of the present invention, wherein a is the original image of colon polyps, b is the segmentation mask image annotated by experts, and c is the actual segmentation mask image of the colon polyp image segmentation model of the present invention;
[0068] Figure 8 (4) is a comparison diagram of the segmentation effects of colon polyp images in the ETIS-LaribPolypDB dataset in an embodiment of the present invention, wherein a is the original image of the colon polyp, b is the segmentation mask image annotated by an expert, and c is the actual segmentation mask image of the colon polyp image segmentation model of the present invention. DETAILED DESCRIPTION
[0069] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.
[0070] This embodiment provides a technical solution: a method for segmenting colon polyp images based on an improved U-Net network, comprising the following steps:
[0071] S1: Original image preprocessing.
[0072] In step S1, the specific contents are as follows:
[0073] S11: Obtain a public dataset of colon polyp images, and perform necessary preprocessing on the images in the dataset according to the task requirements, including data enhancement and data expansion using the Albumenations library, and divide them into training set, validation set and test set in a ratio of 8:1:1; the distribution of each dataset after preprocessing is shown in Table 1. Sample examples of the four datasets (Kvasir-SEG, CVC-ClinDB, CVC-ColonDB, ETIS-LaribPolypDB) are shown in Table 1. Figure 2 As shown in (1)-(4);
[0074] S12: The images in the training set and validation set are uniformly resized to 352×352. The purpose of scaling all the images in the training set and validation set is to reduce the amount of computation during the training process. Too large an image size will slow down the training process. The purpose of unifying the image size is to adapt to the input size requirements of the model. Different input sizes will make it impossible for the computing device to build a unified batch for training.
[0075] Table 1 Distribution of preprocessed datasets
[0076]
[0077]
[0078] S2: After analyzing the structure of the original U-Net network, we learned about the shortcomings of the network and added modules to replace them according to the shortcomings in various aspects. We compared the performance improvement after integrating different modules and found that the performance of the improved network was greatly improved.
[0079] In step S2, the specific contents are as follows:
[0080] S21: Select the U-Net network as the basic model, and use the DUCK Block module to replace the ordinary 3×3 convolution layer except the last downsampling part, so as to enhance the model's ability to recognize polyp features; the structural diagram of the DUCK Block module is shown in Figure 3 As shown in (1) in the figure, the structural diagrams of the four types of convolutions that make up the DUCK Block module are as follows: Figure 3 As shown in (2), (3), (4), and (5);
[0081] Specifically, in step S21, the residual convolution (Residual) is simulated according to a single, two series and three series combination to simulate the size of the convolution block, so as to form three types of convolution blocks with kernel sizes of 5×5, 9×9 and 13×13, and the dilated convolution (Dilated) simulates 7×7 and 15×15 convolution blocks according to a single and two series combination. The application of separable convolution (Separated) can simulate the effect of combining 1×N and N×1 convolutions to produce N×N size convolutions in the module. The value of N in the network is 3, and the six types of convolution blocks are connected in parallel to form a DUCK Block module;
[0082] S22: The MLFC jump connection module is used to replace the jump connection part in U-Net, so that the model can obtain global features at each layer. At the same time, the residual block Resdual 2 corresponding to the number of layers is added to the encoder part to reduce the semantic gap and stack the MLFC modules; the structural diagram of the added residual block Resdual 2 is shown in Figure 4 (1) As shown in the figure, the structural diagram of the MLFC module is as follows Figure 4 (2) as shown;
[0083] Specifically, in step S22, the MFLC jump connection module is introduced into the model to replace the jump connection part to enhance the multi-level feature fusion, obtain the convolution feature maps of different levels and convert them into the same size and then connect them, apply the convolution point-by-point operation to perform the actual representation and merge it with the required corresponding encoder feature map, so as to enrich the output feature map of each layer. For the five different levels of features x in the network, 1 、x 2 、x3 、x 4 、x 5 , the feature map can be enriched with multi-level features. The specific formula is as follows:
[0084]
[0085] Among them, resize i (x j ) is to convert x j Resize to x i The size of C tot =C 1 +C 2 +C 3 +C 4 +C 5 , these operations are performed separately for each level.
[0086] S23: Adding a Coordinate attention mechanism module after the convolutional block in the encoder-decoder structure can enhance the model's long-distance dependency and position information sensitivity, thereby more accurately identifying the location of polyps; the structural diagram of the Coordinate attention mechanism module is shown in Figure 5 As shown;
[0087] In step S23, the Coordinate attention mechanism module can distinguish the key parts from the wide segmentation feature map, and encode the channel relationship and long-distance dependency through coordinate information embedding and coordinate attention generation. First, average pooling is performed in the horizontal and vertical directions to obtain two 1D vectors, and then the Concat operation plus 1x1Conv is used in the spatial dimension to compress the channel, and then the vertical and horizontal spatial information is encoded through the BN layer and Non-linear. Next, after splitting, each is convolved by 1x1 to obtain the input feature. Figure 1 The same number of channels is used, and finally the weighted fusion is normalized by the sigmoid activation function to obtain the final output.
[0088] As an example, in step S23, given an input x, two pooling kernels (H, 1) and (1, W) are used to encode the channels in the horizontal and vertical directions, and the outputs of the kth channel with a height of h and a width of w are respectively:
[0089]
[0090] Connect the above formula and input it into function F 1 In the function F 1 is a 1×1 convolution transformation function, and the formula is as follows:
[0091] f=δ(F 1 ([z h ,z w ]))
[0092] Among them, δ is a nonlinear activation function, [.,.] represents the serial operation in the spatial dimension, f is the feature map that encodes the spatial information in the horizontal and vertical directions, and f is divided into two tensors along the horizontal and vertical directions, where r is the scaling ratio of the control module size, then the two tensors are f h ∈R C / r×H and f W ∈R C / r×W , and then use two 1×1 convolution transformation functions F h and F w f h and f W Convert them to tensors with the same size as input x respectively. The formulas of the two tensors are as follows:
[0093] v h =σ(F h (f h ))
[0094] v w =σ(F w (f w ))
[0095] Among them, σ is the sigmoid activation function, v h and v w are the attention weights respectively, and the output of the Coordinate attention mechanism module is:
[0096]
[0097] S24: The corresponding size of the original image is added to each layer of the encoder input part to supplement the feature information lost during the jump connection process. At the same time, the residual connection is used in the last part of the downsampling structure to capture and process the image information.
[0098] S3: Model training of polyp segmentation network based on the divided dataset
[0099] In this embodiment, step S3 includes the following sub-steps:
[0100] S31: The initial learning rate was set to 0.0001, the batch size was 4, and a total of 600 rounds of training were performed. The Adam optimizer was used and the Dice index of the monitoring model was set to change. The weight of the model with the best effect on the validation set was retained.
[0101] S32: In this embodiment, Table 2 and Table 3 show the impact on the experimental results after adding different modules, where +DUCK represents the impact on the experimental results after adding the DCUK Block module on the basis of the U-Net network. It can be seen that after adding the SE attention module, the Dice index increased from 85.50% to 90.36% on the Kvasir-SEG dataset and from 87.25% to 91.37% on the CVC-ClinDB. +DUCK+MLFC represents the impact on the experimental results after adding the DCUK Block module and the MLFC jump connection module to the U-Net network to realize the jump connection structure. The last one is the experimental result of the network model of the module after the Coordinate attention mechanism is finally added in the present invention. The final Dice index is 95.37% on the Kvasir-SEG dataset, and the final Dice index is 94.21% on the CVC-ClinDB dataset.
[0102] Table 2 Comparison of experimental results of adding different modules on the Kvasir-SEG dataset
[0103] method Dice Iou Precision Recall Accuracy U-Net (original) 0.8550 0.7474 0.8595 0.8515 0.9561 +DUCK+MLFC+DUCK 0.9036 0.8241 0.9231 0.8849 0.9699 +DUCK+MLFC 0.9335 0.8753 0.9345 0.9255 0.9785 +DUCK+MLFC+Coordinate 0.9421 0.8906 0.9542 0.9304 0.9821
[0104] Table 3 Comparison of experimental results of adding different modules on the CVC-ClinDB dataset
[0105] method Dice Iou Precision Recall Accuracy U-Net (add information) 0.8725 0.7738 0.8890 0.8586 0.9794 +DUCK+MLFC+DUCK 0.9137 0.8412 0.9291 0.8988 0.986 +DUCK+MLFC 0.9335 0.8753 0.9412 0.9258 0.9889 +DUCK+MLFC+Coordinate 0.9537 0.9115 0.9538 0.9536 0.9924
[0106] S4: Test the polyp segmentation network on the test set and output the segmentation mask map and data results
[0107] In this embodiment, step S4 includes the following sub-steps:
[0108] S41: In order to better test the actual effect of the network designed in the present invention in the field of polyp segmentation, we compare it with the existing advanced networks. Tables 4, 5, 6 and 7 show the experimental results of different networks on four polyp segmentation data sets, among which U-Net is the improved basic network of this model with added image processing. The HRNetV2 network is a convolutional neural network architecture for human posture estimation, which has a good effect on the segmentation task of small targets or unclear targets. Another network, the MSRF-Net network, is a convolutional neural network specifically used for medical image segmentation. The network designs a dual-scale dense fusion module (DSDF) to exchange multi-scale features with different receptive fields, which can effectively transmit high-level and low-level features to achieve efficient segmentation. PraNet is a convolutional neural network specifically used for automatic polyp segmentation in colon endoscopy. It proposes to use a parallel reverse attention network to accurately segment polyps. It significantly improves the segmentation accuracy of polyp images and has an advantage in the real-time processing efficiency of images, with an inference speed of about 50 frames per second. HardBlock-DFUS is also a convolutional neural network for polyp segmentation. The difference is that in the decoder part of the model, the network uses the Lawin Transformer structure and fuses it with an encoder block called HardBlock to improve the speed and accuracy of the model. FCN-transformer combines the advantages of traditional CNN and Transformer to construct a dual-branch feature extraction. The full convolution branch is used to extract fine features and the Transformer is used to extract coarse features. The advantage of this model is that it uses two main feature extraction methods to obtain results that are better than single-class methods. DUCK-Net is a new supervised convolutional neural network structure that uses an encoder-decoder structure with a residual downsampling mechanism and custom convolution blocks to capture and process multi-resolution image information in the encoded segment to perform accurate segmentation tasks. The original images of colon polyps in each dataset, the segmentation mask maps annotated by experts, and the actual segmentation mask maps of the network are provided by Figure 7 As shown in Figure 2, the comparison between the segmentation mask map generated by the improved U-Net and the segmentation mask map generated by other advanced networks is shown in Figure 2. Figure 8 As shown in (1)-(4) in Figure 3, after a thorough comparison of all the results, it can be seen that our improved network model (Ours) has a Dice index that is not only higher than the original model on the four datasets used, but is also better than the best result. This can also be seen from the visual mask image.
[0109] Table 4 Segmentation results of various networks on the Kvasir-SEG dataset
[0110] method Dice Iou Precision Recall Accuracy U-Net (same augmentation) 0.8655 0.7629 0.8593 0.8718 0.9563 HRNetV2 0.853 0.7438 0.8778 0.8297 0.9539 MSRF-Net 0.8508 0.7404 0.8993 0.8074 0.9543 HarDNet-DFUS (with pretrianed) 0.8626 0.7584 0.9351 0.8005 0.9583 PraNet (with pretrianed) 0.9094 0.8339 0.9599 0.864 0.9738 FCN-Transformer(with pretrianed) 0.922 0.8554 0.9238 0.9203 0.9749 DUCK-Net 0.9343 0.8769 0.935 0.9337 0.9789 Ours 0.9421 0.8906 0.9542 0.9304 0.9821
[0111] Table 5 Segmentation results of various networks on the CVC-ClinDB dataset
[0112]
[0113]
[0114] Table 6 Segmentation results of various networks on the CVC-ColonDB dataset
[0115] method Dice Iou Precision Recall Accuracy U-Net (same augmentation) 0.8032 0.7037 0.81 0.8274 0.9807 HRNetV2 0.6383 0.4687 0.5858 0.701 0.9565 MSRF-Net 0.8371 0.7198 0.8603 0.8151 0.9829 HarDNet-DFUS (with pretrianed) 0.7398 0.587 0.95 0.6057 0.9761 PraNet (with pretrianed) 0.9131 0.8401 0.9657 0.8659 0.9901 FCN-Transformer(with pretrianed) 0.9073 0.8304 0.9107 0.904 0.9899 DUCK-Net 0.945 0.8952 0.9488 0.9406 0.9903 Ours 0.9537 0.9115 0.9538 0.9536 0.9924
[0116] Table 7 Segmentation results of various networks on the ETIS-LaribPolypDB dataset
[0117] method Dice Iou Precision Recall Accuracy U-Net (same augmentation) 0.7984 0.6969 0.8322 0.7724 0.9734 HRNetV2 0.472 0.3089 0.4645 0.4797 0.9433 MSRF-Net 0.7791 0.6382 0.9191 0.6762 0.9797 HarDNet-DFUS (with pretrianed) 0.8662 0.764 0.9708 0.7819 0.9869 PraNet (with pretrianed) 0.8827 0.79 0.9825 0.8013 0.9877 FCN-Transformer(with pretrianed) 0.9163 0.8455 0.9633 0.8736 0.9915 DUCK-Net 0.9324 0.8734 0.9539 0.9118 0.993 Ours 0.9474 0.9001 0.9564 0.9386 0.9948
[0118] S42: In order to verify the performance of the model on unseen experimental data and show its generalization ability, we designed a mutually verified experimental setting. First, each dataset is tested separately and the performance is compared with other methods. Then, after training on the Kvasir-SEG dataset, it is tested on the test set of the CVC-ClinicDB dataset. Conversely, after training on the CVC-ClinicDB dataset, it is tested on the test set of the Kvasir-SEG dataset. Tables 8 and 9 show the experimental results of our model under this experimental setting. Through comparison and verification experiments, it is found that our model has good adaptability and prediction accuracy for new data and unseen data. The effect of such cross-dataset experiments can show that our model has good generalization ability without any pre-training.
[0119] Table 8 Results of training on the Kvasir-SEG dataset and testing of each network on the CVC-ClinicDB dataset
[0120] method Dice Iou Precision Recall Accuracy U-Net (same augmentation) 0.701 0.5397 0.664 0.7423 0.9441 HRNetV2 0.7457 0.5945 0.7642 0.728 0.9561 MSRF-Net 0.6763 0.5109 0.6965 0.6572 0.9444 HarDNet-DFUS (with pretrianed) 0.5784 0.4068 0.5974 0.5605 0.9263 PraNet (with pretrianed) 0.7744 0.6319 0.9494 0.6539 0.9638 FCN-Transformer(with pretrianed) 0.8314 0.7114 0.8839 0.7848 0.9719 DUCK-Net 0.8014 0.6686 0.8851 0.7321 0.9679 Ours 0.8129 0.6847 0.8557 0.7741 0.9707
[0121] Table 9 Results of training on the CVC-ClinicDB dataset and testing of each network on the Kvasir-SEG dataset
[0122] method Dice Iou Precision Recall Accuracy U-Net (same augmentation) 0.5369 0.367 0.4374 0.6951 0.8068 HRNetV2 0.5531 0.3822 0.4242 0.7944 0.7931 MSRF-Net 0.5152 0.3469 0.3939 0.7443 0.7742 HarDNet-DFUS(with pretrianed) 0.7272 0.5714 0.918 0.6021 0.9261 PraNet(with pretrianed) 0.6852 0.5212 0.7647 0.6207 0.913 FCN-Transformer(with pretrianed) 0.88 0.7858 0.9659 0.8082 0.9645 DUCK-Net 0.7525 0.6032 0.6873 0.8314 0.9119 Ours 0.8229 0.6991 0.7791 0.8719 0.9427
[0123] In summary, the colon polyp segmentation method based on multi-module fusion of U-Net in the above embodiment enhances the ability of the network to extract important features of polyps and strengthens the attention to the detailed features of polyps by adding a DUCK Block module to the original U-Net network; after that, an MLFC jump connection module that replaces the jump connection form is proposed, and a residual block corresponding to the number of decoder layers is added before the module, and the information loss of the network in the encoder-decoder information transmission process is significantly reduced through the jump connection, so that the feature transmission of each layer can obtain global features, increase the information exchange between various features, and the convolution block is used to reduce the semantic gap; in the present invention, the added Coordinate attention mechanism module strengthens the long-distance dependency and position information sensitivity of the model, so as to more accurately identify the polyp position to extract more effective features; each layer of the encoder input part is added accordingly The corresponding size of the original image, and the residual connection method is used at the bottom of the downsampling structure to capture and process image information to reduce information loss. Finally, the effectiveness of the improvement is proved through various comparative experiments, and the Dice index is successfully improved from 85.50% to 94.21% on the Kvasir-SEG dataset, and the Dice index is improved from 87.25% to 95.37% on the CVC-ClinDB dataset. The improved network model is also compared with other methods, proving that it has the best Dice index and Iou index; at the same time, Pytorch is used to visualize the output segmentation map, and it can be seen that the output mask map is almost consistent with the mask annotated by professionals, so it can be seen that the automatically segmented polyp area is accurate.
[0124] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A colon polyp image segmentation method based on an improved U-Net network, characterized in that: The following steps are involved: S1: Image preprocessing Obtain a public dataset of colon polyp images and preprocess the images in the dataset according to the segmentation task requirements; S2: Building a colon polyp image segmentation network Based on the original U-Net network, the basic convolution part in the original U-Net network is replaced with the DUCK Block module, the jump connection part is replaced with the MLFC jump connection module and the residual block is added, the Coordinate attention mechanism module is added to the encoder and decoder, and the original image of the corresponding level is added to the encoder, and the residual connection is added to the bottom layer of the encoder, thereby obtaining the improved U-Net network, namely the colon polyp image segmentation network; S3: Model training and validation The colon polyp image segmentation network constructed in step S2 is trained using the training set, and a final colon polyp image segmentation model is selected based on the segmentation effect on the validation set; S4: Model testing The segmentation effect of the colon polyp image segmentation model is tested through an independent test set, and the segmentation mask map is output at the same time.
2. The method for segmenting colon polyp images based on an improved U-Net network according to claim 1, characterized in that: In step S1, the preprocessing process is as follows: S11: Use the Albumenations library to perform data augmentation and data expansion on the images in the dataset; S12: Divide the colon polyp images into training set, validation set and test set according to the set ratio; S13: The images in the training set and the validation set are uniformly adjusted to a set size.
3. The method for segmenting colon polyp images based on an improved U-Net network according to claim 1, characterized in that: In step S2, the specific processing process is as follows: S21: Select the original U-Net network as the base network, and use the DUCK Block module to replace the 3×3 convolutional layer except the last downsampling part; S22: Use the MLFC jump connection module to replace the jump connection part in the original U-Net network, add residual blocks corresponding to the number of layers in the encoder and stack the MLFC jump connection modules; S23: Add the Coordinate attention mechanism module after the convolutional blocks in the encoder and decoder; S24: The corresponding size of the original image is added to each layer of the input part of the encoder, and the residual connection is used in the last part of the downsampling structure to capture and process the image information.
4. The method for segmenting colon polyp images based on an improved U-Net network according to claim 3, characterized in that: In the step S21, the DUCK Block module includes a first residual convolution block, a second residual convolution block, a third residual convolution block, a first dilated convolution block, a second dilated convolution block, and a separable convolution block connected in parallel; wherein the first residual convolution block includes one residual convolution block, the second residual convolution block includes two residual convolution blocks connected in series, the third residual convolution block includes three residual convolution blocks connected in series, the kernel sizes of the first residual convolution block, the second residual convolution block, and the third residual convolution block are 5×5, 9×9, and 13×13, respectively, the first dilated convolution block includes one dilated convolution block, the second dilated convolution block includes two dilated convolution blocks connected in series, the kernel sizes of the first dilated convolution block and the second dilated convolution block are 7×7 and 15×15, respectively, and the separable convolution block uses 1×N and N×1 convolution layers to combine to produce an N×N convolution effect.
5. The method for segmenting colon polyp images based on an improved U-Net network according to claim 4, characterized in that: In step S22, the MFLC jump connection module obtains convolutional feature maps of different levels and converts them into the same size and then connects them with the current level features, thereby enriching the current features through multi-level features. For the five different levels of features x1, x2, x3, x4, and x5 in the network, the specific processing formula is as follows: Among them, resize i (x j ) is to convert x j Resize to x i The size of C tot =C1+C2+C3+C4+C5.
6. The method for segmenting colon polyp images based on an improved U-Net network according to claim 5, characterized in that: In the step S23, the Coordinate attention mechanism module encodes the channel relationship and long-distance dependency through coordinate information embedding and coordinate attention generation. First, average pooling is performed on the horizontal and vertical directions to obtain two 1-dimensional vectors. Next, the Concat operation is used to add a 1x1 convolution layer to compress the channel in the spatial dimension. The vertical and horizontal spatial information is encoded through the BN layer and the nonlinear activation function. After splitting, each of them passes through a 1x1 convolution layer to obtain the same number of channels as the input feature map. Finally, the sigmoid activation function is used to normalize the weighted fusion to obtain the final output.
7. The method for segmenting colon polyp images based on an improved U-Net network according to claim 6, characterized in that: The specific processing process of the Coordinate attention mechanism module is as follows: S231: Given an input x, use two pooling kernels (H, 1) and (1, W) to encode the channels in the horizontal and vertical directions. The outputs of the kth channel with height h and width w are: S232: The above formulas are connected and input into function F1. Function F1 is a 1×1 convolution transformation function, and the formula is as follows: f=δ(F1([z h ,z w ])) Among them, δ is a nonlinear activation function, [.,.] represents the concatenation operation in the spatial dimension, and f is the feature map that encodes the spatial information in the horizontal and vertical directions; S233: Split f into two tensors f along the horizontal and vertical directions h ∈R C / r×H and f W ∈R C / r×W , r is the scaling ratio of the control module size; then two 1×1 convolution transformation functions F h and F w f h and f W Convert them to tensors with the same size as the input x respectively. The expressions of the two tensors are as follows: v h =σ(F h (f h )) v w =σ(F w (f w )) Among them, σ is the sigmoid activation function, v h and v w are the attention weights respectively; S234: The output of the Coordinate attention mechanism module is:
8. The method for segmenting colon polyp images based on an improved U-Net network according to claim 7, characterized in that: In step S3, the colon polyp image segmentation network is trained using the loss function Dice.
9. The method for segmenting colon polyp images based on an improved U-Net network according to claim 8, characterized in that: The loss function Dice is defined as follows: Among them, X and Y represent positive samples and negative samples respectively.
Citation Information
Cited By
Target identification method and system, electronic equipment and storage medium
CN119785007A
