A multi-scale intestinal polyp segmentation method fusing attention mechanism

Abstract

The application discloses a multi-scale intestinal polyp segmentation method fusing an attention mechanism. The basic features of the method are as follows: 1. a multi-scale effective semantic fusion module is constructed to extract more abundant and effective multi-scale semantic information; 2. a new encoding-decoding deep network segmentation model is constructed to improve polyp segmentation accuracy; the method extracts sufficient context information and global information under different receptive fields, and filters out as many features useless for the segmentation task as possible, overcomes the defects that semantic information is limited and a large amount of redundancy exists in the traditional encoding-decoding structure, and has excellent segmentation and generalization performance for two-dimensional enteroscopy images with polyp regions of different shapes and different sizes.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology for image segmentation, specifically, a novel deep network construction method for polyp segmentation in two-dimensional colonoscopy images. Technical Background

[0002] In the medical field, experts rely on their expertise and extensive clinical experience to detect and manually segment polyp regions within colonoscopy images. However, the diverse appearance of polyps and their blurred boundaries with the background not only increase the workload for medical professionals but also lead to missegmentation due to individual patient differences and the high degree of subjectivity among doctors. Combining deep learning with medical imaging technology to achieve precise segmentation of polyp regions can reduce the workload of medical professionals, accelerate polyp segmentation, and improve the accuracy of polyp segmentation.

[0003] In recent years, with the improvement of computing power of graphics processing units (GPUs), convolutional neural networks (CNNs) have been widely used in visual recognition problems, achieving remarkable results in pixel-level image segmentation. Researchers have proposed the Unet deep network segmentation model based on an encoder-decoder structure. This model uses the encoding path to extract the semantic information of the image required for segmentation and the decoding path to gradually restore the resolution, ultimately obtaining a predicted segmentation label map with the same resolution as the input image, thus promoting the development of medical image segmentation. To alleviate the large semantic differences between features in the encoder-decoder paths, researchers proposed the MultiResUNet deep network segmentation model, which gradually reduces these semantic differences by introducing convolutional chains with residual connections in skip connections. To achieve the cyclic reuse of features extracted from building blocks, researchers proposed the Bidirectional O-shaped Network (Bio-Net) deep segmentation model. By constructing bidirectional skip connections between the encoder and decoder paths, an O-shaped inference path is formed, enabling feature reuse and better feature refinement. To mitigate the interference of numerous useless and redundant features in the network on segmentation tasks, researchers proposed the Attention U-Net deep network segmentation model. By introducing a novel attention gate model for medical image segmentation into the decoding path, the network can automatically learn to focus on target object regions of different sizes and shapes, highlighting features useful for the segmentation task. To effectively deepen the network and alleviate the gradient vanishing problem, researchers proposed the R2U-Net deep network segmentation model. By introducing residual connections and recurrent convolutions into the encoding and decoding path building blocks, the network can acquire more expressive features, improving segmentation performance without increasing the number of parameters. To explore the importance of features at different depths of the encoding path, researchers proposed the Unet++ deep network segmentation model. By introducing short and long connections to share the same feature extractor, features at different levels are restored by different decoding paths, allowing the network to automatically learn the importance of features at different depths.

[0004] Existing deep network segmentation models based on encoder-decoder structures typically employ continuous convolution and downsampling to extract multi-scale semantic information from images. However, due to the limited receptive field size, the extracted contextual information is often very limited and insufficient for effective segmentation, frequently leading to suboptimal segmentation masks or inaccurate object boundaries. Furthermore, because convolution operations indiscriminately fuse spatial and channel information from all feature maps within the local receptive field, a large number of features that are ineffective for segmentation exist in the network, wasting significant computational resources and negatively impacting segmentation performance. Summary of the Invention

[0005] This invention discloses a multi-scale intestinal polyp segmentation method that integrates an attention mechanism. This method constructs a novel encoder-decoder deep learning image segmentation network framework that can extract sufficient contextual information and global information under different receptive fields, while filtering out features useless for the segmentation task as much as possible. This allows for the acquisition of richer and more effective semantic information, overcoming the limitations and redundancy of semantic information in traditional encoder-decoder structures. It exhibits excellent segmentation and generalization performance on two-dimensional colonoscopy images of polyp regions with different shapes and sizes.

[0006] The technical solution to achieve the purpose of this invention is: a multi-scale intestinal polyp segmentation method incorporating attention mechanisms, comprising the following steps:

[0007] The first step is to input the colonoscopy image I from the training set into the deep network segmentation model. After the input image I is processed by the encoding path consisting of four identical encoding blocks, the high-level encoding feature map F1 is obtained. Then, the feature map F1 is processed by two consecutive ordinary convolutional layers to obtain the input feature map F2 of the decoding path.

[0008] The second step is to obtain feature map F3 after feature map F2 is processed by a decoding path consisting of four identical decoding blocks. Then feature map F3 is processed by a single ordinary convolutional layer to obtain model output predicted segmentation label map P with the same resolution size as the model input image I.

[0009] The third step involves using the real segmentation label map T in the training set and the predicted segmentation label map P output by the model to calculate the model loss through a loss function. Then, the network is iteratively trained using an optimization algorithm to continuously reduce the model loss until the deep network segmentation model achieves the best segmentation effect.

[0010] The method of this invention has a simple structure. Compared with the prior art, its significant features are: (1) semantic information and spatial structure information of colonoscopy images are extracted through network learning; (2) the constructed multi-scale effective semantic fusion module can obtain multi-scale semantic information under different receptive fields and ensure that these feature information are beneficial to the segmentation task as much as possible, avoiding the phenomenon of rough segmentation results caused by limited receptive fields and a large number of redundant features in the traditional encoder-decoder structure; (3) the network model is an end-to-end segmentation model, and each learning module is trained and inferred in a unified manner without the need for additional supervised training process, and has excellent generalization and segmentation performance. Attached Figure Description

[0011] Figure 1 This is a structural diagram of the method of the present invention.

[0012] Figure 2 This is a schematic diagram of a multi-scale effective semantic fusion module.

[0013] Figure 3 These are images showing the segmentation results of the CVC-ClinicDB dataset using different methods. Detailed Implementation

[0014] The implementation process of the present invention will be further described below with reference to the accompanying drawings.

[0015] This invention provides a multi-scale intestinal polyp segmentation method incorporating attention mechanisms, combined with Figure 1 The steps of this invention are as follows:

[0016] The first step involves inputting colonoscopy images I from the training set into a deep network segmentation model. Image I is processed through an encoding path consisting of four identical encoding blocks E1, E2, E3, and E4 to obtain a high-level encoding feature map F1. Feature map F1 is then processed by two consecutive 3×3 ordinary convolutional layers to obtain the input feature map F2 for the decoding path. Each encoding block consists of two consecutive 3×3 ordinary convolutional layers and a max-pooling layer. The max-pooling layer has a pooling window size of 2 and a stride of 2. The feature map input to the max-pooling layer is halved in size and doubled in number of channels after processing. The output feature maps C1, C2, C3, and C4 of the second ordinary convolutional layer in E1, E2, E3, and E4 are then used for subsequent skip connection operations.

[0017] In the second step, feature map F2 is processed through a decoding path consisting of four identical decoding blocks D1, D2, D3, and D4 to obtain feature map F3. Feature map F3 is then processed by a single 1×1 ordinary convolutional layer to obtain the model output predicted segmentation label map P, which has the same resolution as the model input image I. Each decoding block consists of a transposed convolutional layer, a 3×3 ordinary convolutional layer, a multi-scale effective semantic fusion module, and another 3×3 ordinary convolutional layer in sequence. The feature map input to the transposed convolutional layer is doubled in size and its number of channels is halved after processing by the transposed convolutional layer. Decoding block D... i In (i = 1, 2, 3, 4), the input feature map of the first ordinary convolutional layer is the output feature map of the previous transposed convolutional layer and C. i (i = 1, 2, 3, 4) concatenation along the channel dimension;

[0018] Combination Figure 2 The operation steps of the multi-scale effective semantic fusion module are as follows:

[0019] (1) The input feature map X of the multi-scale effective semantic fusion module is processed by four parallel average pooling layers with pooling ratios of 1, 2, 3 and 6 respectively to obtain a four-level pyramid feature map.

[0020] (2) Each level of the pyramid feature map is processed by a 1×1 ordinary convolutional layer to reduce the number of channels to 1 / 4 of the number of channels in the module input feature map X. Then, it is upsampled by bilinear interpolation to restore the resolution of the module input feature map X. Finally, the feature maps are concatenated along the channel dimension to obtain the feature map. Where H, W, and C are the height, width, and number of channels of feature map Y1, respectively;

[0021] (1) Feature map Y1 undergoes further feature redirection processing. Specifically, feature map Y1 first passes through a global average pooling layer, the pooling formula of which is expressed as:

[0022]

[0023] in, H represents the feature map obtained after global average pooling, where c represents the channel index and H represents the feature map obtained after global average pooling. c W c Let Y1 and Y2 represent the height and width of the feature map in channel c, respectively. Then, feature map Y2 passes through the first fully connected layer to compress the number of channels to C / r, where r represents the scaling parameter. After activation by the ReLU activation function, it enters the second fully connected layer to restore the number of channels to C, resulting in feature map Y3. Finally, the sigmoid activation function is used to limit the values ​​in feature map Y3 to the range [0, 1], which are then multiplied by coefficients onto each channel of the original input feature map X, as follows:

[0024] O = S * X

[0025] in, These represent the coefficients obtained after sigmoid activation, with * indicating channel-by-channel multiplication. This represents the feature map obtained after feature redirection processing.

[0026] (4) Concatenate the feature map O with the original input feature map X of the module in the channel dimension, and perform the same feature redirection process as above again to finally obtain the output feature map of the multi-scale effective semantic fusion module, which is used as the input of the subsequent ordinary convolutional layer.

[0027] The third step involves calculating the model loss using the real segmentation label map T from the training set and the predicted segmentation label map P from the model output through a loss function. Then, the network is iteratively trained using an optimization algorithm to continuously reduce the model loss until the deep network segmentation model achieves the best segmentation effect. In the predicted segmentation label map P, pixels with a value of 0 belong to the background region, and pixels with a value of 1 belong to the polyp region. The loss function used is the binary cross-entropy loss function commonly used in image segmentation problems.

[0028] Loss=-[T logP+(1-T)log(1-P)]

[0029] The effects of the present invention will be further explained below with reference to simulation experiments.

[0030] Simulation conditions

[0031] The simulation experiments used the CVC-ClinicDB dataset. The CVC-ClinicDB dataset is a database of frames extracted from colonoscopy videos by the Barcelona Computer Vision Center in Spain. This dataset contains 612 8-bit RGB images and corresponding ground truth segmentation labels, with an image size of 384×288. In the experiment, 459 colonoscopy images were randomly selected as the training set, and the remainder as the test set. The images were first resized to a uniform 256×256, and after preprocessing by dividing all pixel values ​​by 255, they were input into the deep neural segmentation network. The experiment used Dice, Jaccard, Sensitivity, and Precision coefficients as evaluation metrics. Furthermore, comparison methods included: UNet deep network segmentation model, Attention U-Net deep network segmentation model, BA-Net deep network segmentation model, Dil.ResFCN deep network segmentation model, Bio-Net deep network segmentation model, FCN deep network segmentation model, and MultiResUNet deep network segmentation model.

[0032] In the experiment, all convolutional kernels of the network were initialized using the he_normal distribution. Except for the last ordinary convolutional layer, which used the sigmoid activation function, all other ordinary convolutional layers in the network used the ReLU activation function. In all multi-scale effective semantic fusion modules, the scaling factor r was set to 2. Furthermore, the network optimizer used the Adam optimizer with an initial learning rate of 0.0001. If the loss did not decrease after every 10 iterations, the learning rate was reduced to 0.5 times the original rate. The number of iterations was set to 200, and a Dropout training strategy with a ratio of 0.5 was used to prevent overfitting. The experimental environment was as follows: CPU: E5-2640 v2, GPU: Nvidia Quadro K4000, Memory: 64GB, Tensorflow-1.12.

[0033] Analysis of Simulation Experiment Results

[0034] Table 1 shows the segmentation results of the proposed method on the CVC-ClinicDB dataset in simulation experiments. The experimental results demonstrate that the proposed method is highly effective for polyp segmentation in the CVC-ClinicDB dataset, achieving the best segmentation result among all compared methods. The CVC-ClinicDB dataset contains polyp regions of varying sizes, shapes, and locations. Due to the inherent limitations of traditional encoder-decoder network models—limited receptive fields and interference from numerous redundant features—these methods often exhibit inaccurate segmentation results on this dataset. The proposed method effectively extracts multi-scale contextual information under different receptive fields, achieving better segmentation results and proving its effectiveness. Predicted segmentation label maps obtained through different methods are shown in [reference]. Figure 3 It is evident that the method of this invention can obtain more accurate and refined polyp segmentation results. The above demonstrates that the method of this invention can effectively extract and filter multi-scale semantic information, increasing the receptive field while ensuring the effectiveness of feature information as much as possible, thus achieving better segmentation results.

[0035] Table 1. Segmentation results of different methods on the CVC-ClinicDB dataset.

[0036]

[0037]

[0038] In summary, the method of the present invention has the ability to extract richer contextual information and ensure that these multi-scale feature information are beneficial to the segmentation task as much as possible. It can be applied to supervised segmentation of colonoscopy images with polyp regions of different shapes and sizes, and has excellent generalization and segmentation performance.

Claims

1. A multi-scale intestinal polyp segmentation method incorporating attention mechanisms, characterized in that, Includes the following steps: The first step is to process the colonoscopy images in the training set. Input deep network segmentation model, input image The high-level coding feature map is obtained after coding path operations consisting of four identical coding blocks. Subsequently, feature maps The input feature map of the decoding path is obtained after processing through two consecutive ordinary convolutional layers. ; The second step is feature mapping. The feature map is obtained after a decoding path consisting of four identical decoding blocks. Subsequently, feature maps The image is obtained after processing by a single ordinary convolutional layer, which is similar to the model input image. Models of the same resolution output predicted segmentation label maps ; The third step is to utilize the real segmentation label images from the training set. and model output predicted segmentation label map The model loss is calculated using a loss function, and then the network is iteratively trained using an optimization algorithm to continuously reduce the model loss until the deep network segmentation model achieves the best segmentation effect. In the first step, the colonoscopy images in the training set Input deep network segmentation model, input image After being processed by four identical coded blocks The high-level encoded feature map is obtained after the encoding path operation. Subsequently, feature maps The input feature map of the decoding path is obtained after processing through two consecutive ordinary convolutional layers. Each encoded block consists of two consecutive ordinary convolutional layers and one max-pooling layer, which are obtained respectively. Output feature map of the second ordinary convolutional layer Used for subsequent skip connection operations; In the second step, the feature map After being decoded by four identical blocks The feature map is obtained after the decoding path is constructed. Subsequently, feature maps The image is obtained after processing by a single ordinary convolutional layer, which is similar to the model input image. Models of the same resolution output predicted segmentation label maps Each decoding block consists of a transposed convolutional layer, a regular convolutional layer, a multi-scale effective semantic fusion module, and a regular convolutional layer in sequence; decoding block The input feature map of the first ordinary convolutional layer is the output feature map of the previous transposed convolutional layer and... splicing along the channel dimension; The operation steps of the multi-scale effective semantic fusion module are as follows: (1) Input feature map of multi-scale effective semantic fusion module After processing by four parallel average pooling layers with different pooling ratios, a four-level pyramid feature map is obtained. (2) Each level of the pyramid feature map is processed by a regular convolutional layer to reduce the number of channels to that of the module input feature map. Number of channels The feature map was then recovered to the module input feature map through bilinear interpolation upsampling. The resolution size is then used to stitch the images together along the channel dimension to obtain the feature map. ,in , , Feature maps Height, width, and number of channels; (3) Feature map Further feature redirection processing is performed on the feature map. First, it goes through a global average pooling layer. The pooling formula for the global average pooling layer is expressed as: in, This represents the feature map obtained after global average pooling, where c represents the channel index. These represent the height and width of the feature map in the c-th channel, respectively; subsequently, the feature map... The number of channels is compressed after the first fully connected layer. , This represents the scaling parameter; after the ReLU activation function is activated, the second fully connected layer restores the number of channels to [the specified value]. , to obtain feature map Finally, the sigmoid activation function is used to transform the feature map. The value within is limited to The feature map is multiplied by the original input feature map of the module within the range as a coefficient. On each channel, it is represented as: Among them, S This represents the coefficients obtained after sigmoid activation. Indicates channel-by-channel multiplication, O This represents the feature map obtained after feature redirection processing. (4) Feature map With the original input feature map of the module The feature maps are concatenated along the channel dimension and then subjected to the same feature redirection process as described above. This process yields the output feature map of the multi-scale effective semantic fusion module, which is then used as the input to subsequent ordinary convolutional layers.

2. The multi-scale intestinal polyp segmentation method incorporating attention mechanisms according to claim 1, characterized in that: In the third step, the real segmentation label images from the training set are used. and model output predicted segmentation label map The model loss is calculated using a loss function, and then the network is iteratively trained using an optimization algorithm to continuously reduce the model loss until the deep network segmentation model achieves the best segmentation effect. The loss function used is the binary cross-entropy loss function commonly used in image segmentation problems. 。

Images