A colorectal polyp image segmentation method based on deep learning

Abstract

The present invention discloses a colorectal polyp image segmentation method based on deep learning. First, a colorectal polyp segmentation data set is obtained; data preprocessing is performed; and then a colorectal polyp image segmentation model based on deep learning is constructed; the constructed colorectal polyp image segmentation model based on deep learning is trained through a training set, and a structural loss function is used to supervise the learning of each level of prediction results, and the prediction result of the last level is used as the final prediction result; the present invention constructs a colorectal polyp image segmentation model based on deep learning, accurately extracts the features of the colorectal polyp segmentation image and retains the details intact, and supervises the learning of the results using a structural loss function with a better effect in significant target detection, so that the prediction results are more accurate and have strong robustness, thereby realizing accurate segmentation of colorectal polyp images.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and specifically to a colorectal polyp image segmentation method based on deep learning. Technical Background

[0002] Colorectal cancer is the most common cancer in the world. Several studies have shown that early colonoscopy can reduce the incidence of colorectal cancer by 30%. Therefore, it is crucial to prevent colorectal cancer by screening and removing precancerous lesions through colonoscopy. Doctors can remove colorectal polyps before they develop into colorectal cancer based on the location and appearance information provided by colonoscopy, which is an effective colorectal cancer screening and prevention technology. Therefore, in clinical practice, the research on the precise positioning and segmentation technology of polyps has practical application value, which can assist endoscopists in detecting polyps and thus improve the accuracy. However, this technology is full of challenges, mainly for two reasons: on the one hand, the appearance of polyps is usually different, even if they are of the same type, there will be differences in size, color and texture, which greatly increases the difficulty of polyp segmentation; on the other hand, in colonoscopy images, the boundary between polyps and their surrounding mucosa is usually blurred and lacks the strong contrast required by the segmentation method, which makes polyps a camouflaged object that is difficult to locate and segment.

[0003] Therefore, an automatic and accurate polyp image segmentation method that can detect all potential polyps at an early stage is of great significance for the prevention of colorectal cancer. Summary of the invention

[0004] The present invention proposes a colorectal polyp image segmentation method based on deep learning. The method can accurately and quickly segment polyps, solving the problem of low segmentation accuracy caused by polyps having different sizes, colors and textures and being highly similar to surrounding tissue mucosa.

[0005] The technical solution of the present invention is achieved in this way:

[0006] A colorectal polyp image segmentation method based on deep learning, comprising the following steps:

[0007] Step (1), obtaining a colorectal polyp segmentation dataset.

[0008] The colorectal polyp segmentation dataset uses the existing CVC-ClinicDB dataset. The dataset is randomly divided into a training set and a test set, of which 80% is used as a training set to train the proposed model, and 20% is used as a test set to judge the robustness of the proposed model.

[0009] Step (2), data preprocessing;

[0010] The colorectal polyp segmentation image was cropped to a size of 352 × 352 using a bilinear interpolation algorithm;

[0011] Step (3), construct a colorectal polyp image segmentation model based on deep learning.

[0012] The colorectal polyp image segmentation model based on deep learning includes a ResNet50 feature extraction module, three CBR modules, a search and positioning module, and three recognition and refinement boundary modules RRM. First, the input image is passed through the ResNet50 feature extraction module to obtain 5 features X1, X2, X3, X4 and X5 from shallow to deep layers; then X1, X2 and X3 are enhanced through the CBR module; then, the search and positioning module is used to fuse the three deep features X1, X2, X3 to obtain a rough prediction result P1; finally, the recognition and refinement boundary module is used to obtain accurate prediction results P2, P3, P4 in a cascade manner.

[0013] Step (4), training the constructed colorectal polyp image segmentation model based on deep learning through the training set, using the structural loss function to perform supervised learning on each level of prediction results, and taking the prediction results of the last level as the final prediction results.

[0014] Step (5) verifies the model training results, inputs the test set data into the trained model, and then obtains the predicted results, and compares them with the actual results to verify whether they are effective. The predicted results and the actual results are both binary images with only 0 and 1. The different pixels of an image are obtained by subtracting them, and then divided by the total number of pixels of the entire image to obtain the MAE index. The smaller the index, the better.

[0015] The search and positioning module consists of three non-local modules and a feature aggregation decoder, which is used to fuse the information of three inputs, retain useful details, and remove noise, so as to obtain a rough prediction result.

[0016] The X1, X2 and X3 features obtained by the ResNet50 feature extraction module are passed through a non-local module respectively to obtain the Z1, Z2 and Z3 features; the Z1, Z2 and Z3 features are input into the feature aggregation decoder to obtain a rough global prediction result P1.

[0017] The non-local module is used to enhance features, capture long-distance dependencies, make features have rich contextual information, and increase the accuracy of polyp search. Specifically, a feature X is input and reshaped through three 1×1 convolutions to obtain three different features Y1, Y2 and Y3. The result of matrix multiplication of Y1 and Y2 is passed through the Sigmoid function and then matrix multiplication with Y3 to obtain a long-distance dependency feature, which is added to feature X after a 1×1 convolution to obtain the final enhanced feature Z.

[0018] The feature aggregation decoder, namely the FAD module, aggregates multiple layers of features to generate a rough polyp segmentation prediction map. The FAD module uses a pyramid structure to hierarchically aggregate adjacent features in an asymptotic manner, focusing on adjacent feature nodes, so as to aggregate details and semantic information and discard interference information. The operation steps are as follows: input Z1, Z2 and Z3 features, upsample Z1 and Z2 features to the same size as Z3 features; perform a dot multiplication operation on Z1 and Z2, and then pass through a convolutional layer to obtain Z12; perform a dot multiplication operation on Z2 and Z3, and then pass through a convolutional layer to obtain Z23; then perform a dot multiplication operation on Z12 and Z23 features, and then pass through a convolutional layer; finally, a 1×1 convolutional layer is used to obtain a rough global prediction result P1.

[0019] The identification and refinement boundary module has two inputs, one is the prediction result Prediction, that is, P1, P2 and P3, and the other is the features Features, that is, Z1, Z2 and Z3. First, the Prediction is upsampled to make its size consistent with the size of Features, and then the upsampled Prediction is subtracted from 1. Then 1-Prediction is multiplied with the result of Features after the CBR module to obtain a reverse attention map. This reverse attention map is spliced ​​and fused with Features, and the polyp boundary is further optimized after the CBR module. Finally, this result is multiplied with Features and convolved to obtain the final prediction result.

[0020] The structural loss function is derived from salient target detection, which consists of binary cross entropy loss and intersection-over-union loss. The structural loss function is as follows:

[0021] L=L wbce +L wiou

[0022] The binary cross entropy loss L wbce The expression is as follows:

[0023]

[0024] H is the output height, W is the output width, g ij Indicates the value of the ground truth corresponding to the pixel (i, j), p ij Indicates the value of the predicted result corresponding to the pixel (i, j), γ is a hyperparameter; α ij Represents the weight of pixel (i, j).

[0025] The α ij The expression is as follows:

[0026]

[0027] A ij Represents the area of ​​15 pixels above, below, left and right of pixel point (i, j); if α ij The larger the value is, the more it means that the pixel (i, j) is different from the surrounding pixels, so it is an important pixel and needs more attention.

[0028] The intersection-and-join ratio loss L wiou The expression is as follows:

[0029]

[0030] Beneficial effects of the present invention:

[0031] The present invention is reasonably designed and constructs a colorectal polyp image segmentation model based on deep learning. The features of the colorectal polyp segmentation image are accurately extracted and the details are kept intact. The structural loss function with better effect in significant target detection is used to supervise the learning of the results, so that the prediction results are more accurate and have strong robustness, thereby realizing the accurate segmentation of colorectal polyp images, which is of great significance to society. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is an overall implementation flow chart of an embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of the structure of the colorectal polyp image segmentation model based on deep learning of the present invention;

[0034] Figure 3 is a schematic diagram of the structure of a non-local module according to an embodiment of the present invention;

[0035] Figure 4 It is a schematic diagram of the structure of a feature aggregation decoder according to an embodiment of the present invention;

[0036] Figure 5 It is a schematic diagram of the module structure of the boundary identification and refinement module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in detail below with reference to the accompanying drawings, and the purpose and effects of the present invention will become more apparent.

[0038] like Figure 1 As shown, the overall implementation process of the present invention is as follows:

[0039] Step (1), obtain a colorectal polyp segmentation dataset. Use the CVC-ClinicDB colorectal polyp segmentation dataset available on the Internet as the dataset for this task. The dataset is divided into a training set and a test set. The training set is used to train the proposed model, and the test set is used to judge the robustness of the proposed model.

[0040] Step (2), input the colorectal polyp segmentation image. The colorectal polyp segmentation image is input into the model, and the image is cropped to a size of 352×352 using a bilinear interpolation algorithm;

[0041] Step (3), use the ResNet50 backbone network to extract multi-level features. The input image passes through the ResNet50 backbone network to generate 5 features from shallow to deep;

[0042] Step (4): Use the search and positioning module to fuse the three deep features to obtain a rough prediction result. Then, the prediction result is supervised by learning using the structural loss function;

[0043] Step (5) uses the recognition and refinement boundary module to obtain accurate prediction results through cascading. Then, supervised learning is performed on each level of prediction results using a structural loss function, and the prediction results of the last level are used as the final prediction results.

[0044] like Figure 2 As shown in FIG. 1 , the colorectal polyp image segmentation model based on deep learning has the following operation steps: input a 352×352 colorectal polyp segmentation image, extract features {X1, X2, X3, X4, X5} through the ResNet50 backbone network; then pass the X1, X2 and X3 features through a non-local module respectively to obtain Z1, Z2 and Z3 features; input the Z1, Z2 and Z3 features into Figure 4 The feature aggregation decoder (FAD) shown in FIG. 1 is used to output the prediction result P1; the prediction result P1 and Z1 are input into the identification and refinement boundary module (RRM) to output the prediction result P2; the prediction result P2 and Z2 are input into the identification and refinement boundary module (RRM) to output the prediction result P3; the prediction result P3 and Z3 are input into the identification and refinement boundary module (RRM) to output the prediction result P4. Among them, P1, P2, P3 and P4 are supervised by structural loss, and P4 will be used as the final prediction result.

[0045] like Figure 3 As shown, it is the non-local module of the present invention, which can enhance the features, capture long-distance dependencies, make the features have rich contextual information, and increase the accuracy of polyp search. Specifically, a feature X is input and reshaped through three 1×1 convolutions to obtain three different features Y1, Y2 and Y3. The result of matrix multiplication of Y1 and Y2 is passed through the Sigmoid function and then matrix multiplication with Y3 to obtain a long-distance dependency feature, which is added to the feature X after a 1×1 convolution to obtain the final enhanced feature Z.

[0046] like Figure 4 As shown, it is the feature aggregation decoder of the present invention, which aggregates multiple layers of features to generate a rough polyp segmentation prediction map. The FAD module uses a pyramid structure to hierarchically aggregate adjacent features in an asymptotic manner, focusing on adjacent feature nodes, so as to aggregate details and semantic information and discard interference information. Its operation steps are as follows: input Z1, Z2 and Z3 features, upsample Z1 and Z2 features to the same size as Z3 features; perform a dot multiplication operation on Z1 and Z2, and then pass through a convolutional layer CNN to obtain Z12; perform a dot multiplication operation on Z2 and Z3, and then pass through a convolutional layer CNN to obtain Z23; then perform a dot multiplication operation on Z12 and Z23 features, and then pass through a convolutional layer CNN and finally pass through a 1×1 convolutional layer to obtain the prediction result.

[0047] The convolutional layer CNN refers to a 3×3 convolutional layer, a BN layer and a ReLU activation function connected in series.

[0048] like Figure 5 As shown, the identification and refinement boundary module of the present invention has two inputs, one is the previous prediction result Prediction, that is, P1, P2 and P3, and the other is the feature Features, that is, Z1, Z2 and Z3. First, the Prediction is upsampled to make its size consistent with the size of Features, and then the upsampled Prediction is subtracted from 1. Then, 1-Prediction is multiplied with the result of Features after the CBR module to obtain a reverse attention map. This reverse attention map is spliced ​​and fused with Features, and the polyp boundary is further optimized after the CBR module. Finally, this result is multiplied with Features and convolved to obtain the final prediction result.

[0049] The structural loss function is derived from salient target detection, which consists of binary cross entropy loss and intersection-over-union loss. The structural loss function is as follows:

[0050] L=L wbce +L wiou

[0051] The binary cross entropy loss L wbce The expression is as follows:

[0052]

[0053] H is the output height, W is the output width, g ij Indicates the value of the groundtruth corresponding pixel (i, j), p ij Indicates the value of the predicted result corresponding to the pixel (i, j), γ is a hyperparameter; α ij Represents the weight of pixel (i, j).

[0054] The α ij The expression is as follows:

[0055]

[0056] A ij Represents the area of ​​15 pixels above, below, left and right of pixel point (i, j); if α ij The larger the value is, the more it means that the pixel (i, j) is different from the surrounding pixels, so it is an important pixel and needs more attention.

[0057] The intersection-and-join ratio loss L wiou The expression is as follows:

[0058]

[0059] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0060] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A colorectal polyp image segmentation method based on deep learning, characterized in that: The following steps are involved: Step (1), obtaining a colorectal polyp segmentation dataset; Step (2), data preprocessing; The colorectal polyp segmentation image was cropped to a size of 352 × 352 using a bilinear interpolation algorithm; Step (3), constructing a colorectal polyp image segmentation model based on deep learning; The colorectal polyp image segmentation model based on deep learning includes a ResNet50 feature extraction module, three CBR modules, a search and positioning module, and three recognition and refinement boundary modules RRM; first, the input image is passed through the ResNet50 feature extraction module to obtain 5 features X1, X2, X3, X4 and X5 from shallow to deep layers; then, X1, X2 and X3 are feature enhanced through the CBR module; then, the search and positioning module is used to fuse the three deep features X1, X2, X3, so as to obtain a rough prediction result P1; finally, the recognition and refinement boundary module is used to obtain accurate prediction results P2, P3, P4 in a cascade manner; Step (4), training the constructed colorectal polyp image segmentation model based on deep learning through the training set, using the structural loss function to perform supervised learning on each level of prediction results, and taking the prediction results of the last level as the final prediction results; Step (5), verify the model training results, input the test set data into the trained model, and then obtain the predicted results, and compare them with the actual results to verify whether they are effective; the predicted results and the actual results are both binary images with only 0 and 1. The different pixels of an image are obtained by subtracting them, and then divided by the total number of pixels of the entire image to obtain the MAE index. The smaller the index, the better; The search and positioning module consists of three non-local modules and a feature aggregation decoder, which is used to fuse the information of the three inputs, retain useful details, remove noise, and thus obtain a rough prediction result; The features X1, X2 and X3 obtained by the ResNet50 feature extraction module are passed through a non-local module to obtain the features Z1, Z2 and Z3. The features Z1, Z2 and Z3 are input into the feature aggregation decoder to obtain a rough global prediction result P1. The non-local module is used to enhance features, capture long-distance dependencies, make features have rich context information, and increase the accuracy of polyp search; specifically, a feature X is input and three 1×1 convolutions are performed and reshaped to obtain three different features Y1, Y2 and Y3; the result of matrix multiplication of Y1 and Y2 is passed through the Sigmoid function and then matrix multiplication with Y3 to obtain a long-distance dependency feature, which is added to the feature X after a 1×1 convolution to obtain the final enhanced feature Z; The feature aggregation decoder, i.e., the FAD module, aggregates multiple layers of features to generate a rough polyp segmentation prediction map; the FAD module uses a pyramid structure to hierarchically aggregate adjacent features in an asymptotic manner, focusing on adjacent feature nodes, so as to aggregate details and semantic information and discard interference information; the operation steps are as follows: input Z1, Z2, and Z3 features, upsample Z1 and Z2 features to the same size as Z3 features; perform a dot product operation on Z1 and Z2, and then pass through a convolutional layer to obtain Z12; perform a dot product operation on Z2 and Z3, and then pass through a convolutional layer to obtain Z23; then perform a dot product operation on Z12 and Z23 features, and then pass through a convolutional layer; finally, a rough global prediction result P1 is obtained through a 1×1 convolutional layer; The structural loss function is derived from salient target detection, which consists of binary cross entropy loss and intersection-over-union loss. The structural loss function is as follows: L=L wbce +L wiou The binary cross entropy loss L wbce The expression is as follows: H is the output height, W is the output width, g ij Indicates the value of the ground truth corresponding to the pixel (i, j), p ij Indicates the value of the predicted result corresponding to the pixel (i, j), γ is a hyperparameter; α ij Represents the weight of pixel (i, j); The α ij The expression is as follows: A ij Represents the area of ​​15 pixels above, below, left and right of pixel point (i, j); if α ij The larger the value, the more it means that the pixel (i, j) is different from the surrounding pixels, so it is an important pixel and needs more attention; The intersection-and-join ratio loss L wiou The expression is as follows:

2. According to the method for colorectal polyp image segmentation based on deep learning in claim 1, it is characterized in that: The colorectal polyp segmentation dataset described in step (1) adopts the existing CVC-ClinicDB dataset; the dataset is randomly divided into a training set and a test set, 80% of which is used as a training set to train the proposed model, and 20% is used as a test set to judge the robustness of the proposed model.

3. The colorectal polyp image segmentation method based on deep learning according to claim 2, characterized in that: The identification and refinement boundary module has two inputs, one is the prediction result Prediction, that is, P1, P2 and P3, and the other is the feature Features, that is, Z1, Z2 and Z3; first, the Prediction is upsampled to make its size consistent with the size of Features, and then 1 is subtracted from the upsampled Prediction; then 1-Prediction is multiplied with the result of Features after the CBR module to obtain a reverse attention map, and this reverse attention map is spliced ​​and fused with Features, and the polyp boundary is further optimized after the CBR module, and then this result is multiplied with Features and convolved to obtain the final prediction result.

Images