A polyp image segmentation method based on deep fusion of boundary cues

Abstract

The present invention discloses a polyp image segmentation method based on deep fusion of boundary clues. By constructing a boundary clue deep fusion network, the intestinal polyp image is input into the backbone network to extract feature information of different levels to guide the extraction of polyp regions; a boundary clue mining module is constructed to obtain boundary detail information with the help of partial hierarchical features to achieve more accurate positioning of the polyp region; a foreground target enhancement module is constructed to enhance the learning of the overall structural features of the foreground target by combining the network with the inverted attention branch, so that the network focuses on the foreground target; a deep feature fusion module is constructed to integrate boundary information and foreground information, and the segmentation performance is further improved through cascade feature fusion and boundary information guidance, and finally the polyp segmentation result is obtained using the trained model. The present invention overcomes the limitations of polyps in terms of boundary uncertainty and morphological diversity, and adopts a weighted joint scale difference loss optimization model in training to achieve high-precision recognition and segmentation of polyp regions.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and medical image technology, and specifically relates to a polyp image segmentation method based on deep fusion of boundary cues. Background Art

[0002] Colorectal cancer (CRC) is one of the most common cancer diseases that endangers human health, and its source is generally adenomatous polyps. In order to make up for the shortcomings of traditional manual colonoscopy through deep learning methods and accurately judge hidden polyps in complex environments, a large number of deep learning integration frameworks for polyp segmentation have been proposed, but the existing models still have several limitations and challenges. For example, when there are foreign objects in the intestinal environment, the complex background environment significantly interferes with the segmentation performance of the model, greatly increasing the difficulty of locating polyps. In addition, the diversity of polyp morphology constitutes another major segmentation problem. Therefore, it is necessary to efficiently refine and integrate cross-level multi-scale features through convolutional neural networks to improve model adaptability and segmentation accuracy.

[0003] In recent years, deep learning technology has been widely used in medical image segmentation, especially in polyp segmentation during colorectal endoscopy. The common convolutional neural network-based architecture U-Net adopts an encoder-decoder structure, combining low-level and high-level features through jump connections, which can effectively retain spatial information, but has poor anti-interference performance and is insufficient in practical applications; PraNet aggregates high-level features through parallel partial decoders to generate a global map to roughly locate polyps, and then uses a reverse attention module to refine the features. Despite this, due to the inherent limitations of convolutional neural networks, the segmentation accuracy and robustness of the model still need to be improved, so it is particularly important to improve the existing model to improve the performance of polyp segmentation.

[0004] Current polyp segmentation methods mainly segment the internal core area of ​​the polyp, and extract the texture or color feature information of the polyp by selecting a suitable imaging method. This type of method does not consider the edge information of the polyp, resulting in blurred polyp segmentation boundaries and the segmentation accuracy needs to be further improved. Obviously, incorporating detailed information features of the edge is very important for improving the accuracy of polyp segmentation. Summary of the invention

[0005] In order to solve the shortcomings of the prior art, the present invention adopts the following technical solutions by fusing polyp boundary clue information and fusing and extracting multi-level features to improve the accuracy of polyp segmentation and avoid boundary blur:

[0006] A polyp image segmentation method based on deep fusion of boundary cues includes the following steps:

[0007] Step 1: Acquire polyp images;

[0008] Step 2: Construct a polyp image segmentation model based on deep fusion of boundary cues, including the following steps:

[0009] Step 2.1: Pass the polyp image through the backbone network to extract hierarchical features;

[0010] Step 2.2: Extract high-fidelity boundary information from hierarchical features of different scales, and construct a polyp segmentation mask through multi-level fusion to obtain polyp boundary detail information;

[0011] Step 2.3: Perform cross-level local union of hierarchical features of different scales to extract cross-level features for foreground target enhancement corresponding to the hierarchical features. With the help of the inverted attention mechanism of foreground target enhancement, the learning effect of small and complex polyp features is improved to obtain high-level semantic features;

[0012] Step 2.4: Deeply integrate the high-level semantic features corresponding to each foreground target enhancement with the boundary detail information, adopt a cross-layer information transmission structure, fuse multi-level features in a cascade manner, use 1×1 convolution to compress the number of different feature channels, so that the output features of the lower-level deep feature fusion module can be directly connected to the upper layer, share and fuse feature representations at different levels, and obtain deep fusion features for polyp image segmentation;

[0013] Step 3: Train the polyp image segmentation model to obtain a trained model for polyp image segmentation.

[0014] Furthermore, in step 2.2, boundary clue mining is used to obtain hierarchical features of different scales, output the boundary area contour of the polyp, and perform loss calculation to optimize the model. Specifically, boundary clue mining includes the following steps:

[0015] Step 2.2.1: Convert multiple top-level features into corresponding multiple feature gradient maps respectively;

[0016] Use specific convolution to convert the hierarchical features F1 and F2 into two gradient maps M1 and M2 respectively:

[0017]

[0018]

[0019] Among them, Conv K±45 Represents a 3×3 convolution operation with specific operators in the positive 45° direction and the negative 45° direction:

[0020]

[0021]

[0022] Step 2.2.2: Concatenate multiple feature gradient maps along the channel dimension;

[0023] The feature gradient map M2 is concatenated with the feature gradient map M1 along the channel dimension after 1×1 convolution:

[0024]

[0025] Among them, Conv1 represents 1×1 convolution, || represents channel concatenation operation, and F 12 Represents the output result after feature concatenation;

[0026] Step 2.2.3: Concatenate the underlying features with the concatenated features to obtain aggregate features containing boundary information, and extract boundary clues from the aggregate features as detailed information of the polyp boundary;

[0027] The underlying feature F5 is combined with the concatenated feature F 12 After 1×1 convolution, the dimensions are reduced and then spliced ​​after upsampling to match the scale to obtain aggregated features containing boundary information. , the splicing result After convolution and Sigmoid activation function, boundary clues F are extracted e :

[0028]

[0029]

[0030] in, Represents the Sigmoid activation function, G CBNR-2 It means that after 2 Conv-BatchNormational-Relu operations, U means upsampling operation.

[0031] Furthermore, the foreground object enhancement in step 2.3 includes the following steps:

[0032] Step 2.3.1: Through the cascaded foreground object enhancement method, each level of foreground object enhancement takes the output features of the previous level of foreground object enhancement and the corresponding level features as input;

[0033] Step 2.3.2: In the inverted attention branch of foreground target enhancement, invert the output of the previous level foreground target enhancement to obtain the inverted attention weight;

[0034] Step 2.3.3: Perform element-wise subtraction on the inverted attention weight and its corresponding level feature to extract the difference information between the features, and then perform element-wise multiplication on the corresponding level feature to obtain the complementary enhanced foreground target feature as the high-level semantic information.

[0035] Furthermore, in step 2.3.2, the common information between different features is fully extracted by using a cross-level local joint decoder and a method of multiplying multiple feature elements;

[0036] For the top-down 5-layer network, the bottom-level feature F5 is first refined by upsampling and convolution, and then element-wise multiplied with the level feature F4 to extract and enhance the common features. , extract the common enhanced features of level features F3, F4 and F5 in the same way , further capturing the interdependence of cross-level features, then upsampling and 3×3 convolution are performed on the level feature F5 to obtain the feature map and Splicing, fusing feature information of different resolutions, and obtaining deeper features through 3×3 convolution , and then After upsampling and 3×3 convolution, Splicing, combining details and high-level semantic information, and refining features through 3×3 convolution ,at last After 3×3 convolution and 1×1 convolution, dimensionality reduction is performed to obtain significant cross-level features. :

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] Among them, Conv1 represents a 1×1 convolution operation, G CBNR Indicates one Conv-BatchNormational-Relu operation.

[0043] Furthermore, in step 2.3.3, the attention weight W is reversed. k And the corresponding level feature F k Perform element subtraction to extract the difference information between features, and then add it to the corresponding level feature F k Perform element-wise multiplication to obtain complementary enhanced foreground target features :

[0044]

[0045] Among them, k represents the number of layers, t represents the index of the hierarchical feature corresponding to the foreground target enhancement, Represents element-wise multiplication.

[0046] Furthermore, the step 2.4 comprises the following steps:

[0047] Step 2.4.1: According to the requirements of different levels, the boundary clue information is downsampled to different degrees to obtain multi-scale boundary clue information. The multi-scale boundary clue information and foreground target features are used as the input of clue feature perception to generate a multi-level feature representation. The output of the previous order deep feature fusion is concatenated with the current multi-level feature representation and 1×1 convolution is used for dimensionality reduction to obtain multi-scale integrated features. The channels after dimensionality reduction are evenly divided into multiple layers of features for interlaced fusion to obtain multiple fused neighboring features:

[0048]

[0049]

[0050]

[0051]

[0052] in, Indicates boundary clue information, represents the foreground target feature, FTDM represents the clue feature perception, represents the output of the previous order deep feature fusion, represents dimensionality reduction convolution, represents multi-scale integration features, div represents the division operation, represents the features obtained by division, The expansion coefficient is rj = j, It represents the intermediate result after the partition feature is expanded. represents the Sigmoid activation function, represents element-wise multiplication, represents element-wise addition, Represents multiple adjacent features after fusion;

[0053] Step 2.4.2: Concatenate the 4 adjacent features after fusion, and obtain the features through 1×1 convolution, and then through residual connection and 3×3 convolution operation, finally obtain the deep fusion features:

[0054]

[0055]

[0056] Among them, Conv3 represents a 3×3 convolution operation.

[0057] Furthermore, the clue feature perception in step 2.4.1 takes the downsampled boundary clue information and foreground target features as input, performs element-wise multiplication to obtain their common feature information, adds the multiplication result to the foreground target features, retains the original feature information, and then uses 3×3 convolution to obtain the initial fusion feature , introduce the attention mechanism to perform weighted summation on the initial fusion features, aggregate the convolution features, calculate the corresponding channel attention weights through the fully connected layer and the Sigmoid activation function, and multiply the aggregated convolution features by the channel attention weights element by element to enhance important features. Finally, compress the number of channels through 1×1 convolution to generate preliminary fusion features:

[0058]

[0059]

[0060]

[0061] Among them, FC represents the fully connected layer and ATT represents the attention mechanism.

[0062] Furthermore, in step 2.3, the input of the top-level foreground target enhancement is the second-level features and the features output by the previous-level foreground target enhancement module, and the features output by the second-level foreground target enhancement are only input into the corresponding top-level deep feature fusion;

[0063] In the step 2.4, only the second-bottom-level boundary clue information and the bottom-level foreground target features are used as the input of the bottom-level deep feature fusion, and the output result is the bottom-level deep fusion feature; the top-level deep feature fusion module takes the top-level boundary clue information, the second-level foreground target features and the output of the previous-level deep feature fusion module as input, and the output deep fusion features are directly used as the output result and participate in deep supervision.

[0064] Furthermore, in step 3, a weighted joint scale difference loss and a weighted binary cross entropy loss supervision model are used to train the polyp image segmentation model, including the following steps:

[0065] Step 3.1: Define the predicted area as P and the true area as G, take the union area of ​​the two minus the reduced intersection area of ​​the two, and get the first result:

[0066]

[0067]

[0068]

[0069] Among them, P represents the predicted area, G represents the real area, represents a union, represents the intersection, Indicates the result obtained, the reduction factor is the ratio of the intersection of the predicted area and the true area to the true area;

[0070] Step 3.2: Replace the reduction factor in step 3.1 to obtain the second result:

[0071]

[0072] The second reduction factor ɑ is a hyperparameter less than or equal to the minimum factor, and is calculated adaptively according to the ratio, ɑ<1 and ɑ>0;

[0073] Step 3.3: Construct the joint scale difference loss L by the ratio of the first result to the second result SDou :

[0074] ;

[0075] Step 3.4: During the training process, the deep fusion features and boundary detail information are used as results to calculate the joint scale difference loss respectively, and finally the joint scale difference loss of each part is added to obtain the total joint scale difference loss:

[0076]

[0077] in, represents the joint scale difference loss of each feature;

[0078] Step 3.5: Calculate the binary cross entropy loss for each binary image, and then weight the total joint scale difference loss and total binary cross entropy loss of the binary image with global and local constraints to obtain the weighted joint scale difference loss and weighted binary cross entropy loss, and then get the final loss:

[0079]

[0080]

[0081] in, represents the binary cross entropy loss for each binary image, and denote the weighted joint scale difference loss and binary cross entropy loss based on global and local constraints, respectively.

[0082] Furthermore, in step 1, the collected microscopic polyp image data set is divided into a training set and a test set; in step 3, the polyp image segmentation model is trained using the training set. The method further includes:

[0083] Step 4: Use the best polyp image segmentation model obtained after training to test the polyp images in the test set, obtain the polyp segmentation results after the test, and perform evaluation to obtain the polyp image segmentation model that has completed the training;

[0084] Step 5: Input the polyp image to be segmented into the trained polyp image segmentation model and output the polyp segmentation result.

[0085] The advantages and beneficial effects of the present invention are:

[0086] The polyp image segmentation method based on deep fusion of boundary cues of the present invention combines boundary cues and foreground target detail features on the basis of Res2Net-50 network, and deeply fuses these two kinds of information to generate a fine polyp mask, which effectively improves the accuracy of polyp segmentation; through the boundary clue mining module, the polyp boundary contour is introduced to supervise the training process, and the polyp contour features are enhanced according to the boundary cues by using a specific operator, which significantly improves the difference between the polyp boundary and the background, and improves the distinction of the polyp area; through the foreground target enhancement module, the reverse attention strategy is introduced to strengthen the foreground target enhancement in a targeted manner. The feature learning of small-scale and complex-structure polyps improves the segmentation performance of the model in complex situations; the boundary detail information and high-level semantic information are deeply integrated through the deep feature fusion module to obtain a more accurate and complete polyp mask; the training process is supervised by the joint scale difference loss, the training process of the model is optimized, and the training effect of the model is improved; the present invention also introduces a module parallel cascade mechanism, which strengthens the correlation between features at different levels while performing specific processing on hierarchical features, and gradually enhances and fuses features at different levels to ensure full mining and filtering of multi-level features and improve the utilization rate of feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 The present invention is a flowchart of a polyp image segmentation method based on deep fusion of boundary cues in an embodiment of the present invention.

[0088] Figure 2 It is a structural diagram of a boundary clue mining module in an embodiment of the present invention.

[0089] Figure 3 Schematic diagram of the structure of the foreground object enhancement module in the embodiment of the present invention

[0090] Figure 4 4 is a schematic diagram of the structure of a cross-level local joint decoder according to an embodiment of the present invention.

[0091] Figure 5 It is a structural diagram of a deep feature fusion module in an embodiment of the present invention.

[0092] Figure 6 It is a structural diagram of a clue feature perception module in an embodiment of the present invention.

[0093] Figure 7 2 is a schematic diagram of a formula for a joint scale difference loss and its corresponding area in an embodiment of the present invention.

[0094] Figure 8 This is a comparison chart of the visualization effects of the present invention and other methods obtained through experiments. DETAILED DESCRIPTION

[0095] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0096] like Figure 1 As shown, a polyp image segmentation method based on deep fusion of boundary cues includes the following steps:

[0097] Step 1: Collect colonoscopic polyp images and divide the dataset into training set and test set;

[0098] Data D = {dataset 1, data set 2, data set 3, data set 4, data set 5}, where data set 1 contains 1000 polyp images, data set 2 contains 612 polyp images, data set 3 contains 380 polyp images, data set 4 contains 196 polyp images, and data set 5 contains 60 images. 900 and 550 polyp images are selected from data set 1 and data set 2 respectively as training sets, and the others are used as test sets.

[0099] Step 2: Construct a polyp image segmentation model based on deep fusion of boundary cues, including the following steps:

[0100] Step 2.1: Use the Res2Net-50 backbone network to extract hierarchical features;

[0101] In the embodiment of the present invention, the hierarchical features extracted by the Res2Net-50 backbone network are F={F1, F2, F3, F4, F5}.

[0102] Step 2.2: Use the boundary clue mining module to extract high-fidelity boundary information from hierarchical features of different scales, and construct a polyp segmentation mask through a multi-level fusion strategy to obtain boundary detail information;

[0103] The boundary clue mining module takes F1, F2 and F5 as input, and finally outputs the boundary area contour of the polyp, and performs loss calculation to optimize the model training process, such as Figure 2 As shown, boundary clue mining includes the following steps:

[0104] Step 2.2.1: Use specific convolution to convert features F1 and F2 into two gradient maps M1 and M2 respectively. The specific expression method is as follows:

[0105] ,

[0106] Among them, Conv K±45 It represents a 3×3 convolution operation with specific operators in the 45° direction and the negative 45° direction, which is specifically expressed as:

[0107] ,

[0108] Step 2.2.2: Concatenate the feature gradient map M2 with the feature gradient map M1 along the channel dimension after 1×1 convolution. The specific expression is as follows:

[0109]

[0110] Among them, Conv1 represents 1×1 convolution, || represents channel concatenation operation, and F 12 It is the output result after feature concatenation.

[0111] Step 2.2.3: Combine the underlying feature F5 with F 12 After 1×1 convolution, the dimension is reduced and then up-sampled to match the scale and then spliced ​​to obtain the aggregated features containing boundary information. , the splicing result After convolution and Sigmoid activation function, boundary clues F are extracted e , the specific expression method is as follows:

[0112]

[0113]

[0114] in, Represents the Sigmoid function, G CBNR-2 It means that after 2 Conv-BatchNormational-Relu operations, U means upsampling operation.

[0115] Step 2.3: With the help of the inverted attention mechanism of the foreground object enhancement module, the learning effect of the characteristics of small and complex polyps is improved to obtain high-level semantic features, such as Figure 3 As shown, foreground target enhancement includes the following steps:

[0116] Step 2.3.1: By cascading the foreground object enhancement modules, each level of foreground object enhancement module is output from the previous level of foreground object enhancement module. (k=2,...,5) and the corresponding level features F k as input.

[0117] Step 2.3.2: In the inverted attention branch of the foreground object enhancement module, the output of the previous level foreground object enhancement module is Upsample to the required scale, pass through the activation function and then invert to get the inverted attention weight W k , the specific expression method is as follows:

[0118]

[0119] Where E represents an all-one matrix. When k=5, the hierarchical features F3, F4, and F5 are aggregated through the cross-level local joint decoder as Input, and get the reverse attention weight W5.

[0120] like Figure 4 As shown in Figure 2, the cross-level local joint decoder uses the method of multi-feature element multiplication to fully extract the common information between different features. First, feature F5 is refined by upsampling and convolution, and element-wise multiplication is performed with F4 to extract and enhance the common features. , extract the common enhancement features of F3, F4 and F5 in the same way , further capturing the interdependence of cross-level features, and then upsampling F5 and performing 3×3 convolution to obtain the feature map and Splicing, fusing feature information of different resolutions, and obtaining deeper features through 3×3 convolution , and then After upsampling and 3×3 convolution, Splicing, combining details and high-level semantic information, and refining features through 3×3 convolution ,at last Obtain significant cross-level features through 3×3 convolution and 1×1 convolution dimensionality reduction , the specific expression method is as follows:

[0121]

[0122]

[0123]

[0124]

[0125]

[0126] Conv1 represents a 1×1 convolution operation, G CBNRIndicates one Conv-BatchNormational-Relu operation.

[0127] Step 2.3.3: Set W k And the corresponding level feature F k Perform element subtraction to extract the difference information between features, and then add it to the corresponding level feature F k Perform element-wise multiplication to obtain complementary enhanced foreground target features , the specific expression method is as follows:

[0128]

[0129] in Represents element-wise multiplication.

[0130] The foreground object enhancement module selects feature F k Foreground target enhancement is performed. The input of the top-level foreground target enhancement module is the second-level feature F2 and the output of the previous layer foreground target enhancement module. , Only the corresponding top-level deep feature fusion module is input.

[0131] Step 2.4: Through the deep feature fusion module, high-level semantic features and boundary detail information are deeply integrated, such as Figure 5 As shown in Figure 2, deep feature fusion includes the following steps:

[0132] Step 2.4.1: According to the needs of different levels, the boundary clue information F e Perform different degrees of downsampling to obtain multi-scale boundary clue information , with boundary clue information and foreground target features As the input of the clue feature perception module, a multi-level feature representation is generated, and the output of the previous order deep feature fusion module The output features of the clue feature perception module are concatenated and processed by 1×1 convolution to obtain multi-scale integrated features. , the channel after dimensionality reduction is evenly divided into four layers of features for staggered fusion, and the fused four adjacent features are obtained (j=1,...,4), the specific expression is as follows:

[0133]

[0134]

[0135]

[0136]

[0137] in, represents the features obtained by division, It represents the intermediate result after the partition feature is expanded. represents a 3×3 dilated convolution with a dilation coefficient of rj (rj=j, i.e., r1=1, r2=2, r3=3, r4=4), div represents a division operation, and FTDM represents a clue feature perception module. When j=1, the formula Contains only and Two terms; when j=4, the formula Contains only and Two terms. When k=5, the formula Contains only and Two items.

[0138] like Figure 6 As shown, the clue feature perception module uses the downsampled boundary clue information and foreground target features As input, and Multiply the elements to obtain their common feature information and add the multiplication result to Add, retain the original feature information, and then apply 3×3 convolution to obtain the initial fusion feature , introduce the attention mechanism to perform weighted summation on the initial fusion features and aggregate the convolution features , the corresponding channel attention weight is calculated through the fully connected layer and the Sigmoid function , and and Multiply element by element to enhance important features, and finally compress the number of channels through 1×1 convolution to generate preliminary fusion features , the specific expression method is as follows:

[0139]

[0140]

[0141]

[0142] Among them, FC represents the fully connected layer and ATT represents the attention mechanism.

[0143] Step 2.4.2: Concatenate the 4 adjacent features after fusion , and the features are obtained through 1×1 convolution , and then through residual connection and 3×3 convolution operation, the deep fusion feature is finally obtained , the specific expression method is as follows:

[0144]

[0145]

[0146] Where Conv3 represents a 3×3 convolution operation.

[0147] The deep feature fusion module adopts a cross-layer information transfer structure in the process of fusing hierarchical features, fuses multi-level features in a cascade manner, and uses 1×1 convolution to compress the number of different feature channels, so that the output features of the lower-level deep feature fusion module can be directly connected to the upper layer, sharing and fusing feature representations at different levels.

[0148] In addition, only the boundary clue information and foreground target features As the input of the bottom-level deep feature fusion module, the output result The top-level deep feature fusion module uses boundary clue information , foreground target features And the output of the previous level deep feature fusion module is the input and output features Directly as output and participate in deep supervision.

[0149] Step 3: Use the training data obtained in step 2 to train the boundary cue deep fusion network, such as Figure 7 As shown in Figure 1, the weighted joint scale difference loss and weighted binary cross entropy loss are used to supervise the model training process, which specifically includes the following steps:

[0150] Step 3.1: Define the predicted area as P and the true area as G. Take the union of the predicted area P and the true area G, subtract the intersection area of ​​the predicted area P and the true area G reduced to β (β≤1 and β≥0) times, and get the result , the specific expression method is as follows:

[0151]

[0152]

[0153] Step 3.2: Substitute the S obtained in step 3.1 b The result is obtained by subtracting the intersection area of ​​the predicted area P reduced to ɑ times and the real area G , the specific expression method is as follows:

[0154]

[0155] Step 3.3: Substitute the result from step 3.1 As the numerator, the result obtained in step 3.2 As the denominator, we get the joint scale difference loss L SDou , the specific expression method is as follows:

[0156]

[0157] Where β is the ratio of the intersection of the predicted area P and the true area G to the true area G. The specific expression is as follows:

[0158]

[0159] Among them, ɑ is a hyperparameter, which is calculated adaptively according to the ratio, ɑ<1 and ɑ>0, and ɑ≤β.

[0160] Step 3.4: Use the features output by the deep feature fusion module during training And the output feature F of the boundary clue mining module e As a result, we calculate the joint scale difference loss , and finally the joint scale difference loss of each part Add together to get the total joint scale difference loss L sdt , the specific expression method is as follows:

[0161]

[0162] in, is the joint scale difference loss for each feature.

[0163] Step 3.5: Calculate the binary cross entropy loss for each binary image , and then the binary image P i The total joint scale difference loss L sdt and the total binary cross entropy loss L bcet Weighted global constraints and local constraints are obtained to obtain the weighted joint scale difference loss and weighted binary cross entropy loss , and then get the final loss L total , the specific expression method is as follows:

[0164]

[0165]

[0166] in, is the binary cross entropy loss for each binary image, and denote the weighted joint scale difference loss and binary cross entropy loss based on global and local constraints, respectively.

[0167] Step 4: Use the best model obtained after training to test the colonoscopy images in the test set, obtain the polyp segmentation results after the test, and evaluate the segmentation results of the model.

[0168] Step 5: Input the polyp image to be segmented into the trained polyp image segmentation model and output the segmentation mask.

[0169] To illustrate the effectiveness of this method, some polyp images were selected from the classic polyp segmentation datasets CVC-300, Kvasir, ColonDB, ETIS and ClinicDB to train the complete network, and then tested using five datasets of CVC-300, Kvasir, ColonDB, ETIS and ClinicDB, and the test results were compared with the mainstream polyp segmentation algorithms in the prior art. The proposed model framework is implemented in PyTorch and trained using an NVIDIA GeForce RTX 2080 SUPER GPU with 8GB of memory. The input of all images is uniformly set to 352×352, and the Adam optimization algorithm is used to optimize the overall parameters, and the learning rate is set to 0.25e-4. The test results on the CVC-ClinicDB and Kvasir datasets are shown in Table 1:

[0170] Table 1 Polyp segmentation test results of the present invention and other 8 methods

[0171]

[0172] The experimental results on the CVC-300 and CVC-ColonDB datasets are shown in Table 2:

[0173] Table 2 Polyp segmentation test results of the present invention and other 8 methods

[0174]

[0175] The experimental results on the ETIS dataset are shown in Table 3:

[0176] Table 3 Test results of polyp segmentation of the present invention and other 8 methods

[0177]

[0178] The bold font indicates the best value in each row, and the underline indicates the second best result.

[0179] Among the 7 evaluation indicators, the first two are commonly used evaluation indicators in semantic segmentation tasks, and the closer the values ​​of the first 6 indicators are to 1, the better the segmentation results are.

[0180] like Figure 8As shown, it can be concluded from the results that the polyp segmentation method proposed in the present invention can obtain a segmentation result with more accurate boundaries and more complete semantic structure.

[0181] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A polyp image segmentation method based on deep fusion of boundary cues, characterized in that The steps include: Step 1: Acquire polyp images; Step 2: Construct a polyp image segmentation model based on deep fusion of boundary cues, including the following steps: Step 2.1: Pass the polyp image through the backbone network to extract hierarchical features; Step 2.2: Extract high-fidelity boundary information from hierarchical features of different scales, and construct a polyp segmentation mask through multi-level fusion to obtain polyp boundary detail information; Step 2.3: Perform cross-level local union on hierarchical features of different scales to extract cross-level features for foreground target enhancement corresponding to the hierarchical features. Use the inverted attention mechanism of foreground target enhancement to obtain high-level semantic features. The input of the top-level foreground target enhancement is the second-level features and the features output by the previous-level foreground target enhancement module. The features output by the second-level foreground target enhancement are only input into the corresponding top-level deep feature fusion. Foreground target enhancement includes the following steps: Step 2.3.1: Through the cascaded foreground object enhancement method, each level of foreground object enhancement takes the output features of the previous level of foreground object enhancement and the corresponding level features as input; Step 2.3.2: In the inverted attention branch of foreground target enhancement, invert the output of the previous level foreground target enhancement to obtain the inverted attention weight; Through the cross-level local joint decoder, the common information between different features is extracted by multiplying multiple feature elements; Step 2.3.3: Perform element-wise subtraction on the inverted attention weight and its corresponding level feature to extract the difference information between the features, and then perform element-wise multiplication on the corresponding level feature to obtain the complementary enhanced foreground target feature as the high-level semantic information; For the top-down 5-layer network, the bottom-level feature F5 is first refined by upsampling and convolution, and element-wise multiplication is performed with the level feature F4 to extract the enhanced common feature F4'. The common enhanced feature F3' of the level features F3, F4 and F5 is extracted in the same way to further capture the interdependence of cross-level features. After that, the level feature F5 is upsampled and convolved to obtain the feature map and concatenated with F4' to fuse the feature information of different resolutions. After convolution, the deeper level feature F4 is obtained. g , and then F4 g After upsampling and convolution, it is concatenated with F3', combining details and high-level semantic information, and refining features F3 through convolution. g , and finally F3 g After convolution and convolution dimension reduction, obtain the significant cross-level feature F t 6 ; The inverted attention weight is element-wise subtracted from the corresponding level feature to extract the difference information between the features, and then element-wise multiplied with the corresponding level feature to obtain the complementary enhanced foreground target feature; Step 2.4: Deeply integrate the high-level semantic features corresponding to each foreground target enhancement with the boundary detail information, adopt a cross-layer information transmission structure, fuse multi-level features in a cascade manner, and obtain deep fusion features for polyp image segmentation; including the following steps: Step 2.4.1: According to the requirements of different levels, the boundary clue information is downsampled to different degrees to obtain multi-scale boundary clue information, and the multi-scale boundary clue information and foreground target features are used as the input of clue feature perception to generate a multi-level feature representation. The output of the previous order deep feature fusion is spliced ​​with the current multi-level feature representation to obtain a multi-scale integrated feature. The multi-layer features are interlaced and fused to obtain multiple fused adjacent features. Clue feature perception, taking the downsampled boundary clue information and foreground target features as input, multiplying the elements to obtain their common feature information, adding the multiplication result to the foreground target features, and then using convolution to obtain the initial fusion features. The attention mechanism is introduced to perform weighted summation on the initial fusion features, aggregate the convolution features, calculate the corresponding channel attention weights through the fully connected layer and activation function, and multiply the aggregated convolution features and the channel attention weights element by element. Finally, the number of channels is compressed through convolution to generate preliminary fusion features; Step 2.4.2: Concatenate the fused adjacent features, obtain features through convolution, and then obtain deep fusion features through residual connection and convolution operations; Step 3: Train the polyp image segmentation model to obtain a trained model for polyp image segmentation.

2. The polyp image segmentation method based on deep fusion of boundary cues according to claim 1, characterized in that: In step 2.2, boundary clue mining is used to obtain hierarchical features of different scales, output the boundary area contour of the polyp, and perform loss calculation to optimize the model. Specifically, boundary clue mining includes the following steps: Step 2.2.1: Convert multiple top-level features into corresponding multiple feature gradient maps respectively; Step 2.2.2: Concatenate multiple feature gradient maps along the channel dimension; Step 2.2.3: Concatenate the underlying features with the concatenated features to obtain aggregate features containing boundary information, and extract boundary clues from the aggregate features as polyp boundary detail information.

3. The polyp image segmentation method based on boundary cue deep fusion according to claim 1, characterized in that: In the step 2.4, only the second-bottom boundary clue information and the bottom-level foreground target features are used as the input of the bottom-level deep feature fusion, and the bottom-level deep fusion features are output; The top-level deep feature fusion module takes the top-level boundary clue information, the second-level foreground target features and the output of the previous-level deep feature fusion module as input. The output deep fusion features are directly used as output results and participate in deep supervision.

4. The polyp image segmentation method based on boundary cue deep fusion according to claim 1, characterized in that: In step 3, the polyp image segmentation model is trained using a weighted joint scale difference loss and a weighted binary cross entropy loss supervision model, including the following steps: Step 3.1: Input the predicted area and the real area, take the union area of ​​the two minus the reduced intersection area of ​​the two, and obtain the first result, where the reduction factor is the ratio of the intersection of the predicted area and the real area to the real area; Step 3.2: replacing the reduction factor in step 3.1 with a second reduction factor to obtain a second result, wherein the second reduction factor is less than or equal to the reduction factor; Step 3.3: Construct the joint scale difference loss by the ratio of the first result to the second result; Step 3.4: During the training process, the deep fusion features and boundary detail information are used as results to calculate the joint scale difference loss respectively, and finally the joint scale difference loss of each part is added to obtain the total joint scale difference loss; Step 3.5: Calculate the binary cross entropy loss for each binary image, and then weight the total joint scale difference loss and the total binary cross entropy loss of the binary image with global constraints and local constraints to obtain the weighted joint scale difference loss and the weighted binary cross entropy loss, and then get the final loss.

5. The polyp image segmentation method based on boundary cue deep fusion according to claim 1, characterized in that: In step 1, the collected polyp image data set is divided into a training set and a test set; in step 3, the polyp image segmentation model is trained using the training set. The method further includes: Step 4: Use the best polyp image segmentation model obtained after training to test the polyp images in the test set, obtain the polyp segmentation results after the test, and perform evaluation to obtain the polyp image segmentation model that has completed the training; Step 5: Input the polyp image to be segmented into the trained polyp image segmentation model and output the polyp segmentation result.

Images