A polyp image segmentation method and system based on deep guidance and triple decoder

Abstract

This invention provides a polyp image segmentation method based on depth guidance and a triple decoder. Using RGB images as input, the method expands the channels of the depth image, extracts its multi-layer features, and performs layer-by-layer feature fusion. This effectively utilizes the spatial structural characteristics of polyps in the depth image, enhances multi-scale semantic and spatial location information, and helps the network extract the polyp region more completely. A triple decoder architecture is designed to extract fine-grained detail features and deep semantic features of the polyps. The combined decoder integrates the advantages of both types of features, improving multi-scale feature extraction and fusion capabilities. A high- and low-resolution dual-branch feature extraction strategy is adopted to adapt to high-resolution input. Combined with the advantages of a pre-trained backbone network, this method reduces model computational complexity while ensuring no loss of polyp detail information, thus balancing segmentation performance and operational efficiency.

Description

Technical Field

[0001] This invention relates to the fields of medical image analysis and computer vision technology, specifically to a method and system for polyp image segmentation based on depth guidance and a triple decoder. Background Technology

[0002] Colorectal cancer is a highly prevalent malignant tumor of the digestive system worldwide, with its incidence and mortality rates showing an increasing trend year by year. Colonoscopy, as the gold standard for early screening of colorectal cancer, can detect and remove polyps at an early stage of the disease, effectively reducing the risk of cancer development. Polyp segmentation, as a key task in computer-aided diagnosis, can reduce the workload of doctors and improve the objectivity and consistency of detection.

[0003] With the development of deep learning technology, segmentation methods based on convolutional neural networks (CNNs) have become the core technology for polyp segmentation. Models such as U-Net, FCN and its variants have made some progress in low-resolution image segmentation. However, low-resolution images are prone to loss of details, especially for large polyps with blurred boundaries and irregular shapes. The lack of key information seriously affects the segmentation accuracy. Although high-resolution images can improve diagnostic accuracy, they bring the problem of excessive computational overhead. Existing methods are difficult to balance segmentation accuracy and inference efficiency.

[0004] Existing methods often rely on a single decoder structure or only perform feature enhancement at the encoder end, making it difficult to finely recover target details and boundaries during the decoding process. Large polyps have problems such as complex internal structures, blurred edge transitions, and adhesion to surrounding tissues. Existing methods have obvious shortcomings in capturing complete contours and distinguishing heterogeneous regions. Furthermore, the lack of effective utilization of depth information results in insufficient ability to perceive the three-dimensional structure and morphology of large polyps.

[0005] Therefore, there is an urgent need for a polyp segmentation method that can balance segmentation accuracy, boundary delineation ability, and inference efficiency, and is applicable to images of various resolutions. Summary of the Invention

[0006] Purpose of the invention: To address the problems existing in current polyp image segmentation technologies, this invention proposes a high-resolution polyp image segmentation method and system based on depth guidance and a triple decoder, which is used for automatic segmentation of polyp images during colonoscopy, providing auxiliary diagnostic support for early disease screening.

[0007] Technical solution:

[0008] This invention proposes a polyp image segmentation method based on deep guidance and a triple decoder, comprising:

[0009] A polyp image segmentation model is established, comprising a two-branch asymmetric encoder and a triple decoder. The input image of the polyp image segmentation model is a colonoscopy polyp medical image. The two-branch asymmetric encoder includes an RGB feature extraction branch and a Depth feature extraction branch, which are used to extract image features of the input image through a pre-trained backbone network. The triple decoder includes a structure decoder, a backbone decoder, and a joint decoder, which are used to process the image features and output prediction results.

[0010] Design a composite loss function, select an existing dataset including real annotations to train the polyp image segmentation model until the polyp image segmentation model converges;

[0011] Input the polyp image to be segmented into the trained polyp image segmentation model, and output a polyp segmentation prediction map.

[0012] Furthermore, the RGB feature extraction branch includes a high-resolution sub-branch and a low-resolution sub-branch. The input of the high-resolution sub-branch is the original input image of the polyp image segmentation model, and the input of the low-resolution sub-branch is the image of the original input image after bilinear interpolation downsampling, with a downsampling ratio of 4:1.

[0013] Furthermore, the input to the Depth feature extraction branch is as follows: the input to the low-resolution sub-branch is processed using a computer vision depth estimation algorithm to obtain a corresponding single-channel depth map; the single-channel depth map is expanded to 3 channels to obtain the input to the Depth feature extraction branch; the input to the Depth feature extraction branch has the same resolution as the input to the low-resolution sub-branch.

[0014] Furthermore, the pre-trained backbone network is ResNet50;

[0015] The high-resolution sub-branch outputs five different levels of high-resolution feature sets. , , , , The low-resolution sub-branch outputs five different levels of low-resolution feature sets. , , , , The Depth feature extraction branch outputs five different levels of Depth feature sets. , , , , The subscript numbers 1 to 5 correspond to the five feature levels of ResNet50: 1 and 2 are low-level, 3 is mid-level, and 4 and 5 are high-level.

[0016] By employing a cross-modal hierarchical fusion mechanism, the low-resolution feature set and the depth feature set at the same level are fused to obtain five fused feature sets at different levels. , , , , .

[0017] Furthermore, the input to the structure decoder includes the high-resolution feature set obtained from the high-resolution sub-branch. , , , , and the original feature set The original feature set is obtained by aligning the input image of the polyp image segmentation model through a 1×1 convolution channel.

[0018] Furthermore, the workflow of the structure decoder includes:

[0019] Perform convolutions with 64 kernels on all input features respectively. operate;

[0020] For high-resolution features Perform bilinear interpolation upsampling, according to magnified many times ;

[0021] The structural features are obtained by performing pixel-by-pixel addition and fusion in a layer-by-layer recursive manner. As the output of the structure decoder, the layer-by-layer recursive initialization formula is: The recurrence relation is .

[0022] Furthermore, the input to the backbone decoder is high-level fused low-resolution features. , and high-resolution features in the middle and upper levels , , ;

[0023] The workflow of the backbone decoder includes: performing convolutional operations with 64 kernels on all input features respectively. Operation; matching is performed separately through upsampling processing. and , and , and The size of the feature set is used to perform feature fusion for each set of features, represented as:

[0024]

[0025]

[0026]

[0027] Obtain the main features , , This is the output of the trunk decoder.

[0028] Furthermore, the joint decoder will output features from the structure decoder. , , , , , and backbone decoder output characteristics , , Perform cross-scale fusion to obtain cross-scale fusion features. , , , , , is represented as:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] The cross-scale fused features are processed through a 3×3 convolutional layer to reduce the number of channels and map the features, resulting in a single-channel segmentation prediction map P, denoted as:

[0035]

[0036] In the formula, SE represents the channel attention module.

[0037] Furthermore, the composite loss function is expressed as:

[0038]

[0039] in, Indicates backbone feature loss, Represents structural feature loss, This represents the loss due to mask segmentation.

[0040] On the other hand, the present invention also proposes a polyp image segmentation system based on deep guidance and triple decoder, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the aforementioned methods.

[0041] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0042] This invention provides a high-resolution polyp image segmentation method based on depth guidance and a triple decoder. It uses a cross-modal hierarchical fusion module. Based on RGB images as input, it extracts multi-layer features from the depth image after channel expansion and performs layer-by-layer feature fusion. This effectively utilizes the spatial structural characteristics of polyps in the depth image, strengthens multi-scale semantic information and spatial location information, and helps the network to extract the region where the polyp is located more completely.

[0043] This invention designs a triple decoder architecture, inputting high-resolution RGB features into the structural decoder and low-resolution features fused with depth information into the backbone decoder to extract fine-grained detail features and deep semantic features of polyps. Then, the joint decoder integrates the advantages of the two types of features to improve the multi-scale feature extraction and fusion capabilities.

[0044] This invention also employs a high- and low-resolution dual-branch feature extraction strategy to adapt to high-resolution input, and combines the advantages of pre-trained backbone networks to reduce the computational complexity of the model while ensuring that polyp details are not lost, thus balancing segmentation performance and running efficiency. Attached Figure Description

[0045] Figure 1 This is a flowchart of the data processing method of the present invention;

[0046] Figure 2 This is a structural diagram of a dual-branch asymmetric encoder in a polyp image segmentation model.

[0047] Figure 3 This is a structural diagram of the triple decoder in the polyp image segmentation model;

[0048] Figure 4 This is an example diagram showing the output results of the method of the present invention. Detailed Implementation

[0049] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0050] This invention proposes a high-resolution polyp image segmentation method based on depth guidance and a triple decoder. The data processing flow is as follows: Figure 1 As shown, the specific steps include:

[0051] S1: Establish a polyp image segmentation model, including a two-branch asymmetric encoder, a triple decoder, and a loss function module. The polyp image segmentation model of this invention is an end-to-end deep learning model, specifically:

[0052] (1) A dual-branch asymmetric encoder, comprising an RGB feature extraction branch (including high / low resolution sub-branches) for extracting visual features and a Depth feature extraction branch for extracting spatial structure features, and extracting multi-level features based on a pre-trained backbone network ResNet50. The structure of the dual-branch asymmetric encoder is as follows: Figure 2 As shown.

[0053] The input to the RGB feature extraction branch is a high-resolution RGB image, specifically 1024×1024 in this embodiment, denoted as . The The image is fed into the high-resolution sub-branch of the RGB feature extraction branch. Bilinear interpolation downsampling is performed on the high-resolution RGB image to obtain a low-resolution RGB image of 256×256 resolution, denoted as [image name missing]. The The input to the low-resolution sub-branch of the RGB feature extraction branch is the Depth image.

[0054] The high-resolution RGB images used in this embodiment are from the SUN-SEG colonoscopy medical image dataset, while the depth images are from RGB images of publicly available polyp datasets such as SUN-SEG. The RGB images are processed using a computer vision depth estimation algorithm to generate corresponding single-channel depth maps, which are then expanded to 3 channels to obtain the depth images.

[0055] The backbone network of the feature extraction branch uses a pre-trained ResNet50, and the high-resolution sub-branch of the RGB feature extraction branch obtains a high-resolution RGB feature set of 5 key levels through the backbone network. , , , , The low-resolution sub-branch of the RGB feature extraction branch is then passed through the backbone network to obtain low-resolution RGB feature sets at five key levels. , , , , The Depth feature extraction branch obtains five key Depth feature sets through the backbone network. , , , , Five key layers are extracted here, corresponding to the five feature extraction stages of ResNet50. Therefore, each branch outputs five feature sets. The subscripts of the feature sets are HR for high-resolution RGB images, LR for low-resolution RGB images, and DR for depth images. The numbers 1-5 correspond to the five feature layers of ResNet50.

[0056] The visual features and spatial structural features are fused using a cross-modal hierarchical fusion mechanism to generate a fused feature set. This mechanism only fuses low-resolution RGB features and Depth features at the same feature level (same subscript number) in ResNet50; features from different levels do not participate in cross-level fusion.

[0057] Specifically, this means: transferring low-resolution features and Depth features The fusion process is performed to obtain fused low-resolution features. The fusion process is represented as: Repeat the above fusion process sequentially to obtain , , , The core benchmark for feature fusion is the alignment of the spatial and semantic dimensions of features at the same level in ResNet50. The rationale for fusion at the same level lies in the alignment of both spatial size and semantic abstraction, ensuring the effectiveness of the fusion operation.

[0058] (2) The triple decoder structure includes a structural decoder, a backbone decoder and a joint decoder, which respectively perform refined boundary extraction, main region segmentation and multi-scale information fusion on the fused features.

[0059] like Figure 3 The diagram shows the triple decoder structure of this invention. High-resolution features obtained from the RGB feature extraction branch and the original features are input into the structural decoder. The fused low-resolution features from the top two layers and the high-resolution features from the top three layers are input into the backbone decoder. Finally, the outputs of the two decoders are input together into the joint decoder to obtain the prediction mask. Specifically, it includes the following sub-steps:

[0060] (2-1) The input to the structure decoder is a set of high-resolution RGB features. , , , , and the original feature set After multi-layer fusion, structural features are output sequentially. , , , , , .

[0061] Among them, the original feature set The raw input features, obtained directly from a 1024×1024 high-resolution RGB image after 1×1 convolutional channel alignment, are not processed by the ResNet50 backbone network. Their spatial dimensions are identical to the original input image (1024×1024), providing the structural decoder with the most original pixel-level boundaries and texture details. The multi-layer fusion includes multi-layer 1×1 convolutional channel alignment, upsampling size matching, and cross-layer feature fusion. Multi-layer fusion adheres to the principle of feature alignment at the same size, and the core computational process includes:

[0062] First, analyze all input features separately. , , , , , Perform convolution with 64 kernels The operation involves unifying all input feature channels to 64; then... Perform bilinear interpolation upsampling, according to The image is magnified multiple times to make the resolution of the highest-level structural features match the resolution of the original input image; in this embodiment, it is 1024×1024, resulting in size-matched features. Finally, pixel-by-pixel addition and fusion are performed in a layer-by-layer recursive manner to obtain structural features. The initialization formula is obtained by recursion layer by layer. The recurrence relation is .

[0063] After the above multi-layer fusion, the final generated structural features , , , , , Resolution increases progressively. The resolution is consistent with the original resolution of the input image (1024×1024), and the boundary contours and surface texture information of the polyps are preserved. This technical effect is achieved through the collaborative efforts of four innovative designs: hierarchical input of the structure decoder, step-by-step recursive fusion, fixed parameter settings, and progressively increasing resolution. This provides precise fine-grained detail support for multi-scale information fusion of the joint decoder.

[0064] (2-2) The input of the backbone decoder includes high-level fused low-resolution features. , High-resolution features in the middle and upper levels , , .

[0065] This invention selects , , , , The input to the backbone decoder is a customized design based on the polyp segmentation task: low-resolution fused features. , Provides global contour and spatial structure constraints for polyps as high-level semantic features; high-resolution features , , For mid-to-high-level semantic features, local boundaries and texture details of polyps are supplemented, and the two are fused across levels to achieve precise complementarity of "global + local". At the same time, this selection excludes low-level large-size features, which greatly reduces the amount of computation while ensuring segmentation accuracy, achieving the optimal balance between accuracy and efficiency, and is fully coordinated with the overall architecture of the triple decoder of this invention.

[0066] It is worth noting that the model architecture of this invention is a resolution-independent general architecture. 1024×1024 is only the standard input resolution used in the experiment and is not a hard constraint on the model. It fully supports colonoscopy image input at any resolution. When the original input is not 1024×1024 resolution, only the 4:1 downsampling ratio of high / low resolution images needs to be kept unchanged. The input selection logic of the backbone decoder remains completely unchanged, always selecting the high-resolution branch. , , With low-resolution fusion branch , Only the absolute size of the features scales proportionally with the input resolution. The model architecture, fusion logic, and training process do not need to be modified. It can be directly adapted to the output resolution of different types of endoscopes in clinical practice and has strong clinical applicability.

[0067] The input mixed features are processed by 1×1 convolutional layers with a unified number of channels of 64. Feature fusion is achieved through multi-layer cascaded upsampling-addition-convolution operations; the final output backbone features are then processed. , , These correspond to semantic information at different scales, denoted as multi-scale semantic features, and provide high-level semantic guidance for the joint decoder. The computational representation of the above process is as follows:

[0068]

[0069]

[0070]

[0071]

[0072] yes Matched by 8 times upsampling After sizing, the intermediate features obtained by Conv1×1 fusion have both Polyp spatial structure information and The boundary details provide core support for subsequent multi-scale feature fusion; Similarly, match respectively After the dimensions are adjusted, the data is fused to form a hierarchical multi-scale semantic feature system.

[0073] (2-3) The joint decoder will combine the output features of the structure decoder. and backbone decoder output characteristics , , Cross-scale fusion is performed to obtain cross-scale fusion features F. The cross-scale fusion features F are then passed through a 3×3 convolutional layer to reduce the number of channels and perform feature mapping, resulting in a single-channel final segmentation feature map P. This feature map retains both the deep semantic information of the polyp and fine-grained structural information such as edges and textures, providing core feature support for the subsequent generation of segmentation results.

[0074] Specifically, the working process of the joint decoder includes the following sub-steps:

[0075] (2-3-1) Perform bilinear interpolation upsampling on input features of different scales to unify the resolution of all features to the target segmentation scale;

[0076] (2-3-2) The scale-matched backbone decoder output features (semantic features) T and the structure decoder output features (structural features) S are concatenated along the channel dimension. A fusion module consisting of 3×3 convolutional layers, batch normalization layers, and ReLU activation functions is used to achieve deep fusion of semantic and detailed structural information. The fusion process is represented as follows:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] (2-3-3) Finally, the predicted image P is obtained after multiple convolutions (Conv). The calculation process is expressed as follows:

[0083]

[0084] SE stands for Channel Attention Module.

[0085] S2: Design a loss function, select a dataset, train the polyp segmentation model, calculate the error between the prediction results and the true labels, and update the model parameters in reverse.

[0086] A dataset was selected to train the polyp image segmentation model. The error between the predicted result and the ground truth label was calculated using the loss function module, and the model parameters were updated accordingly. In this embodiment, the model training used the publicly available SUN-SEG polyp segmentation dataset, which contains 45,450 colonoscopy polyp images with a resolution of 1024×1024 and pixel-level ground truth label masks. The dataset was divided into training, validation, and test sets in a 7:2:1 ratio. Before training, data augmentation operations such as normalization and random flipping / rotation were performed on the images.

[0087] The error is calculated using a composite loss function, which is then used to update all trainable parameters of the encoder / decoder in reverse. The loss function is a composite loss function, expressed as:

[0088]

[0089] in, Indicates backbone feature loss, Represents structural feature loss, The loss represents the mask segmentation loss. The calculation method of each loss is based on the research of Salehi SS (arXiv:1706.05721, 2017) and Wang Z (DOI:10.1109 / TIP.2003.819861, 2004).

[0090] S3: Input the polyp image to be segmented into the trained model and output the polyp segmentation prediction map.

[0091] like Figure 4 The image shown is an example of the prediction results obtained in this embodiment. Rows A1 and A2 represent the prediction results of large polyps, and rows B1 and B2 represent the prediction results of small polyps. The first column is the original polyp image to be segmented, the second column is the actual labeled image, and the third column is the polyp segmentation prediction image output by the method of this invention. Figure 4 The abstract model performance is transformed into intuitive visual effects, and the advantages of the model in accurately segmenting polyps are verified from both numerical and visual dimensions.

[0092] Table 1 lists the comparative data of image segmentation methods proposed in recent years. The upward arrows indicate that the larger the performance index, the better, such as maxFm, wFmeasure, S-measure, meanEm, and meanFm; the downward arrows indicate that the smaller the performance index, the better, such as MAE.

[0093] Table 1. Comparison of the segmentation model of this invention with existing technologies

[0094]

[0095] As shown in Table 1, the best performance was almost all achieved by the network model designed in this invention, which verifies that the invention can accurately locate polyps in similar backgrounds. Moreover, the high-resolution polyp image segmentation method based on depth guidance and triple decoder can use the depth guidance region map for constraint refinement to obtain a more accurate polyp image segmentation map, which effectively improves the accuracy of polyp image segmentation.

Claims

1. A polyp image segmentation method based on depth-guided and triple decoder, characterized in that, include: A polyp image segmentation model is established, comprising a two-branch asymmetric encoder and a triple decoder. The input image of the polyp image segmentation model is a colonoscopy polyp medical image. The two-branch asymmetric encoder includes an RGB feature extraction branch and a Depth feature extraction branch, which are used to extract image features of the input image through a pre-trained backbone network. The triple decoder includes a structure decoder, a backbone decoder, and a joint decoder, which are used to process the image features and output prediction results. Design a composite loss function, select an existing dataset including real annotations to train the polyp image segmentation model until the polyp image segmentation model converges; Input the polyp image to be segmented into the trained polyp image segmentation model, and output a polyp segmentation prediction map.

2. The polyp image segmentation method according to claim 1, characterized in that, The RGB feature extraction branch includes a high-resolution sub-branch and a low-resolution sub-branch. The input of the high-resolution sub-branch is the original input image of the polyp image segmentation model, and the input of the low-resolution sub-branch is the image of the original input image after bilinear interpolation downsampling, with a downsampling ratio of 4:

1.

3. The polyp image segmentation method according to claim 2, characterized in that, The input to the Depth feature extraction branch is as follows: the input to the low-resolution sub-branch is processed using a computer vision depth estimation algorithm to obtain a corresponding single-channel depth map; the single-channel depth map is expanded to 3 channels to obtain the input to the Depth feature extraction branch; the input to the Depth feature extraction branch has the same resolution as the input to the low-resolution sub-branch.

4. The polyp image segmentation method according to claim 3, characterized in that, The pre-trained backbone network is ResNet50; The high-resolution sub-branch outputs five different levels of high-resolution feature sets. , , , , The low-resolution sub-branch outputs five different levels of low-resolution feature sets. , , , , The Depth feature extraction branch outputs five different levels of Depth feature sets. , , , , The subscript numbers 1 to 5 correspond to the five feature levels of ResNet50: 1 and 2 are low-level, 3 is mid-level, and 4 and 5 are high-level. The low-resolution feature set of the same level is fused with the Depth feature set through a cross-modal hierarchical fusion mechanism to obtain five fused feature sets of different levels , , , , .

5. The polyp image segmentation method according to claim 4, characterized in that, The input to the structure decoder includes the high-resolution feature set obtained from the high-resolution sub-branch. , , , , and the original feature set The original feature set is obtained by aligning the input image of the polyp image segmentation model through a 1×1 convolution channel.

6. The polyp image segmentation method according to claim 5, characterized in that, The workflow of the structure decoder includes: Perform convolutions with 64 kernels on all input features respectively. operate; For high-resolution features Perform bilinear interpolation upsampling, according to magnified many times ; The structural features are obtained by performing pixel-by-pixel addition and fusion in a layer-by-layer recursive manner. As the output of the structure decoder, the layer-by-layer recursive initialization formula is: The recurrence relation is .

7. The polyp image segmentation method according to claim 6, characterized in that, The input to the backbone decoder is high-level fused low-resolution features. , and high-resolution features in the middle and upper levels , , ; The workflow of the backbone decoder includes: performing convolutional operations with 64 kernels on all input features respectively. Operation; matching is performed separately through upsampling processing. and , and , and The size of the feature set is used to perform feature fusion for each set of features, represented as: Obtain the main features , , This is the output of the trunk decoder.

8. The polyp image segmentation method according to claim 7, characterized in that, The joint decoder will output features from the structure decoder. , , , , , and backbone decoder output characteristics , , Perform cross-scale fusion to obtain cross-scale fusion features. , , , , , is represented as: The cross-scale fused features are processed through a 3×3 convolutional layer to reduce the number of channels and map the features, resulting in a single-channel segmentation prediction map P, denoted as: In the formula, SE represents the channel attention module.

9. The polyp image segmentation method according to claim 8, characterized in that, The composite loss function is expressed as follows: in, Indicates backbone feature loss, Represents structural feature loss, Indicates the mask segmentation loss. This represents the composite loss.

10. A polyp image segmentation system based on deep guidance and a triple decoder, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of any of the methods of claims 1 to 9.

Images