A method for segmenting colorectal polyps based on a three-branch cooperative gating network

Abstract

A colorectal polyp segmentation method based on a three-branch collaborative gating network addresses the challenges of large scale variations, irregular morphology, blurred boundaries, and severe background interference in colonoscopy images. This invention achieves accurate perception and semantic representation of polyp regions at different scales by performing multi-scale parallel modeling on the input image, combining heterogeneous receptive field feature aggregation, spatial attention-guided cross-scale feature fusion, and channel-gated interaction mechanisms. The overall network is based on a shared backbone feature extraction network, introducing weight adjustment and gating mechanisms during multi-scale feature enhancement and fusion. This ensures global contextual consistency while strengthening local details and boundary information, ultimately outputting high-precision pixel-level polyp segmentation results through a decoder.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and computer-aided diagnosis technology, specifically relating to a deep learning method for automatic segmentation of colorectal polyps in colonoscopy images, and more particularly to a precise segmentation method for colorectal polyps based on a three-branch collaborative gating network. Background Technology

[0002] Colorectal cancer is one of the most common and deadliest malignant tumors worldwide, and its development typically follows a sequence of "normal mucosa - adenomatous polyp - cancer". Colonoscopy, as the "gold standard" for screening and diagnosing colorectal lesions, can directly visualize subtle changes in the intestinal mucosa and is a key means of detecting and removing precancerous lesions (i.e., polyps). However, the quality of clinical diagnosis is highly dependent on the experience and focus of the endoscopist. Due to the significant differences in polyps in size (from tiny polyps <5mm to large masses >2cm), shape (lobulated, flat, pedunculated or sessile), and appearance (surface texture, color, boundary contrast), and the complex intestinal environment (with interference from folds, bubbles, fluid residue, specular reflection, etc.), manual interpretation results in a false negative rate of approximately 22%, accompanied by high inter-observer variability, affecting the detection efficiency and consistency of early cancerous changes.

[0003] To improve the objectivity and efficiency of diagnosis, computer-aided polyp segmentation systems based on deep learning have emerged. These systems aim to automatically and accurately locate and delineate polyp regions, providing real-time visual assistance to physicians. In recent years, related technologies have mainly revolved around the following architectures: First, methods based on Convolutional Neural Networks (CNNs), represented by U-Net and its variants, fuse multi-scale features through encoder-decoder structures and skip connections, excelling at capturing local details. However, the inherent local receptive field of CNNs limits their ability to model long-distance spatial dependencies, limiting their effectiveness when processing large-scale polyps or complex morphologies requiring global contextual understanding. Second, methods based on Transformers, drawing inspiration from natural language processing, utilize self-attention mechanisms to capture global image contextual information, such as DBE-Net and Polyp-PVT. While these methods can establish long-range correlations between pixels, they typically have high computational complexity and are prone to losing high-resolution details crucial for polyp edge segmentation during feature downsampling. Finally, hybrid architectures combining CNNs and Transformers attempt to integrate their advantages, such as adding attention modules to the CNN backbone or designing dual-branch structures. However, existing hybrid methods often employ simplistic multi-scale feature fusion strategies (such as direct concatenation or addition), failing to achieve dynamic and adaptive interaction of features across scales. For small polyps, their weak signals are easily drowned out by background noise; for larger polyps, uneven internal texture can lead to incomplete segmentation (holes); and for polyps with blurred boundaries, it is difficult to generate continuous and accurate edges. Furthermore, existing methods often rely on unidirectional skip connections in their decoder design, lacking effective mechanisms for collaborative optimization and progressive refinement of high-level semantic information ("what it is") and low-level detailed features ("where the boundary is"), resulting in a difficulty in achieving both global consistency and local accuracy.

[0004] In summary, existing technologies face the following core bottlenecks when handling colorectal polyp segmentation, directly leading to corresponding segmentation defects: (a) Insufficient adaptability to scale changes: Existing methods mostly employ static or fixed multi-scale processing mechanisms, making it difficult to adaptively capture extreme scale changes from small polyps (<5mm) to large polyps (>2cm). This results in small polyps being easily missed, while the segmentation of large-scale polyps is often incomplete, with internal holes or broken contours; (b) Insufficiently refined feature fusion mechanisms: Current models often use simple splicing or addition operations when fusing features from different levels or branches, lacking dynamic calibration of spatial location and cross-scale semantic relationships. This leads to blurred and inaccurate boundary positioning, especially when the contrast between the polyp and the mucosal background is low and the edges are blurred; (c) Limited optimization of the decoding and recovery process: In the process of recovering detailed segmentation from high-level semantic features, most methods rely on unidirectional skip connections, lacking a collaborative optimization and progressive refinement mechanism for high-level semantic information and low-level detailed features. This results in the loss of local details and difficulty in achieving global consistency, affecting the accuracy and robustness of the segmentation results. Therefore, developing an intelligent segmentation network that can simulate the cognitive process of endoscopists—"global observation, local focusing, and dynamic calibration"—has significant clinical implications and application value. Summary of the Invention

[0005] The purpose of this invention is to overcome the aforementioned shortcomings of existing colorectal polyp segmentation methods and provide a colorectal polyp segmentation method based on a three-branch collaborative gating network. Inspired by clinical diagnostic paradigms, this method constructs a progressive architecture of three-branch input, dynamic multi-scale perception, spatial attention-guided fusion, and hierarchical gating decoding to achieve accurate and robust segmentation of polyps of different sizes, shapes, and boundary clarity, significantly improving segmentation accuracy and generalization ability.

[0006] To achieve the above objectives, the present invention provides the following technical solution: To address the challenges of large scale variations, irregular shapes, blurred boundaries, and severe background interference in colonoscopy images of polyps, this invention proposes a three-branch collaborative multi-scale adaptive gating segmentation method. This method performs multi-scale parallel modeling of the input image, combining heterogeneous receptive field feature aggregation, spatial attention-guided cross-scale feature fusion, and channel-gating interaction mechanisms to achieve accurate perception and semantic representation of polyp regions at different scales. The overall network is based on a shared backbone feature extraction network, introducing weight adjustment and gating mechanisms during multi-scale feature enhancement and fusion. This ensures global contextual consistency while strengthening local details and boundary information, ultimately outputting high-precision pixel-level polyp segmentation results through a decoder.

[0007] Step 1: Three-branch scaling and backbone network feature extraction The input colonoscopy images are processed in parallel into three views at different scales: the original resolution master-scale image, a global auxiliary-scale image downscaled by 0.5x, and a magnified image by 1.5x. These three scale images are then fed into a shared backbone feature extraction network, preferably using Pyramid Vision (PVTv2), to extract multi-level features, forming three complementary feature pyramids. Let the features output by each layer of the backbone feature extraction network be denoted as... , among which scale Feature abstraction layer subscript B Features extracted from the backbone network are used to identify features, where low-level features preserve detailed textures and high-level features encode semantic information.

[0008] Step 2: Heterogeneous receptive field feature aggregation module Next, regarding Feature enhancement is performed on each feature using a heterogeneous receptive field feature aggregation module to obtain features. , among which scale Feature abstraction layer subscript D Used to identify features enhanced by the heterogeneous receptive field feature aggregation module. For the first feature at scale s... l Layer features This module first performs preprocessing operations sequentially using 3×3 depthwise convolution, batch normalization, and ReLU activation to extract basic features while maintaining the feature map size. The preprocessed features are then divided into two paths. One path sequentially performs global average pooling and 1×1 convolution to compress the spatial dimension to capture global contextual information, generating a set of channel-dimensional dynamic weight vectors, which are then divided into four independent weight coefficients along the channel dimension. , , , After activation by a sigmoid function, another path enters four convolutional branches in parallel: a 1×1 standard convolution branch, a 3×3 standard convolution branch, a 3×3 dilated convolution branch with a dilation rate of 3 (equivalent to a 7×7 receptive field), and a 3×3 dilated convolution branch with a dilation rate of 6 (equivalent to a 13×13 receptive field). Then, the output features of each of the four branches are weighted and modulated channel-wise using the four generated weight coefficients. Finally, the modulated four features are concatenated and fused across channels using a 1×1 convolutional layer to output the enhanced features. This module enhances the ability to capture contextual information of small target edges or large lesions by combining multi-scale dilated convolution with dynamic channel attention mechanism to adaptively adjust the contribution of different receptive field branches.

[0009] Step 3: Spatial attention-guided multi-scale fusion module To fuse inputs from different scales Based on the feature information, a spatial attention-guided multi-scale fusion module is designed. The input of this module is a set of features from the same semantic level but located in different branches of the network, enhanced by a heterogeneous receptive field feature aggregation module. , Feature abstraction layer First, feature preprocessing and resolution alignment are performed on input features at different scales. Specifically, low-resolution features are aligned using 3×3 convolution and upsampling; high-resolution features are aligned to an intermediate scale by aligning the spatial dimensions after 3×3 convolution followed by max pooling and average pooling. After resolution alignment, max pooling and average pooling are performed on the features along the channel dimension, and the results are concatenated along the channel dimension. Then, a 7×7 convolutional layer is passed through, and a spatial attention map generated by sigmoid activation is generated and multiplied element-wise with the original features to suppress background region responses and enhance the feature representation of the target region. Subsequently, concatenation is performed along the channel dimension using a convolutional structure consisting of 1×1, 3×3, and 1×1 convolutions in sequence, and corresponding spatial weights are generated along the scale dimension using a softmax operation. Finally, the learned spatial weights are used to perform element-wise weighted summation of the features to obtain multi-scale fused features. Among them, the feature abstraction layer subscript F This is used to identify the features fused by the multi-scale fusion module guided by spatial attention. The above process realizes a dynamic multi-scale feature fusion mechanism based on spatial attention.

[0010] Step 4: Hierarchical Gated Interaction Module This invention's hierarchical gating interaction module progressively fuses high-level semantics with low-level details. The highest-level semantic features are output by a multi-scale fusion module guided by spatial attention. As the initial input, the feature abstraction layer First, channel expansion is performed using 3×3 convolution, batch normalization, and ReLU activation, and then the channel is uniformly divided into 3 groups along the channel dimension. ,in This is used to identify the number of feature groups. During the channel interaction phase, the first set of feature maps... After passing through the channel fusion layer, this unit consists of a 3×3 convolutional layer, a batch normalization (BN) layer, a ReLU layer, and a 1×1 convolutional layer. Its output features are divided into three parts: the current group output features. , out Features used to identify the output features of the current group and the interactive features passed to the next group. , frkUsed to identify the features passed to the next group for interaction, and the local control weight of this group. , gat Used to identify gating weights. The second set of features... Compared with the previous group After splicing, the layers are processed by a channel fusion layer and divided into three equal parts to obtain: , , For the last group, the features... With accumulation After splicing, processing with a channel fusion layer, and bisecting, we obtain: and Subsequently, the local control weights generated by all groups are... The input features are concatenated, then subjected to global average pooling and a bottleneck structure consisting of two 1×1 convolutions, ReLU, and Sigmoid activation functions to generate channel-gated weights. In the final fusion stage, the concatenated output feature channels are multiplied element-wise with the generated channel-gated weights to achieve feature modulation. The modulated features are then spatially integrated and their dimensions restored using a fusion unit consisting of a 3×3 convolutional layer, a BN layer, a ReLU layer, and a 1×1 convolutional layer. Finally, the fused features are combined with the original input features. The residuals are summed and then processed by the ReLU activation function to obtain the output feature map. subscript H Used to identify features processed by the channel gating interaction unit; subsequently, higher-level features The data is passed down level by level to lower layers, and then upsampled and combined with the features of the third layer. The element-wise addition is then input into the channel gating interaction unit to obtain the result. Repeat this process until you get Through the aforementioned layer-by-layer gating interaction process, high-level abstract semantic information can gradually permeate into lower-level feature representations and fully integrate with multi-level fine-grained spatial information, significantly enhancing the segmentation results in terms of regional integrity and boundary accuracy. This design achieves progressive refinement of cross-level features.

[0011] Step 5: Feature Decoding and Segmentation Result Output This invention relates to the final features Resolution restoration and prediction mapping are performed. Through operations such as feature smoothing, channel refinement and spatial scale restoration, the high-dimensional decoded features are mapped to segmentation prediction results with the same size as the original input image, and finally the pixel-level segmentation map is output.

[0012] During the training phase, a composite loss function is applied to the segmentation probability map output by the network. Jointly constrained segmentation results from multiple supervision dimensions at the pixel, region, and structure levels: Structural similarity loss Enhance the model's ability to model the overall structure of the target and fuzzy boundary regions:

[0013] subscript in formula p, g They are respectively P, G Pixels at the same position in the middle , They represent respectively with p, g The average pixel value within a local window centered on the pixel; , These represent the variance of pixel values ​​within the corresponding local window; This represents the covariance between the predicted image and the true label's pixel values ​​within that local window; , This is a stability constant used to prevent loss of the denominator. Pixel-level discrimination loss. A binary cross-entropy loss is used to constrain the global pixel classification accuracy.

[0014] In the formula i and j These represent the row and column indices of the segmentation result in the spatial dimension, respectively. H and W These represent the height and width of the segmentation probability map or the true label map in the spatial dimension, respectively. For real labels, For predicting probabilities. Regional consistency loss. Directly optimize the spatial overlap between the segmented region and the true label to enhance the regional integrity of the segmentation result:

[0015] By jointly optimizing the aforementioned multi-dimensional supervised loss, collaborative constraints on the segmentation results at the pixel, structure, and region levels are achieved.

[0016] During the inference phase, the colonoscopy image to be segmented is input into the trained three-branch collaborative gating network. The image sequentially passes through the three-branch scaling and backbone network feature extraction in step 1, the heterogeneous receptive field feature aggregation module for multi-scale adaptive perception in step 2, the spatial attention-guided multi-scale fusion module for spatial attention-guided multi-scale fusion in step 3, and the hierarchical gating interaction in step 4. Finally, the output layer at the end of the decoder in step 5 is activated by Sigmoid to obtain a pixel-level polyp segmentation probability map, which is then thresholded to generate the final binary segmentation mask.

[0017] Optionally, the backbone feature extraction network is not limited to PVTv2 and can be replaced with other visual Transformer or convolutional neural network architectures as needed.

[0018] Optionally, the number of parallel convolutional branches, the kernel size, and the dilation rate in the heterogeneous receptive field feature aggregation module can be adjusted according to task requirements.

[0019] Optionally, the number of groups in the spatial attention-guided multi-scale fusion module The size of the attention convolution kernel can be adjusted according to the feature dimension.

[0020] Optionally, the number of feature groups in the hierarchical gating interaction module can be set according to the decoding complexity requirements.

[0021] The beneficial effects of this invention are: This invention provides a colorectal polyp segmentation technique based on a three-branch collaborative gating network. First, it simulates the clinical diagnostic logic of "global observation - local focus" through three-branch input. Then, it enhances multi-scale feature representation using a heterogeneous receptive field feature aggregation module. Next, it achieves adaptive calibration and fusion of cross-scale features through a spatial attention-guided multi-scale fusion module. Finally, it progressively optimizes the feature decoding process using a hierarchical gating interactive decoder. Compared with existing methods, the segmentation method provided by this invention can more accurately address the challenges of large polyp scale differences, blurred boundaries, and complex morphologies. While reducing background interference, it effectively preserves edge details and improves segmentation integrity, thereby significantly improving segmentation accuracy and model generalization ability.

[0022] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above technical solution, purpose, features and advantages of the present invention more obvious and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0023] Figure 1 This is a diagram of the overall architecture of the three-branch collaborative gating network of the present invention.

[0024] Figure 2 This is a detailed structural diagram of the heterogeneous receptive field feature aggregation module of the present invention.

[0025] Figure 3 This is a detailed structural diagram of the spatial attention-guided multi-scale fusion module of the present invention.

[0026] Figure 4 This is a detailed structural diagram of the channel gating interaction unit of the present invention. Detailed Implementation

[0027] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0028] This invention proposes a three-branch cooperative gating network for colorectal polyp segmentation. To achieve accurate segmentation of colorectal polyps, this invention employs the following technical solution.

[0029] Phase 1: Data preparation and preprocessing.

[0030] The data used to train the model of this invention consists of colorectal endoscopy images, which must include pixel-level annotations of polyp regions. Images and corresponding labels are obtained from colorectal polyp endoscopy image datasets, including publicly available datasets such as Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS, and EndoScene. Each sample contains the original endoscopy image and its corresponding binary segmentation mask (polyp region is 1, background is 0).

[0031] Then, the images are preprocessed by scaling them all to a fixed height. H pixels × width W Pixels; pixel value normalization; online data augmentation strategies, including random horizontal / vertical flipping, random rotation (-15° to +15°), and brightness / contrast fine-tuning, are employed to increase model robustness.

[0032] Finally, to simulate the diagnostic process of an endoscopist's "global observation-local focusing," three scales of input images are generated for each training image: a main scale (1.0×, original size) image, a global auxiliary scale (0.5×, downsampled to half size) image, and a local magnification scale (1.5×, upsampled to 1.5 times size) image. The three scale images serve as inputs to the three branches of the network, respectively.

[0033] Phase 2: Construction of a segmentation network model for colorectal polyps.

[0034] like Figure 1As shown, the three-branch collaborative gating network proposed in this invention consists of: a hierarchical feature extraction module, a heterogeneous receptive field feature aggregation module, a spatial attention-guided multi-scale fusion module, and a hierarchical gating interaction module. The overall process is as follows: Given three images generated from the original colorectal endoscopy image at the main scale (1.0×), global auxiliary scale (0.5×), and local magnification scale (1.5×), they are respectively fed into a PVTv2 backbone network with shared weight parameters to extract hierarchical features. After the features of each layer are enhanced by the heterogeneous receptive field feature aggregation module, cross-scale fusion is performed by the spatial attention-guided multi-scale fusion module, and then different layer features are interacted through the hierarchical gating interaction module. After decoding and restoring the resolution, the final output is a segmentation map of the same size as the input. In this network, unless otherwise specified, convolution operations are performed with a stride of 1 and edge padding.

[0035] Step 1: Deep Feature Extraction Module In the feature extraction stage, the Pyramid Visual Transformer (PVTv2) is used as a shared backbone network to perform multi-level feature encoding on the input image. First, the original image is processed into three images with different resolutions, corresponding to scaling factors. The input includes the global auxiliary scale, the main scale, and the local magnification scale. For each scale... s Image data is input into a backbone network, where it generates feature maps at different levels of abstraction through four cascaded feature extraction stages. Specifically, the four output stages of the backbone network, from shallow to deep, are mapped to feature abstraction layers. The high-resolution features output from the second stage of the network are taken as the first-level features. The features output from the third stage are taken as the features of the second level. The features output from the fourth stage are taken as the features of the third level. The low-resolution semantic features output from the fifth stage are taken as the fourth-level features. Therefore, for each frame of the input image, a multi-scale feature pyramid set containing three scales, each with four abstract levels, is ultimately extracted. subscript B Used to identify features extracted from the backbone network, scale Feature abstraction layer This provides rich basic feature representations for subsequent multi-scale interaction and fusion.

[0036] Step 2: Heterogeneous receptive field feature aggregation module Heterogeneous receptive field feature aggregation module structure as follows Figure 2 As shown, this method enhances the feature representation capability of targets at different scales, with the input being the output features of the backbone network. The enhanced features are denoted as subscript D Used to identify features enhanced by the heterogeneous receptive field feature aggregation module, scale Feature abstraction layer The specific steps are as follows: First, the input features are preprocessed. The preprocessing process includes a 3×3 convolution operation, batch normalization (BN) operation, and ReLU activation function operation to obtain preprocessed features. The 3×3 convolution extracts local features independently for each channel while maintaining the feature map spatial size, reducing computational complexity and enhancing local representation capabilities. Subsequent operations are processed in parallel by two paths. One path generates modulation weights for different receptive field branches, based on the preprocessed features. Specifically, global average pooling is first performed on the preprocessed features to compress their spatial dimensions, obtaining channel-level description vectors reflecting the overall semantic information. Then, a 1×1 convolution is used to map these channel-level description vectors, generating a weight feature tensor with 4×C channels, where C is the number of channels corresponding to a single receptive field branch. Next, the weight feature tensor is divided into four groups along the channel dimension, corresponding to the four receptive field branches, and the weights in each group are normalized using the Sigmoid activation function, limiting their values ​​to between 0 and 1. The weight coefficients are directly generated from the input features through the aforementioned computational process and are used to perform channel-wise weighted modulation on the features of the corresponding receptive field branches, thereby achieving adaptive allocation of the response intensity of different receptive field features. Simultaneously, the preprocessed features are fed into another heterogeneous receptive field convolutional structure, which includes four parallel convolutional branches: branch 1 uses a 1×1 convolution to focus on local details; branch 2 uses a 3×3 convolution, corresponding to a 3×3 receptive field; branch 3 uses a 3×3 dilated convolution with a dilation rate of 3, its equivalent receptive field being 7×7; and branch 4 uses a 3×3 dilated convolution with a dilation rate of 6, its equivalent receptive field being 13×13. After obtaining the output features of each convolutional branch, the weight coefficients are... , , , The output features applied to the corresponding branches are weighted and modulated by multiplying each receptive field branch feature through channel-wise multiplication. This adaptively adjusts the response intensity and contribution ratio of different receptive field features according to the actual semantic distribution of the input features. Subsequently, the four weighted and modulated features are concatenated along the channel dimension and then passed through a 1×1 convolutional layer for cross-channel information fusion and feature compression, finally outputting the feature aggregated from the heterogeneous receptive field features. .

[0037] Step 3: Spatial attention-guided multi-scale fusion module Spatial attention-guided multi-scale fusion module structure, such as Figure 3 As shown, this is used to fuse feature information from different input scales. The input is the feature data enhanced by the heterogeneous receptive field feature aggregation module. This module is used for the same level. Three scale features within Alignment, weighting, and fusion are performed to obtain a fused feature representation with stronger discriminative power.

[0038] In the specific implementation process, spatial resolution alignment is first performed on the input features from different scales. For resolutions below the intermediate scale (…),… The features of the ) are sequentially adjusted to have spatial dimensions consistent with those of the intermediate scale features through 3×3 convolution and upsampling operations; for features with resolutions higher than the intermediate scale ( The features of the ) are then processed sequentially through 3×3 convolution, as well as parallel max pooling and average pooling operations. The results of the two pooling operations are then added element-wise to achieve spatial size downsampling alignment, thereby ensuring that features of different scales maintain a consistent spatial resolution.

[0039] After completing resolution alignment, at intermediate scale s Features corresponding to =1.0 For reference, spatial attention is further employed to enhance the target region and suppress background interference. Specifically, parallel max pooling and average pooling operations are performed on the features along the channel dimension, and then the two are concatenated along the channel dimension. Subsequently, the concatenated features are input into a 7×7 convolutional layer, and a spatial attention map is generated using the Sigmoid activation function. The spatial attention map is multiplied element-wise with the original aligned features to enhance the response of the target region and suppress feature representation of irrelevant regions.

[0040] After spatial attention weighting, the three scale features are concatenated along the channel dimension to obtain cross-scale features. Then, spatial weights are generated by sequentially applying 1×1, 3×3, and 1×1 convolutions followed by Softmax operations. Finally, these weights are element-wise weighted onto the cross-scale features to obtain the final multi-scale fusion feature. subscript F Feature abstraction layer used to identify features fused by the spatial attention-guided multi-scale fusion module. .

[0041] Step 4: Hierarchical Gated Interaction Module The main structure of the hierarchical interactive decoding module is as follows: Figure 1As shown, this module progressively fuses high-level features containing semantic information down to low-level features rich in detail, enhancing the richness and discriminative power of feature representation. From Initially, the data is directly input into the channel gating interaction unit for processing to obtain the initial features. subscript H Used to identify features processed by the channel gating interaction unit, feature abstraction layer Subsequently, regarding Perform upsampling to improve its spatial resolution and integrate it with the third-layer features. Alignment and element-wise addition are performed to fuse high-level semantic information with the spatial detail features of the current layer; the fused features are then input into the channel-gated interaction unit to obtain... Repeat this process to finally obtain the output features. .

[0042] The channel gating interaction unit, as shown in Figure 4 As shown, the fourth layer features output by the multi-scale fusion module guided by spatial attention. As the initial input, the feature is first expanded by a factor of three through 3×3 convolution, batch normalization (BN), and ReLU activation. Then, the expanded feature is evenly divided into three sub-features along the channel dimension. ,in , used to identify the number of feature groups.

[0043] During the channel interaction phase, the first set of features is first... The input is processed in an interactive unit, which consists of a 3×3 convolutional layer, a batch normalization layer, a ReLU activation function, and a 1×1 convolutional layer connected in sequence. After processing by the channel fusion layer, the output features are divided into three parts along the channel dimension, representing the output features of the current group. , out Features used to identify the output of the current group, and interactive features used to pass to the next group. , frk Features used to identify the features passed to the next group for interaction, and local control weights representing the importance of the current group. , gat Used to identify gating weights. Then, the second set of features... With the previous set of interaction features The channels are spliced ​​together and input into a channel fusion layer with the same structure for processing. The output is also divided into three parts. , and The last set of features Combine it with the accumulated interaction features After channel concatenation, the input is processed by the channel fusion layer. The output features are divided into two parts along the channel dimension, which are the third set of output features. and the third group of departmental control weight characteristics .

[0044] After completing the channel interaction of each group of features, generate local control weight features for all groups. The concatenation is performed along the channel dimension, and a global average pooling (GAP) operation is performed on the concatenated features to compress the spatial dimension and extract global channel description information. Subsequently, the total gating weights for feature modulation are generated by sequentially passing through a 1×1 convolution, ReLU activation, 1×1 convolution and Sigmoid activation structure.

[0045] In the final fusion stage, the output features of each group are first... The features are concatenated along the channel dimension and then element-wise multiplied with the generated total gating weights to achieve adaptive modulation of features from different channels. The modulated features are then input into an interaction unit consisting of a 3×3 convolution, batch normalization, ReLU activation function, and a 1×1 convolutional layer to complete spatial information integration and channel dimension restoration. Finally, the fused features are combined with the original input features. The features are added together and then processed using the ReLU activation function to obtain the final output feature map of this layer. .

[0046] Through the above specific implementation methods, the hierarchical gating interaction module realizes the step-by-step transmission and progressive refinement of cross-level feature information, effectively enhancing the collaborative modeling capability between high-level semantics and low-level details.

[0047] Step 5: Feature Decoding and Output Segmentation Results In obtaining the underlying layer Output characteristics Subsequently, resolution restoration and prediction mapping are performed on the output features. Specifically, bilinear interpolation is first used to upsample the features to improve spatial resolution. Then, a feature transformation unit containing a 3×3 convolutional layer, a normalization layer, and a nonlinear activation function is used to smooth and refine the channels. On this basis, upsampling is performed again to make the feature map size consistent with the original input image, and the channel dimension is gradually compressed through convolution operations. Finally, a 1×1 convolutional layer is used to map the features into a single-channel feature map, and a normalized activation function can be combined to obtain pixel-level segmentation results, thus completing the process of generating fine-grained segmentation output from high-dimensional decoded features.

[0048] The third stage: training and inference of the network model.

[0049] Based on the above network architecture, the total loss function used for training the deep network in this invention consists of three parts:

[0050] in, and Weight the loss term. Structural similarity loss. By comparing the segmentation prediction probability map P Compared with real label images G The brightness, contrast, and structural correlation within a local window enhance the network's ability to model fuzzy boundaries and local structural consistency. The formula is as follows:

[0051] subscript in formula p, g They are respectively P, G Pixels at the same position in the middle , They represent respectively with p, g The average pixel value within a local window centered on the pixel; , These represent the variance of pixel values ​​within the corresponding local window; This represents the covariance between the predicted image and the true label's pixel values ​​within that local window; , It is a stability constant used to prevent the loss of the denominator.

[0052] Regional Discrimination Loss The binary cross-entropy loss is used to constrain the global pixel-level classification accuracy, and its formula is as follows:

[0053] In the formula i and j These represent the row and column indices of the segmentation result in the spatial dimension, respectively. H and W These represent the height and width of the segmentation probability map or the true label map in the spatial dimension, respectively. Indicates the network at pixel position The probability value that is predicted to be the foreground. This represents the actual labeled value of the corresponding pixel. Shape consistency loss. The formula for directly optimizing the spatial overlap between the segmented region and the real label is:

[0054] The three loss components are weighted and accumulated to form the main loss, which is used to optimize the supervised network model at the pixel level, local region level, and whole image level.

[0055] This invention builds, trains, and tests the proposed network model on the PyTorch deep learning framework, and uses an NVIDIA GeForceTX 2080Ti GPU for accelerated model training. During model training, the SGD gradient descent algorithm is used, with a learning rate of 0.001, a batch size of 32, and an image size of 352×352.

[0056] Performance verification of the network model: To verify the effectiveness of the method of this invention, it was comprehensively compared with four representative polyp segmentation methods, including CNN-based UNet and PraNet, and Transformer-based DBE-Net and Polyp-PVT. Two of the most commonly used metrics in the field of medical image segmentation were used for evaluation: the Dice similarity coefficient, Dice = (2×TP) / (2×TP+FP+FN), which measures the overlap between the predicted and real regions; and the Intersection over Union (IoU): IoU = TP / (TP+FP+FN), which also measures overlap, but with a more stringent standard. Here, TP, FP, and FN represent the number of true positive, false positive, and false negative pixels, respectively.

[0057] The quantitative results on the main test sets Kvasir-SEG and CVC-ClinicDB are shown in the table below. The method of this invention achieved the best Dice score (0.933) and the highest IoU score (0.890) on the Kvasir-SEG dataset. It also achieved optimal performance on the CVC-ClinicDB dataset.

[0058] Comparison results of different methods on the Kvasir-SEG and CVC-ClinicDB datasets

[0059] This invention details and verifies a colorectal polyp segmentation method based on a three-branch cooperative gating network. This method effectively addresses core challenges in polyp segmentation, such as large scale differences, ambiguous boundaries, and complex morphologies, through a network architecture simulating clinical diagnostic logic, a dynamically adaptive multi-scale processing mechanism, and a refined feature fusion and decoding strategy. Extensive experimental results demonstrate that the three-branch cooperative gating network not only achieves state-of-the-art performance on multiple standard test sets but also exhibits outstanding robustness in cross-center and cross-device generalization tests, providing an efficient and reliable solution for the development of clinical computer-aided polyp detection systems.

[0060] Those skilled in the art will understand that various changes and modifications can be made to the embodiments without departing from the core idea of ​​the present invention: the backbone network can be replaced with other visual backbones such as SwinTransformer and ConvNeXt; the scale factors of the three branches (e.g., 0.5×, 1.0×, 1.5×) can be adjusted according to the distribution of polyp scales in the specific dataset; the number of parallel branches, convolutional kernel size, and dilation rate in the heterogeneous receptive field feature aggregation module can be adjusted; the number of groups and the size of the attention convolutional kernel in the spatial attention-guided multi-scale fusion module can be adjusted; the number of feature groups and the specific number of layers in the hierarchical gating interaction module can be changed; boundary loss can be introduced into the loss function; and conditional random field post-processing can be used to further optimize the edges. The core module of the present invention can be applied to other medical image segmentation tasks, such as skin lesion segmentation, retinal vessel segmentation, and lung nodule segmentation.

[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for segmenting colorectal polyps based on a three-branch cooperative gating network, characterized in that, Includes the following steps: Step 1: Three-branch scaling and backbone network feature extraction The input colonoscopy images are processed in parallel into three views at different scales: the original resolution master-scale image, a global auxiliary-scale image downscaled by 0.5x, and a magnified-scale image by 1.5x. These three scale images are then fed into a shared backbone feature extraction network to extract multi-level features, forming three complementary feature pyramids. The features output by each layer of the backbone feature extraction network are denoted as follows: , among which scale Feature abstraction layer subscript B Used to identify features extracted from the backbone network, where low-level features preserve detailed textures and high-level features encode semantic information; Step 2: Heterogeneous receptive field feature aggregation module Next, regarding Feature enhancement is performed on each feature using a heterogeneous receptive field feature aggregation module to obtain features. , among which scale Feature abstraction layer subscript D Used to identify features enhanced by the heterogeneous receptive field feature aggregation module; for the first feature at scale s l Layer features First, preprocessing is performed using 3×3 depthwise convolution, batch normalization, and ReLU activation. The preprocessed features are divided into two paths. One path sequentially undergoes global average pooling and 1×1 convolution to compress the spatial dimension and capture global contextual information, generating a set of channel-dimensional dynamic weight vectors, which are then divided into four independent weight coefficients along the channel dimension. , , , After activation by a sigmoid function, another path enters four convolutional branches in parallel: a 1×1 standard convolutional branch, a 3×3 standard convolutional branch, a 3×3 dilated convolutional branch with a dilation rate of 3, and a 3×3 dilated convolutional branch with a dilation rate of 6. Then, the output features of each of the four branches are weighted and modulated channel-wise using the generated four weight coefficients. Finally, the modulated four features are concatenated and fused across channels using a 1×1 convolutional layer to output the enhanced features. ; Step 3: Spatial attention-guided multi-scale fusion module To fuse inputs from different scales Based on the feature information, a spatial attention-guided multi-scale fusion module is designed; the input of this module is a set of features from the same semantic level but located in different branches of the network, which have been enhanced by a heterogeneous receptive field feature aggregation module. , Feature abstraction layer First, feature preprocessing and resolution alignment are performed on input features at different scales. Specifically, low-resolution features are aligned by 3×3 convolution and upsampling. High-resolution features are aligned by 3×3 convolution followed by max pooling and average pooling, and then the spatial size is adjusted to the intermediate scale features. After resolution alignment, max pooling and average pooling operations are performed on the features along the channel dimension. The results of these two pooling operations are then concatenated along the channel dimension. Subsequently, a 7×7 convolutional layer is passed through which a spatial attention map is generated by sigmoid activation and multiplied element-wise with the original features. Next, concatenation is performed along the channel dimension using a convolutional structure consisting of 1×1, 3×3, and 1×1 convolutions in sequence. Softmax operations are then applied along the scale dimension to generate corresponding spatial weights. Finally, the learned spatial weights are used to perform element-wise weighted summation of the features to obtain multi-scale fused features. Among them, the feature abstraction layer subscript F Used to identify features fused by multi-scale fusion modules guided by spatial attention; Step 4: Hierarchical Gated Interaction Module The highest-level semantic features output by the spatial attention-guided multi-scale fusion module As the initial input, the feature abstraction layer First, channel expansion is performed using 3×3 convolution, batch normalization, and ReLU activation, and then the channel is uniformly divided into 3 groups along the channel dimension. ,in This is used to identify the number of feature groups; during the channel interaction phase, the first set of feature maps... After passing through the channel fusion layer, this unit consists of a 3×3 convolutional layer, a batch normalization (BN) layer, a ReLU layer, and a 1×1 convolutional layer. Its output features are divided into three parts: the current group output features. , out Features used to identify the output features of the current group and the interactive features passed to the next group. , frk Used to identify the features passed to the next group for interaction, and the local control weight of this group. , gat Used to identify gating weights; the second set of features Compared with the previous group After splicing, the layers are processed by a channel fusion layer and divided into three equal parts to obtain: , , For the last group, the features With accumulation After splicing, processing with a channel fusion layer, and bisecting, we obtain: and Subsequently, the local control weights generated by all groups are... The input features are concatenated, then subjected to global average pooling and a bottleneck structure consisting of two 1×1 convolutions, ReLU, and Sigmoid activation functions to generate channel-gated weights. In the final fusion stage, the output feature channels of each group are concatenated and multiplied element-wise with the generated channel-gated weights to achieve feature modulation. The modulated features are then spatially integrated and their dimensions restored using a fusion unit consisting of a 3×3 convolutional layer, a BN layer, a ReLU layer, and a 1×1 convolutional layer. Finally, the fused features are combined with the original input features. The residuals are summed and then processed by the ReLU activation function to obtain the output feature map. subscript H Used to identify features processed by the channel gating interaction unit; subsequently, higher-level features The data is passed down level by level to lower layers, and then upsampled and combined with the features of the third layer. The element-wise addition is then input into the channel gating interaction unit to obtain the result. Repeat this process until you get ; Step 5: Feature Decoding and Segmentation Result Output For final features Resolution restoration and prediction mapping are performed. Through feature smoothing, channel refinement and spatial scale restoration, the high-dimensional decoded features are mapped to the segmentation prediction results with the same size as the original input image, and finally the pixel-level segmentation map is output. During the training phase, a composite loss function is applied to the segmentation probability map output by the network. Jointly constrained segmentation results from multiple supervision dimensions at the pixel, region, and structure levels: Structural similarity loss Enhance the model's ability to model the overall structure of the target and fuzzy boundary regions: subscript in formula p, g They are respectively P, G Pixels at the same position in the middle , They represent respectively with p, g The average pixel value within a local window centered on the pixel; , These represent the variance of pixel values ​​within the corresponding local window; This represents the covariance between the predicted image and the true label's pixel values ​​within that local window; , The stability constant; pixel-level discrimination loss A binary cross-entropy loss is used to constrain the global pixel classification accuracy. In the formula i and j These represent the row and column indices of the segmentation result in the spatial dimension, respectively. H and W These represent the height and width of the segmentation probability map or the true label map in the spatial dimension, respectively. For real labels, For predicting probabilities; regional consistency loss Directly optimize the spatial overlap between the segmented region and the true label to enhance the regional integrity of the segmentation result: 。 2. The method according to claim 1, characterized in that: Step 1: Deep Feature Extraction Module In the feature extraction stage, the Pyramid Visual Transformer (PVTv2) is used as a shared backbone network to perform multi-level feature encoding on the input image. First, the original image is processed into three images with different resolutions, corresponding to scaling factors. The input includes the global auxiliary scale, the main scale, and the local magnification scale; for each scale... s Image data is input into the backbone network, where it generates feature maps at different levels of abstraction through four cascaded feature extraction stages. Specifically, the four output stages of the backbone network, from shallow to deep, are mapped to feature abstraction layers. The high-resolution features output from the second stage of the network are taken as the first-level features. The features output from the third stage are taken as the features of the second level. The features output from the fourth stage are taken as the features of the third level. The low-resolution semantic features output from the fifth stage are taken as the fourth-level features. Therefore, for each frame of the input image, a multi-scale feature pyramid set containing three scales, each with four abstract levels, is finally extracted. subscript B Used to identify features extracted from the backbone network, scale Feature abstraction layer .

3. The method according to claim 2, characterized in that: Step 2: Heterogeneous receptive field feature aggregation module The input is the output features of the backbone network. The enhanced features are denoted as subscript D Used to identify features enhanced by the heterogeneous receptive field feature aggregation module, scale Feature abstraction layer The specific steps are as follows: First, the input features are preprocessed. The preprocessing process includes a 3×3 convolution operation, a batch normalization operation, and a ReLU activation function operation to obtain the preprocessed features. The 3×3 convolution, while maintaining the spatial size of the feature map, independently extracts local features from each channel to reduce computational complexity and enhance local representation capabilities. Subsequent operations are processed in parallel on two paths. Specifically, firstly, global average pooling is performed on the preprocessed features to compress their spatial dimension, obtaining channel-level description vectors that reflect the overall semantic information. Then, 1×1 convolutions are used to map these channel-level description vectors, generating a weighted feature tensor with 4×C channels, where C is the number of channels corresponding to a single receptive field branch. Next, the weighted feature tensor is divided into four groups along the channel dimension, corresponding to the four receptive field branches, and the weights in each group are normalized using a Sigmoid activation function. The weight coefficients are determined by limiting their values ​​to between 0 and 1. These weight coefficients are directly generated from the input features through the aforementioned computational process and used to perform channel-wise weighted modulation on the corresponding receptive field branch features, thereby achieving adaptive allocation of response intensity for different receptive field features. Simultaneously, the preprocessed features are fed into another heterogeneous receptive field convolutional structure, which includes four parallel convolutional branches: Branch 1 uses a 1×1 convolution to focus on local details; Branch 2 uses a 3×3 convolution, corresponding to a 3×3 receptive field; Branch 3 uses a 3×3 dilated convolution with a dilation rate of 3, its equivalent receptive field being 7×7; and Branch 4 uses a 3×3 dilated convolution with a dilation rate of 6, its equivalent receptive field being 13×13. After obtaining the output features of each convolutional branch, the weight coefficients are... , , , The output features applied to the corresponding branches are weighted and modulated by multiplying each receptive field branch feature through a channel-wise multiplication method. This adaptively adjusts the response intensity and contribution ratio of different receptive field features according to the actual semantic distribution of the input features. Subsequently, the four weighted and modulated features are concatenated and concatenated along the channel dimension, and then passed through a 1×1 convolutional layer for cross-channel information fusion and feature compression. Finally, the aggregated heterogeneous receptive field features are output. .

4. The method according to claim 3, characterized in that: Step 3: Spatial attention-guided multi-scale fusion module The structure of the spatial attention-guided multi-scale fusion module is shown in Figure 3, which is used to fuse feature information from different input scales. The input is the feature enhanced by the heterogeneous receptive field feature aggregation module. This module is used for the same level. Three scale features within Alignment, weighting, and fusion are performed to obtain a fusion feature representation with stronger discriminative power; First, spatial resolution alignment is performed on the input features from different scales; for features with resolutions lower than the intermediate scale (… The features of the ) are sequentially adjusted to have spatial dimensions consistent with those of the intermediate scale features through 3×3 convolution and upsampling operations; for features with resolutions higher than the intermediate scale ( The features of the ) are then processed sequentially through 3×3 convolution, as well as parallel max pooling and average pooling operations. The results of the two pooling operations are then added element by element to achieve spatial size downsampling alignment, so that features of different scales maintain a consistent spatial resolution. After completing resolution alignment, at intermediate scale s Features corresponding to =1.0 For reference, spatial attention is further employed to enhance the target region and suppress background interference. Specifically, parallel max pooling and average pooling operations are performed on the features in the channel dimension, and then the two are concatenated in the channel dimension. Subsequently, the concatenated features are input into a 7×7 convolutional layer, and a spatial attention map is generated using the Sigmoid activation function. The spatial attention map is multiplied element-wise with the original aligned features to enhance the response of the target region and suppress the feature representation of irrelevant regions. After spatial attention weighting, the three scale features are concatenated along the channel dimension to obtain cross-scale features. Then, corresponding spatial weights are generated by sequentially performing 1×1 convolution, 3×3 convolution, 1×1 convolution, and Softmax operations. Finally, these weights are used to perform element-wise weighting on the cross-scale features to obtain the final multi-scale fusion features. subscript F Feature abstraction layer used to identify features fused by the spatial attention-guided multi-scale fusion module. .

5. The method according to claim 4, characterized in that: Step 4: Hierarchical Gated Interaction Module from Initially, the data is directly input into the channel gating interaction unit for processing to obtain the initial features. subscript H Used to identify features processed by the channel gating interaction unit, feature abstraction layer Subsequently, regarding Perform upsampling to improve its spatial resolution and integrate it with the third-layer features. Alignment and element-wise addition are performed to fuse high-level semantic information with the spatial detail features of the current layer; the fused features are then input into the channel-gated interaction unit to obtain... Repeat this process to finally obtain the output features. ; The channel-gated interaction unit, as shown in Figure 4, outputs the fourth layer features from the multi-scale fusion module guided by spatial attention. As the initial input, the feature is first processed by 3×3 convolution, batch normalization (BN), and ReLU activation, which multiplies the number of channels. Then, the expanded feature is evenly divided into three sub-features along the channel dimension. ,in , used to identify the number of feature groups; During the channel interaction phase, the first set of features is first... The input is processed in an interactive unit, which consists of a 3×3 convolutional layer, a batch normalization layer, a ReLU activation function, and a 1×1 convolutional layer connected in sequence. After processing by the channel fusion layer, the output features are divided into three parts along the channel dimension, which are the output features of the current group. , out Features used to identify the output of the current group, and interactive features used to pass to the next group. , frk Features used to identify the features passed to the next group for interaction, and local control weights representing the importance of the current group. , gat Used to identify gating weights; subsequently, the second set of features... With the previous set of interaction features The channels are spliced ​​together and input into a channel fusion layer with the same structure for processing. The output is also divided into three parts. , and The last set of features Combine it with the accumulated interaction features After channel concatenation, the input is processed by the channel fusion layer. The output features are divided into two parts along the channel dimension, which are the third set of output features. and the third group of departmental control weight characteristics ; After completing the channel interaction of each group of features, generate local control weight features for all groups. The features are concatenated along the channel dimension, and a global average pooling (GAP) operation is performed on the concatenated features to compress the spatial dimension and extract global channel description information. Subsequently, the total gating weights for feature modulation will be generated by sequentially passing through 1×1 convolution, ReLU activation, 1×1 convolution and Sigmoid activation structures. In the final fusion stage, the output features of each group are first... The data is concatenated along the channel dimension and then multiplied element-wise with the generated total gating weight to achieve adaptive modulation of different channel features. The modulated features are then input into an interaction unit consisting of a 3×3 convolution, batch normalization, ReLU activation function, and a 1×1 convolutional layer to complete spatial information integration and channel dimension restoration. Finally, the fused features are combined with the original input features. The features are added together and then processed using the ReLU activation function to obtain the final output feature map of this layer. .

6. The method according to claim 5, characterized in that: Step 5: Feature Decoding and Output Segmentation Results In obtaining the underlying layer Output characteristics Subsequently, the output features undergo resolution restoration and prediction mapping. Specifically, bilinear interpolation is first used to upsample the features to improve spatial resolution. Then, a feature transformation unit containing a 3×3 convolutional layer, a normalization layer, and a nonlinear activation function is used to smooth and refine the channels. On this basis, upsampling is performed again to make the feature map size consistent with the original input image, and the channel dimension is gradually compressed through convolution operations. Finally, a 1×1 convolutional layer is used to map the features into a single-channel feature map, and a normalized activation function can be combined to obtain pixel-level segmentation results, thus completing the process of generating fine-grained segmentation output from high-dimensional decoded features.

Images