Mixed CNN-Transform colon polyp image segmentation method combining edge guidance and double attention mechanism

Abstract

The invention provides a hybrid CNN-Transform colon polyp image segmentation method combining edge guidance and a double attention mechanism, and is applied to the technical field of medical data processing. According to the method, a CNN encoder is adopted to extract multi-scale local features, a Transform encoder is adopted to extract global context, an edge probability graph is generated by introducing an edge guide branch based on shallow layer features, a gating coefficient is calculated according to the edge probability graph and / or statistics obtained by intermediate prediction, weighted fusion is performed on the local and global features, and the local feature and the global feature are integrated. And the decoder performs up-sampling step by step and outputs a segmentation result. Compared with the prior art, the method has the advantages in boundary integrity and small target detection. Experiments show that on a Kvasair-SEG data set, the optimal Dice of a verification set of the scheme is 0.892, the optimal HD95 of the verification set of the scheme is 12.2, the Dice of a test set of the scheme is 0.89, and the optimal HD95 of the test set of the scheme of the scheme is 12.3.

Description

Technical Field

[0001] This invention relates to the field of medical data processing technology, and in particular to a hybrid CNN-Transformer method for segmenting colon polyps by combining edge guidance and dual attention mechanisms. Background Technology

[0002] With the incidence of colorectal cancer (CRC) rising year by year, early screening is of great significance for improving survival rates and reducing mortality. Digestive endoscopy, as a primary clinical diagnostic method, plays a crucial role in the early detection of CRC. However, in clinical practice, polyps or lesions often exhibit characteristics such as small size, indistinct borders, similar color, or attachment to complex backgrounds, leading to a high rate of missed diagnoses due to the reliance on experience and strong subjectivity inherent in traditional manual interpretation.

[0003] Traditional image segmentation models, such as U-Net and its variants, while capable of extracting local texture features and achieving certain results in static image scenes, still face problems such as insufficient global modeling capabilities, coarse edge detail processing, and difficulty in uniformly handling multi-scale structures, especially struggling to identify small targets and lesions with weak boundaries. Furthermore, most segmentation models do not explicitly model edge information, resulting in broken lesion contours and missed markings, severely impacting their clinical value.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background disclosed in this invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a hybrid CNN-Transformer image segmentation method for colon polyps that combines edge guidance and dual attention mechanisms, which at least to some extent overcomes the problems existing in the prior art. It aims to solve the problems of edge blurring, missed detection of small targets, and insufficient global modeling in existing endoscopic image segmentation methods. The application proposes a hybrid CNN-Transformer image segmentation method, EDTrans, which integrates edge guidance and dual attention mechanisms, and enhances generalization ability and boundary recognition ability while maintaining segmentation accuracy.

[0006] According to one aspect of this application, a hybrid CNN-Transformer image segmentation method for colonic polyps combining edge guidance and dual attention mechanisms is provided, comprising: acquiring preprocessed digestive endoscopy image data; constructing a training set, validation set, and test set based on the Kvasir-SEG dataset, wherein the dataset contains the original images and corresponding colonic polyp segmentation mask annotation information; training a colonic polyp segmentation model using a hybrid structure fusing CNN and Transformer based on the constructed dataset; inputting the preprocessed digestive endoscopy images into an encoder, extracting multi-scale feature maps layer by layer; introducing a hybrid CNN-Transformer module into the deepest multi-scale feature map of the encoder, extracting local texture features through CNN branches and global context features through Transformer branches; and optimizing the fusion ratio of local and global features in real time based on the dynamic characteristics of the polyp region using an adaptive weight adjustment gating function, thereby achieving deep integration with polyp morphological features. A degree-bound differential dynamic gating fusion mechanism is implemented. An edge guidance module is introduced into the shallow multi-scale feature map of the encoder, using dilated convolutions with dilation rates of 1, 3, 5, and 7 to extract multi-scale edge responses, which are then activated by the Sigmoid function to generate an edge probability map, and fused to the backbone feature map through an attention mechanism. In the feature fusion stage, a dual attention module and a spatial pyramid pooling module are introduced. The dual attention module includes channel forward attention, channel backward attention, and spatial attention sub-modules to enhance the model's sensitivity to salient and low-response regions. A UNet++-style decoder structure is used to fuse shallow skip connection features, edge enhancement features, and multi-scale context features from the encoder, progressively upsampling to restore resolution and generate a target colon polyp segmentation mask. The target colon polyp segmentation mask is processed, using a weighted sum of binary cross-entropy loss and Dice loss as the joint loss function. The model is trained using the Adam optimizer and a dynamic learning rate scheduling strategy to generate colon polyp segmentation results.

[0007] Another aspect of this application discloses a hybrid CNN-Transformer image segmentation device for colonic polyps, combining edge guidance and dual attention mechanisms. The device includes: an acquisition module for acquiring preprocessed digestive endoscopy image data and constructing a training set, validation set, and test set based on the Kvasir-SEG dataset, wherein the dataset contains original images and corresponding colonic polyp segmentation mask annotations; a processing module for training a colonic polyp segmentation model based on the constructed dataset using a hybrid structure fusing CNN and Transformer; inputting the preprocessed digestive endoscopy images into an encoder, extracting multi-scale feature maps layer by layer, introducing a hybrid CNN-Transformer module into the deepest multi-scale feature map of the encoder, extracting local texture features through CNN branches and global context features through Transformer branches, and optimizing the fusion ratio of local and global features in real time based on the dynamic characteristics of the polyp region using an adaptive weight adjustment gating function, thereby achieving segmentation of the polyp. A differentiated dynamic gating fusion mechanism deeply bound to morphological features is implemented. An edge guidance module is introduced into the shallow multi-scale feature map of the encoder, using dilated convolutions with dilation rates of 1, 3, 5, and 7 to extract multi-scale edge responses, which are then activated by the Sigmoid function to generate an edge probability map, and fused to the backbone feature map through an attention mechanism. In the feature fusion stage, a dual attention module and a spatial pyramid pooling module are introduced. The dual attention module includes channel forward attention, channel backward attention, and spatial attention sub-modules to enhance the model's sensitivity to salient and low-response regions. A UNet++-style decoder structure is used to fuse shallow skip connection features, edge enhancement features, and multi-scale contextual features from the encoder, progressively upsampling to restore resolution and generate a target colon polyp segmentation mask. The target colon polyp segmentation mask is processed, using a weighted sum of binary cross-entropy loss and Dice loss as the joint loss function. The model is trained using the Adam optimizer and a dynamic learning rate scheduling strategy to generate colon polyp segmentation results.

[0008] According to another aspect of this application, an electronic device includes: a first processor; and a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the above-described hybrid CNN-Transformer colon polyp image segmentation method combining edge guidance and dual attention mechanisms by executing the executable instructions.

[0009] This application presents a hybrid CNN-Transformer method for colonic polyp image segmentation that combines edge guidance and dual attention mechanisms. The edge guidance module improves the model's sensitivity to polyp boundaries, significantly reducing HD95. It has strong recognition capabilities for small targets, and the dual attention mechanism enhances recall and difficacy for small polyps and blurred regions, reducing false negatives. It considers both global context and local structure, using a hybrid CNN and Transformer model to achieve high-precision segmentation in complex backgrounds. The model exhibits excellent generalization ability; cross-dataset experiments show that Boundary-Aware Hybrid Transformer with DualAttention for Clinically Reliable Endoscopic Segmentation (EDTrans) outperforms mainstream methods on the Kvasir dataset. The model has a lightweight structure and fast inference speed, making it suitable for embedding in clinical auxiliary diagnostic platforms.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0011] Figure 1 This document illustrates a flowchart of a hybrid CNN-Transformer method for segmenting colonic polyps that combines edge guidance and dual attention mechanisms, according to an embodiment of this application. Figure 2 This paper presents a visualization comparing the effects of a colon polyp segmentation model provided in one embodiment of this application. Figure 3 This illustration shows a schematic diagram of a hybrid CNN-Transformer colon polyp image segmentation device that combines edge guidance and dual attention mechanisms, according to an embodiment of this application. Detailed Implementation

[0012] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0013] The following is combined Figure 1 This paper describes a hybrid CNN-Transformer method for segmenting colonic polyps, combining edge guidance and dual attention mechanisms, according to exemplary embodiments of this application. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application are applicable to any suitable scenario.

[0014] In one implementation, Figure 1 The schematic diagram illustrates a flowchart of a hybrid CNN-Transformer colon polyp image segmentation method that combines edge guidance and dual attention mechanisms according to an embodiment of this application.

[0015] S101. Obtain preprocessed digestive endoscopy image data and construct training, validation, and test sets based on the Kvasir-SEG dataset.

[0016] In one implementation, the publicly available Kvasir-SEG dataset is obtained. This dataset contains 1000 digestive endoscopy images with resolutions ranging from 332×487 to 1920×1072. Each image is accompanied by a manually annotated colonic polyp segmentation mask (pixel-level annotation of the polyp region, with clear boundaries between the polyp region and the background region in the mask). The dataset includes "polyp001.jpg" (the original endoscopy image showing a pedunculated polyp in the colon) and the corresponding "polyp001_mask.png" (mask image, with white areas precisely marking the polyp outline and black areas representing the normal mucosal background).

[0017] The original images and their corresponding masks are standardized. Specific steps include cropping all images to a uniform 256×256 resolution to eliminate the impact of original image size differences on model training; employing random rotation (±30°), horizontal / vertical flipping, brightness adjustment (±20%), and adding Gaussian noise to expand dataset diversity and improve model generalization ability; scaling the input images to a uniform resolution and performing normalization or standardization (e.g., scaling to [0,1] or subtracting the mean / dividing the variance); and maintaining a single-channel {0,1} index format for the labeled masks to ensure data format consistency. A 1200×800 resolution original image is cropped to obtain a 256×256 sub-image, which is then horizontally flipped to generate enhanced samples. Simultaneously, the corresponding mask image is flipped to maintain spatial consistency between the polyp region labeling and the image.

[0018] The 1000 preprocessed images and their corresponding masks were divided into three groups in a 7:1.5:1.5 ratio: a training set of 700 images and masks for learning and fitting model parameters; a validation set of 150 images and masks for monitoring the model's generalization ability during training and assisting in adjusting hyperparameters (such as learning rate and number of iterations); and a test set of 150 images and masks as independent data for evaluating the model's final segmentation performance. Images 001-700, 701-850, and 851-1000 were randomly selected from the preprocessed dataset (numbered 001-1000) as the training set, 701-850 as the validation set, and 851-1000 as the test set. These three sets of data do not overlap and cover polyp samples of different morphologies (such as small polyps and polyps with blurred boundaries). The proportion of small polyps (diameter <5mm) in each of the training, validation, and test sets remained around 30%, consistent with the overall distribution of the original dataset, ensuring that the model's ability to recognize small targets can be effectively validated.

[0019] S102, based on the constructed dataset, trains a colon polyp segmentation model using a hybrid architecture that combines CNN and Transformer.

[0020] In one implementation, a hybrid network architecture integrating CNN and Transformer is constructed, including an encoder that adopts a ResUNet-based structure to extract multi-scale features (low-level to high-level semantic information) layer by layer; a hybrid encoding module that introduces a HybridCNN-Transformer module in the deepest layer of the encoder, where the CNN branch extracts local texture features and the Transformer branch extracts global contextual features, integrating the two types of features through a gating fusion mechanism; an edge guidance module that is shallowly embedded in the encoder, using dilated convolutions with dilation rates of 1, 3, 5, and 7 to extract edge features and fuse them into the backbone features; a feature fusion module that includes a dual attention module (channel forward / backward attention, spatial attention) and a spatial pyramid pooling module (1×1, 2×2, 4×4, and 8×8 pooling scales); and a decoder that adopts a UNet++-style nested dense skip connection structure, fusing multi-source features and upsampling to restore resolution. Specifically, the encoder extracts four layers of feature maps through four ResBlocks (with resolutions of 128×128, 64×64, 32×32, and 16×16, respectively). The deepest layer (16×16) features are fed into the mixing module. The CNN branch uses 3×3 convolutions to extract local textures, and the Transformer branch divides the features into 8×8 image blocks (k=2) and introduces positional encoding. Global dependencies are modeled through multi-head self-attention.

[0021] The edge guidance module is fused with the main features, and the attention mechanism strengthens the weight of high-response regions (such as polyp boundaries) in the edge probability map. A dual attention module enhances the channel response of salient regions (such as the main body of the polyp) through positive channel attention, and enhances low-response regions (such as small polyps) through global max pooling to generate masks. Spatial attention focuses on key spatial locations. Multi-scale fusion is achieved through a spatial pyramid pooling module capturing context at different scales, and the decoder fuses shallow details and deep semantics through skip connections. For polyps with blurred boundaries, the edge probability map generated by the edge guidance module (activated by Sigmoid) is fused with mid-to-shallow features to improve the feature response of the boundary region. The dual attention module compensates for the channel and spatial weights of small polyp regions to avoid missed detections.

[0022] The training parameters are configured as follows: input image size is 256×256 (consistent with the preprocessed data size); batch size is 16; training epochs are 100; the optimizer is Adam, with an initial learning rate of 1e-4; the learning rate scheduling uses the ReduceLROnPlateau strategy (factor=0.5, patience=5), halving the learning rate when the validation set loss does not decrease for 5 consecutive epochs; regularization is enabled with gradient clipping (clip_norm=1.0) to prevent gradient explosion; the early stopping mechanism has a patience value of 10, stopping training and saving the optimal model when the validation set loss does not improve for 10 consecutive epochs. At epoch 30, the validation set Dice coefficient no longer improves, triggering learning rate scheduling, and the learning rate decreases from 1e-4 to 5e-5; if there is still no improvement at epoch 40, the early stopping mechanism is triggered, saving the model parameters from epoch 30.

[0023] The loss function is designed using a weighted sum of the binary cross-entropy (BCE) loss and the Dice loss as the joint loss function. The BCE loss quantifies pixel-level classification error, focusing on distinguishing between the foreground (polyp) and background; the Dice loss measures the overlap between the predicted mask and the ground truth mask, mitigating class imbalance (low proportion of polyp regions). Example: For small polyp samples, the Dice loss can amplify the error caused by missed detections, prompting the model to pay more attention to small target regions.

[0024] Training images are input into the model in batches. The encoder extracts features layer by layer, and after processing by the mixing module, edge guidance module, and feature fusion module, the decoder outputs a segmentation mask. The validation set is used in each round to evaluate the model performance (Dice coefficient, HD95, etc.) and monitor overfitting. After each batch of 16 images is input, the model generates 16 256×256 segmentation masks, which are then used to calculate the joint loss with the ground truth mask.

[0025] The gradient is calculated based on the loss function, and the model parameters (convolution kernel weights, attention weights, etc.) are updated through backpropagation using the Adam optimizer. After each training round, the model is validated on a validation set, and the parameters of the model with the best validation performance are saved. Example: After the 5th round of training, the model's Dice coefficient on the validation set is 0.82, which improves to 0.85 in the 10th round. The parameters of the 10th round are saved as the current optimal model.

[0026] After training, the model with the best performance on the validation set (e.g., highest Dice coefficient, lowest HD95) is selected as the final training model. Model parameters are saved in .pt (PyTorch) or .h5 (TensorFlow) format, including network weights, optimizer status, and other information. The final saved model achieved a Dice coefficient of 0.892 and an HD95 of 12.2 on the validation set, outperforming models from other rounds.

[0027] like Figure 2 As shown, the visualization comparison of colon polyp segmentation models is presented from left to right: Original (original image): the real scene containing polyps under colonoscopy, which is the basis for model input and preserves the original visual information of polyps and the intestinal environment; Label (realistic mask): the reference standard with precise white annotation of polyp areas, serving as the "ground truth" for model training and performance evaluation; EDTrans, U-Net, U-Net++, SwinUNet, TransUNet: the prediction results of different segmentation models, with the segmentation prediction of polyps shown in red areas. By comparing with the Label, the performance of each model in identifying polyp locations, completely delineating boundaries, and distinguishing polyps from the background can be intuitively evaluated, helping to analyze the merits of the models.

[0028] S103, the preprocessed digestive endoscopy image is input into the encoder, and multi-scale feature maps are extracted layer by layer. A hybrid CNN-Transformer module is introduced into the deepest multi-scale feature map of the encoder. Local texture features are extracted through the CNN branch and global context features are extracted through the Transformer branch. Based on the adaptive weight adjustment gating function, the fusion ratio of local and global features is optimized in real time according to the dynamic characteristics of the polyp region, so as to realize a differentiated dynamic gating fusion mechanism that is deeply bound to the polyp morphological features.

[0029] In one implementation, the preprocessed gastrointestinal endoscopy image is processed and fed into a ResUNet-based encoder to extract multi-scale feature maps layer by layer, generating a feature map set covering low- to high-level semantic information. The preprocessed gastrointestinal endoscopy image (256×256 pixels) is processed and input into the ResUNet-based encoder, which extracts features layer by layer through multiple ResBlocks (residual blocks). The specific process is as follows: Shallow feature extraction (low-level semantic information): The first two ResBlock layers of the encoder primarily capture low-level semantic information of the image. ResBlock 1: Through 3×3 convolution and downsampling operations, it outputs a feature map with a resolution of 128×128, focusing on extracting detailed information such as edges, textures, and color gradients of the polyp region, including subtle wrinkles on the polyp surface and initial boundary differences with the surrounding mucosa. ResBlock 2: Further downsampling to 64×64, the feature map contains more abstract local shape features, such as local contour fragments of the polyp and surface unevenness distribution, etc., representing mid-level semantic information.

[0030] Deep feature extraction (high-level semantic information): The last two ResBlock layers of the encoder focus on capturing global context and high-level semantic information. The third ResBlock layer: downsampled to 32×32, the feature map incorporates regional correlation information, enabling preliminary differentiation between polyps and complex backgrounds (such as intestinal folds, bubbles, etc.), reflecting the spatial distribution characteristics of polyps within local regions. The fourth ResBlock layer (the deepest layer): downsampled to 16×16, the feature map contains global semantic information, representing the long-distance spatial relationship between the polyp as a whole and surrounding tissues, such as the polyp's position in the entire intestinal field of view and its relative distribution with adjacent structures (such as blood vessels).

[0031] After four layers of ResBlock processing, a multi-scale feature map set consisting of 128×128, 64×64, 32×32, and 16×16 maps is generated, covering complete semantic information from low to high levels. These feature maps preserve both shallow details and textures and contain deep global context, providing multi-level feature inputs for subsequent hybrid encoding, edge guidance, and other modules. Furthermore, they perfectly match the multi-scale feature requirements of subsequent modules (e.g., the edge guidance module relies on shallow features, and the hybrid encoding module uses the deepest features), without any logical conflicts.

[0032] The multi-scale feature maps output by the encoder undergo hierarchical filtering, selecting the deepest feature map as the input feature for hybrid encoding. Based on the semantic complexity and global information representation capability of the feature maps, the deepest (16×16) feature map is selected as the input feature for hybrid encoding. From the four feature layers of 128×128, 64×64, 32×32, and 16×16, the 16×16 feature map is selected for the hybrid encoding process because it contains the richest global contextual information.

[0033] The deepest feature map is processed through branching. A CNN branch extracts local texture features to generate a local feature map. Simultaneously, a Transformer branch divides the image into k×k convolutions and introduces positional encoding. A multi-head self-attention mechanism is used to extract global contextual features, generating a global feature map. The CNN branch uses 3×3 convolutions (stride 1, padding 1) to process the 16×16 deepest feature map. Two convolutional layers enhance local detail representation: the first 3×3 convolution focuses on the microstructure of the polyp surface, such as the direction of wrinkles and the gray-level gradient changes in the boundary region, enhancing the correlation between local pixels; the second 3×3 convolution further integrates local features, highlighting the local differences between the polyp and the surrounding mucosa (such as color depth and texture density), generating a 16×16 local feature map. This branch, by maintaining the feature map size (16×16), accurately preserves local information such as polyp edges and surface details, providing fine-grained feature support for subsequent boundary optimization, complementing the edge guidance module's need for enhancing boundary details.

[0034] The Transformer branch models global dependencies through the following steps: image block partitioning, using a 2×2 convolution with k=2 to divide the 16×16 feature map into 8×8 2×2 image blocks, each block is flattened into a fixed-dimensional feature vector (e.g., 256-dimensional), achieving feature serialization; and the introduction of positional encoding, adding a sinusoidal positional code to each image block, which contains the coordinate information (x / y axis position) of the image block in the original feature map, ensuring that the Transformer preserves spatial positional features when modeling global relationships.

[0035] Multi-head self-attention computation employs an 8-head self-attention mechanism to globally model the feature vectors of 8×8 image patches. Each attention head focuses on global relationships at different scales (such as the spatial distance between the polyp and the left intestinal wall, and the right air bubble), calculating attention weights between image patches through matrix operations to enhance the feature responses of associated regions, ultimately generating a 16×16 global feature map. This branch compensates for the shortcomings of CNN in modeling global context by capturing long-distance dependencies across regions (such as the location association between the polyp and distant blood vessels, and the distribution of the polyp in the entire intestinal field of view), complementing the local features of the CNN branch.

[0036] For an endoscopic image containing a pedunculated polyp, in the local feature map of the CNN branch, the connection edge between the polyp stalk and the main body, as well as the texture gradient of the surface folds, are significantly enhanced, with local response values ​​higher than the surrounding areas. In the global feature map of the Transformer branch, the spatial distribution relationship between the polyp and the surrounding intestinal folds, and the distance information from the image edges are emphasized. High attention weights are used to associate the polyp with distant landmark structures (such as intestinal bends), providing a global reference for polyp localization in complex backgrounds. The two branches focus on local details and global relationships, respectively, providing differentiated inputs for the subsequent gating fusion mechanism, which is consistent with the model's design logic of co-modeling "local-global" features.

[0037] Local and global feature maps are fused. An adaptive weight adjustment gating function optimizes the fusion ratio of local and global features in real time based on the dynamic characteristics of the polyp region. This achieves a differentiated dynamic gating fusion mechanism deeply bound to the polyp morphological features, integrating the two types of features to generate a fused feature map. The input to the gating function consists of two types of dynamic characteristics of the polyp region, both calculated based on the original image and corresponding mask: variance is calculated using the edge probability map (activated by Sigmoid) generated by the edge guidance module. Higher variance values ​​indicate greater pixel response fluctuations in the edge region and more blurred polyp boundaries (e.g., diffuse boundaries caused by inflammation); lower variance values ​​indicate clearer boundaries (e.g., the defined outline of a pedunculated polyp). The pixel percentage in the polyp segmentation mask (number of polyp pixels / total number of image pixels) is statistically analyzed. A percentage <5% is defined as a small polyp, 5%-20% as a medium-sized polyp, and >20% as a large polyp. The quantification result directly reflects the spatial proportion of the polyp in the image.

[0038] The gate function is constructed based on a multilayer perceptron (MLP) and includes the following steps: concatenating the boundary ambiguity (normalized to [0,1]) and the size quantization value (normalized to [0,1]) into a two-dimensional vector, which serves as the input to the MLP; the MLP outputs local feature weights through a two-layer fully connected network (with ReLU activation in the hidden layers). Global feature weights Through 1- The calculation ensures that the weights sum to 1. The MLP learns the mapping relationship between polyp characteristics and weights through training. For small polyps with blurred boundaries (high blurriness and low size ratio), it outputs higher global weights to strengthen contextual guidance. For large polyps with clear boundaries (low blurriness and high size ratio), it outputs higher local weights to highlight details and textures.

[0039] The fused feature map is generated by weighted summation: ,in, To fuse feature maps, ∈[0,1] represents a scalar, channel vector, or spatial weight map, used to dynamically adjust the local feature map. ) and global feature map ( The fusion ratio is adjusted to generate a 16×16 fused feature map, which preserves local texture details, integrates global context information, and has the same size as the deepest feature of the encoder, providing compatible input for subsequent dimension restoration processing.

[0040] For an endoscopic image containing a small polyp with blurred boundaries (mask pixel ratio 3%, edge probability map variance 0.8): the gating function input is [0.8, 0.03] (after blur and size normalization), and the MLP output is... =0.3、 =0.7; During fusion, global features (such as the spatial relationship between polyps and surrounding blood vessels) are given higher weights. The global context compensates for the lack of local features at ambiguous boundaries, helping the model to accurately locate polyp regions. If the polyp boundary is clear and the size is large (accounting for 15%, variance 0.2), the gate function output is... =0.6、 =0.4, emphasizing local texture features to refine the contours. This fusion mechanism is consistent with the overall model design logic: by dynamically balancing local details and global context, it addresses the segmentation difficulties of polyps of different shapes, laying the foundation for accurate segmentation by the subsequent decoder.

[0041] The fused feature map undergoes dimensionality reduction processing. Transposed convolutions are used to restore the feature dimensions to their original dimensions, generating the final feature map of the deepest layer of the encoder. Dimensional reduction processing is performed on the fused feature map, with 1×1 convolutions used for channel alignment. Feature upsampling uses bilinear interpolation or transposed convolutions with a stride greater than 1 to reduce the feature channel dimensions from 512 to 256 (consistent with the original feature dimensions of the deepest layer of the encoder), generating the final feature map of the deepest layer of the encoder. After transposed convolution, the fused feature map retains both local texture details and global contextual information, and the number of channels matches the deep features of the encoder, laying the foundation for the decoder input.

[0042] S104 introduces an edge guidance module in the shallow multi-scale feature map of the encoder. It uses dilated convolutions with dilation rates of 1, 3, 5, and 7 to extract multi-scale edge responses, activates them with the Sigmoid function to generate an edge probability map, and then fuses them into the backbone feature map through an attention mechanism.

[0043] In one implementation, the shallow to medium-scale feature maps output by the encoder are filtered to determine the input feature maps for the edge guidance module, where each layer of feature maps corresponds to different levels of semantic information. The multi-scale feature maps (e.g., 128×128, 64×64, 32×32) output by the encoder are filtered hierarchically, selecting the shallow to medium-scale feature maps as the input to the edge guidance module. The filtering is based on the semantic level of the feature maps: shallow feature maps (128×128): retain rich low-level semantic information such as edges and textures, suitable for capturing subtle boundaries; medium-scale feature maps (64×64): contain medium-level semantic information such as local shapes, which can assist in locating polyp contour fragments. From the feature maps output by the encoder, the 128×128 (layer 1) and 64×64 (layer 2) feature maps are filtered, while the deep (32×32, 16×16) feature maps are excluded (because the semantics of deep features are abstract, edge details are weakened).

[0044] Edge feature extraction was performed on selected mid-to-shallow multi-scale feature maps. Multi-scale edge responses were extracted using dilated convolutional layers with dilation rates of 1, 3, 5, and 7 to generate multi-scale edge feature maps. For the selected mid-to-shallow feature maps, edge responses were extracted using four parallel dilated convolutional layers (dilation rates of 1, 3, 5, and 7): Dilation rate 1: captures fine edges within a 1×1 local area (e.g., the boundaries of tiny folds on the polyp surface); Dilation rate 3: covers a 3×3 area, extracting medium-scale edges (e.g., the connection boundary between the polyp body and its stalk); Dilation rates 5 and 7: cover 5×5 and 7×7 areas respectively, capturing large-scale edges (e.g., the blurred boundaries between the polyp and the surrounding mucosa). After dilation rate 7, the 128×128 feature map effectively captured the blurred boundaries of larger-diameter polyps, generating the corresponding edge feature map.

[0045] Multi-scale edge feature maps were activated, and edge probability maps were generated by calculating the Sigmoid function to quantify the response intensity of edge regions. After concatenating the multi-scale edge feature maps, the Sigmoid function was applied to generate edge probability maps with pixel values ​​in the range [0,1]. Probability values ​​close to 1 correspond to high-response edge regions (such as clear polyp boundaries); probability values ​​close to 0 correspond to non-edge regions (such as uniform mucosal surfaces). Clear outlines of pedunculated polyps exhibited continuous high probability values ​​(0.8-1.0) in the edge probability map, while blurred boundaries caused by inflammation showed moderate probability values ​​(0.3-0.6).

[0046] Attention fusion processing is applied to the edge probability map and the corresponding backbone feature map, strengthening the weight of high-response regions in the edge probability map through an attention mechanism. This mechanism involves calculating global statistics (such as mean and variance) of the edge probability map to generate attention weights; high-response edge regions (probability value > 0.7) are assigned higher weights, while the weights of low-response regions are suppressed. In the edge probability map, the weight of high-response regions of polyp contours is increased by 1.2 times, while the weight of non-edge regions is reduced to 0.5 times, highlighting effective edge information.

[0047] The attention-weighted edge probability map and the backbone feature map are integrated to generate an enhanced backbone feature map that incorporates edge information, improving the boundary representation capability of the feature map. The attention-weighted edge probability map is then pixel-by-pixel added to the corresponding mid-to-shallow backbone feature map to achieve the fusion of edge information and backbone features: the fused feature map retains the semantic information of the backbone features while strengthening the feature response of the edge regions; the output is an enhanced feature map with the same size as the input backbone feature map (e.g., 128×128, 64×64). After fusing the 128×128 backbone feature map with the weighted edge probability map, the feature response intensity of the polyp edge is increased by 30%, providing more accurate feature support for the subsequent boundary reconstruction of the decoder.

[0048] S105 introduces a dual attention module and a spatial pyramid pooling module in the feature fusion stage. The dual attention module includes channel forward attention, channel reverse attention, and spatial attention sub-modules to enhance the model's sensitivity to salient regions and low-response regions.

[0049] In one implementation, the feature map output by the encoder undergoes a fusion stage preprocessing and is then fed into the feature fusion module to provide input features for the dual attention mechanism and the spatial pyramid pooling module. Feature maps at different levels exhibit resolution differences due to the encoder's downsampling operation, requiring upsampling or downsampling to adjust them to the same size (64×64 in this application): For downsampling, a 2×2 max pooling method is used for downsampling the high-resolution feature map (128×128 edge enhancement feature map), reducing resolution while preserving key features; for upsampling, a bilinear interpolation method is used to upsample the low-resolution feature map (32×32 hybrid coding feature map) to 64×64, restoring spatial details. The 128×128 edge enhancement feature map becomes 64×64 after max pooling downsampling, and the 32×32 hybrid coding feature map is upsampled to 64×64 through bilinear interpolation; both have the same spatial size and can be used for subsequent fusion operations.

[0050] The number of channels in feature maps from different sources varies (e.g., edge enhancement feature maps have 128 channels, while hybrid coding feature maps have 256 channels). Channel standardization is required using 1×1 convolutions: applying a 1×1 convolution kernel to each feature map uniformly adjusts the number of channels to 256. While compressing or expanding the number of channels, the 1×1 convolution integrates features through cross-channel information interaction, enhancing semantic consistency. The 128-channel edge enhancement feature map expands its dimension through a 1×1 convolution (128→256 channels), while the 256-channel hybrid coding feature map maintains its dimension through a 1×1 convolution (256→256 channels). Ultimately, all feature maps have 256 channels, eliminating channel differences.

[0051] All feature maps after resolution alignment and channel unification are concatenated along the channel dimension to form the input features for the fusion module. The concatenated feature map contains multi-source information (edge ​​details, local textures, global context, etc.) and has a dimension of 64×64×(256×N), where N is the number of feature maps. Standardized input ensures that the dual attention module can perform unified channel and spatial attention calculations on multi-source features, and the spatial pyramid pooling module can stably capture multi-scale context. The 64×64×256 edge enhancement features are concatenated with the 64×64×256 hybrid encoded features to form a 64×64×512 input feature, which is fed into the fusion module to provide a complete multi-source feature foundation for subsequent processing.

[0052] The input features are processed with channel-positive attention. A channel-positive attention submodule enhances the channel feature responses of salient regions, generating a channel-positive enhanced feature map. The submodule strengthens the channel responses of salient regions (such as the polyp itself): global average pooling is applied to the input feature map to generate channel-level statistical vectors; channel attention weights are calculated using a two-layer MLP (with ReLU activation in the hidden layers) to highlight high-response channels (such as those representing polyp texture); these weights are multiplied channel-by-channel with the input features to generate the channel-positive enhanced feature map. The channel weights corresponding to the polyp region are increased by 1.5 times, while the channel weights for the background region are reduced to 0.6 times, resulting in a more significant response in the enhanced feature map for the polyp region.

[0053] The input features are processed using channel-wise inverse attention. The channel-wise inverse attention submodule uses global max pooling to generate a significant inverse channel mask, enhancing the channel feature representation of low-response regions and generating a channel-wise enhanced feature map. As a core component of the dual attention mechanism, the channel-wise inverse attention submodule specifically compensates for channel features in low-response regions (such as small polyps or weak boundaries), complementing the channel-wise forward attention. The specific process is as follows: Global max pooling captures extreme features. It performs a global max pooling operation on the standardized feature map (64×64×256) of the input feature fusion module: it iterates through all pixels in each channel, extracting the maximum value as the feature representative for that channel, generating a 1×1×256 channel-level vector. Unlike global average pooling used in channel positive attention, global max pooling is more sensitive to extreme pixel values ​​in a channel (such as weak responses in small polyp regions or local gradient changes at weak boundaries). It can accurately locate channels corresponding to low-response regions "submerged" by average pooling. Small polyp regions have low response values ​​in the original feature map (e.g., pixel values ​​0.2-0.3, background region 0.1-0.4). Global average pooling will average these values ​​with the background to a similar value, while global max pooling can capture the 0.3 peak value in this region, marking it as a potentially valuable channel.

[0054] The 1×1×256 vector obtained from global max pooling is input into a multilayer perceptron (MLP) to generate a reverse mask that complements the channel-wise positive attention. The MLP consists of one hidden layer (using ReLU activation) and one output layer (using Sigmoid activation), outputting a 1×1×256 mask vector, with each element corresponding to the weight of a channel. The mask generation logic is the opposite of the channel-wise positive attention: channels with higher global max pooling values ​​(corresponding to latent features in low-response regions) are assigned high weights, while channels with lower global max pooling values ​​(corresponding to background or strong-response regions) are assigned low weights, achieving "reverse compensation." The global max pooling value of the channel containing the small polyp is 0.3 (higher than 0.1 for the background channel), and the MLP outputs a mask weight of 0.8 for this channel, while the main channel of the strong-response polyp is output with a weight of 0.2, forming a distribution that complements the positive attention (high weight in the main channel).

[0055] The inverse mask is multiplied channel-by-channel with the input feature map to generate a channel-wise inverse enhancement feature map: the feature response of the channel corresponding to the low response region is amplified under the mask (e.g., the original response is multiplied by 1.3); the output feature map size is the same as the input (64×64×256), preserving the spatial structure while enhancing the channel representation of the low response region. After weighting by the inverse mask, the feature response of the channel where the small polyp region is located is increased from 0.3 to 0.39, significantly improving the distinguishability from the surrounding background; the channel response of the weak boundary region is increased from 0.25 to 0.32, providing a clearer feature basis for subsequent spatial attention focusing.

[0056] By accurately capturing the potential features of low-response regions through global max pooling, and using MLP to generate targeted inverse masks, channel-level enhancement of easily missed regions such as small polyps and weak boundaries is achieved. This forms a "positive-inverse synergy" mechanism with channel positive attention (enhancing salient regions), providing reliable channel feature support for cross-dimensional inverse enhancement (spatial attention).

[0057] A cross-dimensional fusion process is performed on the forward and reverse enhanced feature maps of the channels to construct a cross-dimensional reverse enhancement mechanism. In the spatial dimension, leveraging the spatial attention submodule and combining the channel reverse attention results, the key spatial locations of low-response regions containing small polyps and weak boundaries are focused, and the features of these regions are simultaneously enhanced to generate spatially enhanced feature maps. The "channel-space" cross-dimensional enhancement mechanism, as the core component of the dual attention module, aims to combine the advantages of channel forward / reverse enhanced features to further focus on the spatial locations of low-response regions such as small polyps and weak boundaries, achieving precise feature enhancement. The specific process is as follows: The forward-enhanced feature map (64×64×256) and the reverse-enhanced feature map (64×64×256) are concatenated along the channel dimension to form a fused feature map of 64×64×512. This concatenated feature map retains the strong channel response (from forward enhancement) in salient regions and the compensation channel information (from reverse enhancement) in low-response regions, providing a comprehensive basis for precise spatial localization. A 3×3 convolution (stride 1, padding 1) is applied to the concatenated 512-channel feature map, integrating local spatial information through nonlinear transformation to generate a single-channel spatial attention map (64×64×1). The receptive field design of the 3×3 convolution can effectively capture the local correlation between pixels, making it suitable for locating the spatial position of local regions such as small polyps and weak boundaries. The concatenated feature map contains both the high-response channel information of the polyp body and the compensation channel information of the small polyp region. The 3×3 convolution is calculated through a sliding window to generate an initial high-response signal at the spatial location of the small polyp.

[0058] After the spatial attention map is generated, the weights are further calibrated by combining the output (inverse mask) of the channel inverse attention submodule: the inverse mask (1×1×256) of the channel inverse attention has marked the channels corresponding to low response regions. After its spatial dimension is broadcast to 64×64×256, it is associated with the inverse enhancement channels in the splicing features. The spatial position corresponding to the high-weight channels of the inverse mask in the spatial attention map is strengthened through a gating mechanism, that is, higher weights are assigned to the spatial coordinates of small polyps and weak boundaries, while the weights of non-target regions are suppressed. The channel corresponding to the small polyp marked in the channel inverse mask (weight 1.3) has its spatial position identified by the spatial attention map, and the weight of this position is increased from the initial 0.6 to 0.9, achieving targeted enhancement.

[0059] The adjusted spatial attention map (64×64×1) is multiplied pixel-by-pixel with the stitched feature map (64×64×512) to generate a spatially enhanced feature map (64×64×512). The spatial locations of low-response regions such as small polyps and weak boundaries are significantly enhanced due to their high weights, resulting in improved feature strength. The spatial locations of prominent regions such as the main polyp remain moderately responsive to avoid feature imbalance caused by over-enhancement. The output feature map size is consistent with the input, providing feature input containing accurate spatial positioning information for the subsequent spatial pyramid pooling module. The small polyp region with a diameter <5mm has a weight of 0.9 in the spatial attention map. After multiplying with the original response (0.5) of this region in the stitched feature map, the feature strength is increased to 0.45, which is about 40% higher than before enhancement (0.32). The distinction between the polyp region and the surrounding background is significantly improved, laying the foundation for subsequent multi-scale context aggregation.

[0060] The cross-dimensional enhancement mechanism achieves a dual enhancement of "channel guidance and spatial focus" by synergistically using channel back attention to mark the channel features of low-response regions and spatial attention to accurately locate their spatial positions.

[0061] Multi-scale context aggregation processing is performed on the spatial augmented feature map. Multi-scale contextual information is captured through adaptive pooling layers at 1×1, 2×2, 4×4, and 8×8 scales in the spatial pyramid pooling module, generating a multi-scale fused feature map. The spatial pyramid pooling module captures context through adaptive pooling layers at four scales (1×1, 2×2, 4×4, and 8×8): 1×1 pooling: extracts global semantics (e.g., the overall location of the polyp in the image); 2×2 / 4×4 pooling: captures mid-scale context (e.g., the relationship between the polyp and surrounding folds); 8×8 pooling: preserves local details (e.g., the surface texture distribution of the polyp).

[0062] The pooling results are concatenated after unifying the channels through 1×1 convolutions to generate a multi-scale fused feature map. An example is shown below: an 8×8 pooling layer preserves the local contours of small polyps, while a 1×1 pooling layer provides their global location information in the intestine; the fused features take into account both details and context.

[0063] The positive and negative channel enhancement features, spatial enhancement features, and multi-scale fusion features are integrated and processed to form a feature map enhanced by dual attention and multi-scale pooling, improving the model's sensitivity to salient and low-response regions. Four types of features are stitched together according to the channel dimension to aggregate multi-dimensional information: Positive channel enhancement features (64×64×256): Focusing on salient regions such as the polyp body, highlighting core lesion features by strengthening high-response channels; Negative channel enhancement features (64×64×256): For low-response regions such as small polyps and weak boundaries, compensating for channel response through a negative mask generated by global max pooling to avoid feature loss; Spatial enhancement features (64×64×512): Combining the results of negative channel attention, strengthening the spatial location features of low-response regions through a spatial attention map to improve target localization accuracy; Multi-scale fusion features (64×64×512): Generated through spatial pyramid pooling (1×1, 2×2, 4×4, 8×8 scales), covering multi-scale information from global context to local details, adapting to polyp features of different sizes. The stitched feature map has a dimension of 64×64×(256+256+512+512)=64×64×1536, fully preserving the unique information of various features and providing a rich foundation for subsequent integration. In the stitched features, the strong channel response of the polyp body, the compensating channel information of small polyps, the spatial localization features of the boundary, and multi-scale contextual information are preserved in parallel, forming a comprehensive feature set.

[0064] A 3×3 convolution (stride 1, padding 1) is applied to the stitched 1536-channel feature map to achieve feature fusion and optimization. Cross-channel information interaction: The 3×3 convolution kernel captures the correlation between features from different sources, such as the spatial relationship between the polyp and surrounding smaller polyps, and the matching relationship between boundary features and the global context. Semantic aggregation and dimensionality compression: The number of convolution kernels is set to 256, compressing the 1536 channels to 256 channels, reducing computation while aggregating multi-source information into unified semantic features. Spatial detail preservation: The convolution operation maintains the same resolution (64×64), ensuring the integrity of spatial location information and meeting the decoder's requirement for detail restoration. The 3×3 convolution learns weights to focus on associating the channel-inverse features of smaller polyps with the corresponding spatial location enhancement features, while weakening irrelevant background interference, generating more discriminative integrated features.

[0065] The final output is a 64×64×256 integrated feature map with the following core characteristics: multi-information collaboration, simultaneously preserving strong responses in salient regions, compensation information in low-response regions, and multi-scale context, achieving comprehensive coverage of "global-local-boundary"; semantic consistency, after convolutional integration, semantic conflicts between features from different sources are resolved, and the feature distribution is more in line with the needs of polyp segmentation tasks (e.g., significantly improving the distinction between polyps and background); decoder compatibility, the resolution (64×64) and the number of channels (256) match the decoder input requirements, and can be gradually upsampled to restore the resolution to 256×256 through UNet++-style nested dense skip connections and shallow feature fusion, providing a high-quality feature foundation for the final segmentation mask generation. In the integrated feature map, the feature response of the main polyp region is improved by 15%, the response of the small polyp region is improved by 30%, the response of the boundary region is improved by 22%, and the overall Dice coefficient is improved by 2.3% compared with a single feature (e.g., only channel positive enhancement feature), effectively improving the segmentation accuracy.

[0066] S106 utilizes a UNet++-style decoder structure, fusing shallow skip connection features, edge enhancement features, and multi-scale contextual features from the encoder to progressively upsample and restore resolution, generating a segmentation mask image of the target colon polyp.

[0067] In one implementation, the shallow skip connection features output by the encoder, the edge enhancement features generated by the edge guidance module, and the multi-scale context features generated by the spatial pyramid pooling module are collected and processed to determine the input feature set of the decoder, where each type of feature corresponds to information in different dimensions. The multi-source features output by the encoder are collected and organized to form the input feature set of the decoder. Each type of feature provides information in different dimensions: shallow skip connection features, from the first two layers of the encoder (128×128, 64×64), contain rich low-level semantic information such as edges and textures, used to reconstruct polyp details; edge enhancement features, generated by the edge guidance module (128×128, 64×64), enhance boundary responses through attention fusion, focusing on the polyp contour; multi-scale context features, output by the spatial pyramid pooling module (64×64), cover global and local contexts from 1×1 to 8×8 scales, adapting to polyps of different sizes. Collecting the 128×128 shallow features, 128×128 edge enhancement features, and 64×64 multi-scale context features forms a multi-dimensional input set containing details, boundaries, and context.

[0068] The input feature set undergoes hierarchical fusion processing. A UNet++-style nested dense skip connection structure enables cross-layer interaction between features, generating preliminary fused features. This cross-layer interaction is achieved through the UNet++-style nested dense skip connection structure: each decoder feature establishes skip connections with all shallow encoder features; for example, a 64×64 decoder feature is simultaneously fused with 128×128 and 64×64 encoder features. Feature fusion is achieved through 3×3 convolutions and BatchNorm, strengthening the correlation between information from different levels (e.g., associating global context with local details). The output is a feature map matching the current decoder level (e.g., 64×64), providing a foundation for upsampling. For example, a 64×64 decoder feature is fused with 128×128 shallow texture features and 64×64 edge features through dense connections, generating preliminary fused features that simultaneously contain boundaries and details.

[0069] The initial fused features are upsampled to gradually restore the feature map resolution, while continuously integrating feature information from different levels to generate a high-resolution fused feature map. The initial fused features are upsampled in a stepwise manner to gradually restore the original image resolution (256×256): upsampling uses bilinear interpolation or transposed convolution, doubling the feature map resolution at each step (e.g., 64×64→128×128→256×256); during the upsampling process, encoder features from the corresponding level are continuously fused (e.g., 128×128 decoder features fused with 128×128 edge enhancement features); finally, a 256×256 high-resolution fused feature map is output, preserving multi-scale information and spatial details. The 64×64 initial fused features are upsampled to 128×128 via transposed convolution, fused with 128×128 shallow skip connection features, and then upsampled again to 256×256 to complete the resolution restoration.

[0070] The high-resolution fused feature map is processed to generate a segmentation mask. The feature map is transformed into a pixel-level probability distribution of polyp regions, generating an initial segmentation mask. Pixel-level classification is performed on the high-resolution fused feature map to generate the initial segmentation mask: a 1×1 convolution maps the 256-channel feature map to a 1-channel probability map (pixel value ∈ [0,1]), representing the probability that each pixel is a polyp; a default threshold of 0.5 is used, and pixels with a probability ≥ 0.5 are marked as polyps (1), otherwise as background (0); a 256×256 binary mask is generated to initially separate the polyp and background regions. The polyp region exhibits a high response (0.7-1.0) in the probability map, forming a continuous white region after thresholding, while the background is black, generating the initial segmentation mask.

[0071] The initial segmentation mask image is refined to enhance the distinction between the polyp region and the background, ultimately generating a segmentation mask image of the target colon polyp, achieving accurate segmentation of the polyp region. The initial mask is optimized to further enhance the distinction between the polyp and the background: morphological operations, such as erosion and dilation, remove small-area noise (e.g., isolated noise points) and fill holes in the polyp region; boundary smoothing is achieved by applying Gaussian blur and thresholding to the mask edges, reducing jagged contours and making the boundaries more continuous; finally, a colon polyp segmentation mask with the same size as the original image (256×256) is generated, ensuring the polyp region is complete and the boundaries are clear. Three-pixel noise points in the initial mask are removed through erosion, and the polyp boundaries are smoothed, resulting in a 1.2 reduction in the HD95 index, making it closer to the true mask contour.

[0072] S107. The target colon polyp segmentation mask image is processed. The weighted sum of binary cross-entropy loss and Dice loss is used as the joint loss function. The Adam optimizer and dynamic learning rate scheduling strategy are used to train the model and generate colon polyp segmentation results.

[0073] In one implementation, the target colon polyp segmentation mask is compared with the true segmentation mask in the dataset to extract pixel-level difference information and generate the initial loss calculation basis. The target colon polyp segmentation mask (256×256) output by the model is compared pixel-by-pixel with the true segmentation mask (256×256) in the dataset to extract difference information: difference types include "false positives" (background pixels misclassified as polyps by the model) and "false negatives" (polyp pixels missed by the model). The mask image is converted into a binary matrix (polyp=1, background=0), and a difference matrix is ​​obtained through pixel-by-pixel subtraction (1 indicates a prediction error, 0 indicates a correct prediction). The difference matrix serves as the original input for subsequent loss calculation, retaining all pixel-level error details. A pixel value of 1 for a small polyp region in the true mask, while the corresponding region in the target mask is 0, is marked as a false negative difference; a region in the background misclassified as 1 by the target mask is marked as a false positive difference.

[0074] The extracted discrepancy information is subjected to loss quantification processing. A weighted sum of binary cross-entropy loss and Dice loss is used as the joint loss function to calculate the model prediction error and generate a comprehensive loss value. The weighted sum of binary cross-entropy (BCE) loss and Dice loss is used as the joint loss function to quantify the model prediction error: the binary cross-entropy loss focuses on pixel-level classification error, and the formula is as follows: ,in These are the actual mask pixel values. To improve the model's probability prediction, the penalty for difficult areas such as small polyps is strengthened.

[0075] Dice loss measures the degree of overlap between the prediction and the true mask, and the formula is: This mitigates the class imbalance issue (low percentage of polyp pixels). The overall loss value is Loss = 0.5. BCE+0.5 Dice balances the contributions of the two types of losses. For the scenario of missed detection of small polyps, the Dice loss increases significantly due to low overlap (e.g., 0.6), while the BCE loss increases due to the large number of false negative pixels (e.g., 0.3), with a combined loss value of 0.45.

[0076] The overall loss value is optimized by backpropagating the model parameters using the Adam optimizer with an initial learning rate of 1e-4, generating parameter adjustment gradients. The training process is optimized through dynamic learning rate adjustment and auxiliary mechanisms. Specifically, the ReduceLROnPlateau scheduler halves the learning rate (factor=0.5) when the validation set loss does not decrease for five consecutive rounds (patience=5) to avoid overfitting; a gradient clipping threshold (clip_norm=1.0) is set to prevent abnormal parameter updates caused by gradient explosion.

[0077] If the validation set loss does not improve for 10 consecutive rounds (patience value 10), training is stopped and the current best model is saved to avoid invalid iterations. By the 30th round of training, the validation set Dice coefficient no longer improves, and the learning rate decreases from 1e-4 to 5e-5; if there is still no improvement by the 40th round, the early stopping mechanism is triggered, and the model from the 30th round is saved.

[0078] The learning rate is dynamically scheduled using the ReduceLROnPlateau scheduler to adjust the learning rate based on changes in loss. Gradient pruning and early stopping mechanisms are also enabled to generate an optimized learning strategy. Iterative training follows a cycle of "loss calculation → parameter update → learning rate adjustment → performance validation": In each round, the joint loss (a weighted sum of binary cross-entropy loss and Dice loss) is calculated using the training set, and model parameters (such as convolutional kernel weights and attention module parameters) are updated via backpropagation using the Adam optimizer. After each training round, model performance is evaluated on the validation set (key metrics include Dice coefficient and HD95), and the model parameters for the current round are saved. During training rounds 1-50, as iterations progress, the overall loss on the training set gradually decreases from 0.52 to 0.15, while the Dice coefficient on the validation set increases from 0.78 to 0.87, indicating a continuous improvement in the model's segmentation accuracy for polyp regions.

[0079] The model is considered convergent when the following conditions are met: the validation set loss changes by less than 0.001 for five consecutive rounds, and the Dice coefficient remains stable above 0.88 without significant fluctuations; the HD95 score on the validation set remains stable below 13, indicating that the boundary segmentation accuracy has reached expectations. In rounds 70-74, the validation set loss stabilizes at 0.142 ± 0.0008, the Dice coefficient remains at 0.89, and the HD95 score is 12.5, meeting the convergence criteria.

[0080] The model parameters with the best performance on the validation set (highest Dice coefficient, lowest HD95) from all rounds were selected as the final model. The model parameters were saved in .pt format (PyTorch framework), containing information such as network weights and training configurations (e.g., learning rate, number of iterations) to ensure direct loading for inference later. The model in round 65 achieved a Dice coefficient of 0.892 and an HD95 of 12.2 on the validation set, outperforming other rounds, and was therefore selected as the final model and saved.

[0081] The model parameters are iteratively optimized, with repeated loss calculations, parameter updates, and learning rate adjustments until the model converges, generating a trained segmentation model that ultimately outputs accurate colon polyp segmentation results. Preprocessed images from the test set are input into the trained final model, and through collaborative processing by the encoder, feature fusion module, and decoder, corresponding colon polyp segmentation masks are generated. On an endoscopic image in the test set containing small polyps with blurred boundaries, the segmentation mask generated after model inference accurately covers the polyp region, improving boundary integrity by 20% compared to the initial model. The segmentation results for the test set are output, including pixel-level segmentation masks and quantization metrics (such as the Dice coefficient and HD95). Evaluation results show that the model achieves a Dice coefficient of 0.89 and an HD95 of 12.3 on the test set, outperforming existing mainstream methods (such as U-Net and TransUNet), validating the effectiveness of the proposed method.

[0082] The average Dice coefficient of the 150 images in the test set was 0.89, with the Dice coefficient of small polyp samples reaching 0.85, which is 3.2 percentage points higher than the comparison method, and the false negative rate was reduced by 15%. The final segmentation results meet the clinical requirements for polyp segmentation accuracy (especially for small polyps and blurred boundaries).

[0083] This application constructs a hybrid architecture integrating CNN and Transformer, incorporating an edge guidance module, a dual attention mechanism, and a multi-scale spatial pyramid pooling module. First, the preprocessed endoscopic image is input into a ResUNet-based encoder to extract multi-scale features. The deepest layer of the encoder introduces a hybrid CNN-Transformer module to fuse local and global features. In the middle and shallow layers, the edge guidance module extracts and fuses edge features using dilated convolutions with dilation rates of 1, 3, 5, and 7. The feature fusion stage employs a dual attention module, including channel forward / backward and spatial attention sub-modules, to enhance sensitivity to salient and low-response regions. A spatial pyramid pooling module is introduced to capture multi-scale context. The decoder uses a UNet++ architecture, fusing multiple types of features and upsampling the output segmentation mask. On the Kvasir-SEG dataset, the Dice coefficient reaches 0.892, and the HD95 coefficient is 12.20, outperforming mainstream methods such as U-Net. It can accurately segment polyps, especially improving the recognition ability of small polyps and blurred boundaries, demonstrating clinical application value.

[0084] In one implementation, such as Figure 2 As shown, this application also provides a hybrid CNN-Transformer colon polyp image segmentation device that combines edge guidance and dual attention mechanisms, comprising: The acquisition module 201 is used to acquire preprocessed digestive endoscopy image data and construct training, validation and test sets based on the Kvasir-SEG dataset, wherein the dataset contains the original images and corresponding colon polyp segmentation mask annotation information; Processing module 202 is used to train a colon polyp segmentation model based on the constructed dataset using a hybrid structure that integrates CNN and Transformer. Preprocessed digestive endoscopy images are input into the encoder, which extracts multi-scale feature maps layer by layer. A hybrid CNN-Transformer module is introduced into the deepest multi-scale feature map of the encoder. Local texture features are extracted through CNN branches, and global context features are extracted through Transformer branches. Based on an adaptive weight adjustment gating function, the fusion ratio of local and global features is optimized in real time according to the dynamic characteristics of the polyp region, achieving a differentiated dynamic gating fusion mechanism deeply bound to the polyp morphological features. An edge guidance module is introduced into the middle and shallow multi-scale feature maps of the encoder, using dilated convolutions with dilation rates of 1, 3, 5, and 7 to extract multi-scale features. Edge responses are activated by the Sigmoid function to generate edge probability maps, which are then fused to the backbone feature map through an attention mechanism. A dual attention module and a spatial pyramid pooling module are introduced during the feature fusion stage. The dual attention module includes channel forward attention, channel reverse attention, and spatial attention sub-modules to enhance the model's sensitivity to salient and low-response regions. A UNet++-style decoder structure is used to fuse shallow skip connection features from the encoder, edge enhancement features, and multi-scale contextual features, progressively upsampling to restore resolution and generate a segmentation mask for the target colon polyp. The target colon polyp segmentation mask is then processed using a weighted sum of binary cross-entropy loss and Dice loss as the joint loss function. The model is trained using the Adam optimizer and a dynamic learning rate scheduling strategy to generate the colon polyp segmentation result.

[0085] The computer-readable storage medium provided in the above embodiments of this application and the hybrid CNN-Transformer colon polyp image segmentation method combining edge guidance and dual attention mechanism provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0086] The various embodiments in this application are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for evaluating the hybrid CNN-Transformer colon polyp image segmentation method, electronic device, electronic device, and readable storage medium combining edge guidance and dual attention mechanisms are basically similar to the embodiments of the hybrid CNN-Transformer colon polyp image segmentation method combining edge guidance and dual attention mechanisms described above, and are therefore described simply. Relevant parts can be referred to in the description of the embodiments of the hybrid CNN-Transformer colon polyp image segmentation method combining edge guidance and dual attention mechanisms described above.

Claims

1. A hybrid CNN-Transformer colon polyp image segmentation method combining edge guidance and dual attention mechanism, characterized in that, The method comprises the following steps: acquiring preprocessed digestive endoscopy image data, and constructing a training set, a verification set and a test set based on a Kvasir-SEG data set, wherein the data set comprises original images and corresponding colon polyp segmentation mask annotation information; training a colon polyp segmentation model based on the constructed data set by using a hybrid structure combining CNN and Transformer; inputting the preprocessed digestive endoscopy image into an encoder, extracting multi-scale feature maps layer by layer, introducing a hybrid CNN-Transformer module in the multi-size feature map at the deepest layer of the encoder, extracting local texture features through the CNN branch, extracting global context features through the Transformer branch, adjusting the fusion proportion of local and global features in real time based on an adaptive weight adjustment gating function, and realizing a differentiated dynamic gating fusion mechanism deeply bound with polyp morphological features according to the dynamic characteristics of the polyp region; introducing an edge guiding module in the multi-scale feature map at the middle and shallow layers of the encoder, extracting multi-scale edge responses by using a dilated convolution with expansion rates of 1, 3, 5 and 7, generating an edge probability map through Sigmoid function activation, and fusing the edge probability map into the main feature map through an attention mechanism; introducing a double attention module and a spatial pyramid pooling module in the feature fusion stage, the double attention module comprising a channel forward attention, a channel reverse attention and a spatial attention submodule, and enhancing the sensitivity of the model to significant areas and low response areas; using a UNet++ style decoder structure to fuse the shallow layer jump connection features, edge enhanced features and multi-scale context features of the encoder, gradually restoring the resolution through upsampling, and generating a target colon polyp segmentation mask map; processing the target colon polyp segmentation mask map to generate a colon polyp segmentation result.

2. The method of claim 1, wherein, The method comprises the following steps: inputting the preprocessed digestive endoscopy image into an encoder, extracting multi-scale feature maps layer by layer, introducing a hybrid CNN-Transformer module in the multi-size feature map at the deepest layer of the encoder, extracting local texture features through the CNN branch, extracting global context features through the Transformer branch, adjusting the fusion proportion of local and global features in real time based on an adaptive weight adjustment gating function, and realizing a differentiated dynamic gating fusion mechanism deeply bound with polyp morphological features according to the dynamic characteristics of the polyp region, comprising: processing the preprocessed digestive endoscopy image and inputting it into a ResUNet-based encoder, extracting multi-scale feature maps layer by layer, and generating a feature map set covering low-level to high-level semantic information; performing hierarchical screening processing on the multi-scale feature maps output by the encoder, and selecting the deepest layer feature map as the input feature of hybrid encoding; performing branch processing on the deepest layer feature map, extracting local texture features through the CNN branch to generate a local feature map; meanwhile, performing kxk convolution on the image block through the Transformer branch and introducing position encoding, extracting global context features by using a multi-head self-attention mechanism to generate a global feature map; The local feature map and the global feature map are fused, an adaptive weight adjustment gating function is used, a dynamic feature of a polyp region is used to optimize a fusion ratio of the local and global features in real time, a differential dynamic gating fusion mechanism is realized which is deeply bound with a polyp shape feature, the two types of features are integrated, and a fused feature map is generated; The fused feature map is dimensionally reduced, the feature dimension is reduced to an original feature dimension through transposed convolution, and a final feature map of a deepest layer of the encoder is generated.

3. The method of claim 1, wherein, An edge guiding module is introduced in the middle and shallow layer multi-scale feature map of the encoder, multi-scale edge responses are extracted through the dilated convolution with the dilated rate of 1, 3, 5 and 7, an edge probability map is generated through the Sigmoid function activation, and the edge probability map is fused into the main feature map through the attention mechanism, including: The middle and shallow layer multi-scale feature map output by the encoder is screened, and the input feature map of the edge guiding module is determined, wherein each layer of feature map corresponds to different levels of semantic information; The selected middle and shallow layer multi-scale feature map is subjected to edge feature extraction processing, multi-scale edge responses are extracted through the dilated convolution layer with the dilated rate of 1, 3, 5 and 7, and a multi-scale edge feature map is generated; The multi-scale edge feature map is activated, and the edge probability map is generated through the Sigmoid function calculation, and the response intensity of the edge region is quantified; The edge probability map and the corresponding main feature map are subjected to attention fusion processing, and the weight of the high response region in the edge probability map is strengthened through the attention mechanism; The edge probability map and the main feature map are integrated, an enhanced main feature map with fused edge information is generated, and the boundary representation ability of the feature map is improved.

4. The method of claim 1, wherein, In the feature fusion stage, a double attention module and a spatial pyramid pooling module are introduced, the double attention module includes a channel forward attention, a channel reverse attention and a spatial attention submodule, and the sensitivity of the model to the significant region and the low response region is enhanced, including: The feature map output by the encoder is preprocessed in the fusion stage, and is sent into the feature fusion module to provide input features for the double attention mechanism and the spatial pyramid pooling module; The input feature is subjected to channel forward attention processing, the channel feature response of the significant region is strengthened through the channel forward attention submodule, and a channel forward enhanced feature map is generated; The input feature is subjected to channel reverse attention processing, a significant reverse channel mask is generated through the channel reverse attention submodule by using global maximum pooling, the channel feature representation of the low response region is enhanced, and a channel reverse enhanced feature map is generated; The channel forward and reverse enhanced feature maps are subjected to cross-dimension fusion processing, a cross-dimension reverse enhancement mechanism is constructed, in the spatial dimension, the key spatial position of the low response region containing small polyps and weak boundaries is focused by means of the spatial attention submodule combined with the channel reverse attention result, the features of these regions are simultaneously enhanced, and a spatial enhanced feature map is generated; The spatial enhanced feature map is subjected to multi-scale context aggregation processing, multi-scale context information is captured through the 1x1, 2x2, 4x4 and 8x8 scale adaptive pooling layers of the spatial pyramid pooling module, and a multi-scale fused feature map is generated; The channel forward, reverse enhancement features, spatial enhancement features and multi-scale fusion features are integrated and processed to form a feature map enhanced by double attention and multi-scale pooling, thereby improving the sensitivity of the model to significant areas and low response areas.

5. The method of claim 1, wherein, The decoder structure in the UNet++ style is used to fuse the shallow layer skip connection features of the encoder, edge enhancement features and multi-scale context features, gradually recover the resolution by upsampling, and generate the target colon polyp segmentation mask graph, including: The shallow layer skip connection features output by the encoder, the edge enhancement features generated by the edge guiding module, and the multi-scale context features generated by the spatial pyramid pooling module are collected and processed to determine the input feature set of the decoder, wherein each type of feature corresponds to information of different dimensions; The input feature set is processed by hierarchical fusion, the cross-layer interaction between features is realized through the nested dense skip connection structure in the UNet++ style, and preliminary fusion features are generated; The preliminary fusion features are processed by upsampling to gradually recover the resolution of the feature map and continuously integrate the feature information of different levels to generate a high-resolution fusion feature map; The high-resolution fusion feature map is processed to generate an initial segmentation mask graph by feature mapping to a pixel-level polyp region probability distribution; The initial segmentation mask graph is processed to strengthen the distinction between the polyp region and the background, and finally a target colon polyp segmentation mask graph is generated to complete the accurate segmentation of the polyp region.

6. The method of claim 1, wherein, The target colon polyp segmentation mask graph is processed to generate a colon polyp segmentation result, including: The target colon polyp segmentation mask graph and the real segmentation mask in the data set are compared and processed to extract the pixel-level difference information between the two, and an initial loss calculation basis is generated; The extracted difference information is quantified as a loss, and the weighted sum of the binary cross-entropy loss and the Dice loss is used as the joint loss function to calculate the model prediction error and generate a comprehensive loss value; The comprehensive loss value is optimized, and the Adam optimizer with an initial learning rate of 1e-4 is used to update the model parameters by back propagation to generate parameter adjustment gradients; The learning rate is dynamically scheduled by the ReduceLROnPlateau scheduler according to the loss change to adjust the learning rate, and the gradient clipping and early stopping mechanism are enabled to generate an optimized learning strategy; The model parameters are iteratively optimized, and the loss calculation, parameter update and learning rate adjustment process is repeated until the model converges, generating a trained segmentation model that outputs accurate colon polyp segmentation results.

7. A hybrid CNN-Transformer colon polyp image segmentation device combining edge guidance and dual attention mechanism, characterized in that, A hybrid CNN-Transformer colon polyp image segmentation method combining edge guidance and double attention mechanisms is used to implement the method of any one of claims 1-6.

8. An electronic device, comprising: It includes: a first processor; and a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the method of any one of claims 1-6 by executing the executable instructions.

Images