Endoscopic polyp segmentation method based on multi-scale feature aggregation

Abstract

The invention discloses an endoscopic polyp segmentation method based on multi-scale feature aggregation, and belongs to the technical field of polyp segmentation processing methods, and the method comprises the following steps: 1, preparing an endoscopic image data set, 2, preprocessing data, 3, carrying out an endoscopic image polyp segmentation task, 4, setting an optimizer and a loss function, and 5, training and storing a PVT-MSFF model. And step 6, a PVT-MSFF model test stage, by using the method of the invention, the PVT-MSFF has higher accuracy and generalization ability, and is mainly applied to the aspect of endoscopic image polyp segmentation.

Description

Technical Field

[0001] The present invention belongs to endoscopic image processing technology, and in particular relates to an endoscopic polyp segmentation method based on multi-scale feature aggregation. Background Art

[0002] Colorectal cancer, a malignant tumor with high morbidity and mortality rates worldwide, originates from polyp lesions and takes decades to develop and worsen, providing a window for early prevention and diagnosis. Endoscopy, as the preferred diagnostic method, can visually observe polyps and mucosal lesions, which is of great significance in reducing morbidity. However, its accuracy depends on the doctor's experience. Complex environments and long-term operations can easily lead to fatigue, increasing the risk of missed detection. In addition, polyps have diverse shapes and sizes and are similar to surrounding tissues, with high randomness, which poses a challenge to accurate segmentation. Therefore, how to effectively extract low-level and high-level features of polyps, while ensuring that no polyps are missed, effectively distinguish the difference between polyps and their boundary tissues, has become a key difficulty in endoscopic image polyp segmentation technology.

[0003] Early polyp segmentation methods primarily relied on manually extracting low-level features such as color and shape, and then using various classifiers to distinguish polyps from surrounding tissue. Common techniques include edge detection, threshold segmentation, region growing, cluster analysis, and active contour models. These methods primarily rely on low-level image features such as grayscale, texture, and shape to distinguish polyps from background. However, due to the diverse morphology and color of polyps in endoscopic images, traditional methods have low segmentation accuracy in complex backgrounds. In recent years, with the rapid development of deep learning technology, its remarkable capabilities in deep feature extraction have attracted widespread attention and gained favor among numerous researchers. Deep learning has achieved significant breakthroughs in a range of computer vision tasks, such as image segmentation, image classification, and object detection. These breakthroughs not only demonstrate the powerful potential of deep learning but also provide new insights and methods for research in related fields. Furthermore, the continuous development of deep learning has led to the emergence of a variety of innovative network architectures. Convolutional neural networks (CNNs), with their outstanding performance in image processing, have become a foundational model in computer vision. The diversification and innovation of these deep learning architectures have led to the increasingly widespread application of deep learning technology in the task of polyp segmentation in endoscopic images, promoting research progress in this field.

[0004] With the rapid development of deep learning technology, methods such as convolutional neural networks (CNNs) can automatically learn multi-scale, complex image semantic features, significantly improving the accuracy and robustness of polyp segmentation, achieving higher precision and better adaptability. In 2015, Long et al. first proposed a fully convolutional neural network (FCN) that used a fully convolutional approach to solve the pixel-wise prediction problem. Building on this, Ronneberger et al. proposed the UNet network architecture. This unique U-shaped network structure improves the difficulty in fusing extracted low-level and high-level information, further improving the model's segmentation performance. However, their use of a single scale prevents accurate polyp segmentation. To address the problem that most methods do not aggregate multi-scale or multi-sensory features and do not consider region boundary constraints, Fang et al. introduced cascaded layers in the encoder and decoder, used a selective kernel module (SKM) in the convolutional layers to achieve selective feature aggregation, and employed a boundary-sensitive loss function to constrain the relationship between regions and boundaries. While these methods have demonstrated the effectiveness of the Transformer and its variants in medical segmentation, they still suffer from limited attention to local information, low computational efficiency, and high memory requirements.

[0005] To solve this problem, Fan proposed a PraNet model that is different from the traditional UNet. It first roughly estimates the lesion area and then combines the reverse attention module to further explore the boundary clues, which significantly improves the positioning accuracy of the polyp boundary. Chen, Huang and others use the reverse attention module as the core to conduct in-depth research on boundary feature extraction and have achieved advanced performance in multiple polyp segmentation benchmarks. However, existing methods are not global.

[0006] The Transformer architecture has demonstrated outstanding image analysis capabilities thanks to its unique self-attention mechanism. In particular, when addressing the challenges of medical segmentation, the Transformer's expertise in global correlation is crucial when dealing with lesion structures with irregular morphology and unclear edges, effectively improving the accuracy and reliability of segmentation. ImageTransformer was the first to apply the Transformer architecture to the field of computer vision, bringing new changes to the image field. Since then, a large number of Transformer-based visual models have been proposed. Dosovitskiy et al. first applied the most primitive Transformer model to image classification and proposed a structure based entirely on the self-attention mechanism (ViT). Yan et al. proposed a new ConTNet that combines the characteristics of Transformer and CNN to achieve an expansion of the receptive field. Jha et al. proposed TransNetR, which incorporates residual connections into the Transformer backbone to provide accurate real-time polyp segmentation and improved model generalization.

[0007] In addition, PVT built a pure Transformer backbone network that can serve multiple downstream tasks (such as segmentation and detection). Given the hierarchical structure and window attention, Liu proposed a general Transformer architecture for visual tasks. After the initial success of PVT and PVTv2, Dong et al. introduced it to the task of polyp segmentation, and handled the polyp segmentation task by fully exploiting cross-stage semantic clues. However, this method still has defects in generalization ability, so their application scenarios are limited. Zhang et al. made some extensions based on Polyp-PVT and obtained good results. However, their method lacks boundary sensitivity, resulting in larger mis-segmented areas.

[0008] So far, deep learning models have shown significant advantages in the task of polyp segmentation in endoscopic images. However, existing deep learning methods related to polyp segmentation in endoscopic images still face several challenges, including the following:

[0009] (1) Poor adaptability to multi-scale morphology: Existing models mostly adopt a single-scale feature modeling strategy. In complex clinical scenarios, significant information loss occurs during the multi-scale feature fusion process, resulting in a high polyp missed detection rate.

[0010] (2) Boundary segmentation is fuzzy: The spectral features of polyps and surrounding tissues are highly similar. Most methods ignore the fusion of low-level information and only focus on high-level features, resulting in the loss of boundary information and making boundary segmentation fuzzy.

[0011] (3) Insufficient fusion of local and global features: Existing methods focus on the extraction of local information and ignore the fusion of global information, resulting in a loss in segmentation accuracy. Summary of the Invention

[0012] In order to solve the technical problems raised in the background technology, the present invention provides an endoscopic polyp segmentation method with multi-scale feature aggregation (A Pyramid Vision Transformer with Multi-Scale Feature Fusion, PVT-MSFF), which adopts the following technical solutions:

[0013] Step 1: Preparation of endoscopic polyp segmentation image dataset

[0014] This method uses five hyperspectral image datasets: The Kvasir-SEG dataset is an enhanced version of the Kvasir dataset, providing more refined pixel-level segmentation annotations for detection tasks and verified by professional physicians. Kvasir-SEG consists of 1,000 polyp images extracted from colonoscopy videos, with image resolutions ranging from 332×487 to 1920×1072. The CVC-ClinicDB dataset is a publicly available polyp segmentation dataset. It first captures 31 frames showing polyp tissue from 23 colonoscopy videos. Then, 25 frames with good quality are retained after screening. Finally, 612 polyp images are obtained. The CVC-ColonDB dataset contains 380 polyp images extracted from 15 colonoscopy videos, with a fixed image resolution of 500×574. Each image is accompanied by a corresponding binary mask label to highlight the polyp region. The CVC-300 dataset is a widely used dataset for polyp segmentation, with a fixed image resolution of 500×574. ETIS dataset: ETIS is an early public polyp segmentation dataset consisting of 196 images with a resolution of 1225×996.

[0015] Step 2: Data Preprocessing

[0016] The endoscopic images of the Kvasir-SEG dataset and the CVC-ClinicDB dataset in step 1 are combined with the labels of the manually marked polyp areas. The labels mark the polyp areas as 1 and the background as 0 to achieve accurate supervised training. The images and labels are enhanced to enhance the model's performance under different lighting and imaging conditions.

[0017] 2.1 Normalization and Standardization: Normalization and standardization operations are important steps in data preprocessing, aiming to give the input data a uniform numerical range and distribution to promote the stability and convergence speed of model training. It refers to mapping the image pixel values to the interval [0,1], that is, dividing the original pixel value by the maximum value (for example, 255) for scaling; standardization further adjusts the mean and variance of the image data on the basis of normalization. Usually, the statistical values of common datasets (such as ImageNet's mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]) are used for channel normalization to ensure that the pixel distribution of different channels tends to be consistent. These two steps can effectively eliminate the differences in brightness and contrast caused by different acquisition devices and scenes, making it easier for the model to learn the features in the data, while also reducing the interference of numerical fluctuations on the gradient descent process. Label processing (binarization): In the segmentation task, label processing is a key step to ensure the accurate transmission of the supervisory signal during the model training process. For the binary segmentation task, the conventional operation is to binarize the manually labeled or automatically generated segmentation mask. Binarization usually involves first graying the label image (if the original label is in color), and then mapping the label pixels to two categories (background is 0, target is 1) by setting a threshold (the original image is 0-255, with 128 as the boundary, and the value below 128 is set to 0, otherwise it is set to 1) to facilitate the calculation of cross entropy, Dice or IoU loss. The binarization process must ensure that the label boundaries are clear and the categories are clear. At the same time, during the data enhancement process, the binary label should be geometrically transformed together with the image using nearest neighbor interpolation to ensure that the category information of each pixel is not confused due to interpolation.

[0018] 2.2 Dataset Settings:

[0019] The same data distribution scheme as used in the Polyp-PVT and PraNet experiments was strictly adopted. The Kvasir-SEG dataset was divided into 900 training images and 100 test images, while the CVC-ClinicDB dataset contained 548 training images and 64 test images. This division ensured target diversity within each dataset while also meeting commonly used evaluation criteria in the field. Furthermore, to verify the model's generalization capabilities in real-world application scenarios, three completely unseen datasets, CVC-ColonDB, ETIS, and CVC-300, were used for testing. These datasets differed in acquisition equipment, image quality, and annotation style, thereby verifying the model's cross-dataset adaptability.

[0020] Step 3: Initialize the PVT-MSFF model

[0021] Based on step 2, a three-stage progressive architecture is used, which consists of a PVT encoder, a Feature Enhancement Module (FEM), a Multi-Scale Fusion Module (MSFM), a Global Context Sensing (GCS), and a cascade fusion module CFM. In addition, a multi-scale fusion module (MSFM) is used to enhance the multi-level feature output from the encoder. After the image is input into the PVT encoder, features are obtained from the four stages of PVT. In the initialization stage, first in the PVT encoder, a single image is divided into four layers of features, denoted as where i∈{1,2,3,4}, where S=2 i+1 and the number of channels C selected from the set {64,128,320,512} i In order to better illustrate the operation logic of the model, the model is divided into three stages. The first stage is the low-level feature extraction stage, the second stage is the high-level semantic aggregation stage, and the third stage is the cross-level feature fusion stage. In the low-level feature extraction stage, the high-resolution X1 with rich edge and texture information is passed through the MSFM module to obtain X′1 after multi-scale feature fusion, and then processed by the FEM module to obtain features rich in boundary information. In the high-level semantic aggregation stage, the number of channels of {X2, X3, X4} is first adjusted to 32 and then input into MSFM respectively to obtain {X′2, X′3, X′4} with multi-scale features, which are then input into CFM for fusion. In the cross-level feature fusion stage, the output of the first stage is and the second stage output After global context awareness GCS, we get Finally, R1 and R2 are added to obtain the final prediction result R of the model. In addition, during the training process, the loss of R1 and R2 relative to the ground truth (GT) is added as the training loss.

[0022] Step 4: Endoscopic image polyp segmentation task;

[0023] After initialization in step 3, the endoscopic image preprocessed in step 2 is first input into the Pyramid Vision Transformer (PVT) encoder, which uses its hierarchical pyramid structure to extract multi-scale, low-level detail features, capture edges, and low-level texture information, laying the foundation for subsequent segmentation.

[0024] Subsequently, the network further processes the features output by the encoder through the Multi-Scale Fusion Module (MSFM) and the Feature Enhancement Module (FEM). The MSFM dynamically weights and fuses information of different scales through a lightweight convolution chain and the SE-FF submodule.

[0025] Furthermore, FEM focuses on leveraging a channel-wise attention mechanism to enhance edge information extraction, suppress noise, and preserve low-level details, improving the ability to capture tiny or blurred polyps. Compared to other methods, this process better incorporates edge information extraction, a feature rarely addressed by other methods. This method effectively layers features, optimizing the weight of edge information and thus enhancing the advantages of boundary segmentation.

[0026] Secondly, GCS combines the two branches of global perception and local perception, and uses global average pooling and local feature retention methods to integrate low-level details and high-level semantic information, improve the overall feature expression ability, and ensure that the segmentation results meet high standards in terms of regional coherence and boundary fineness. Unlike existing methods, this step increases the processing of global perception while ensuring efficiency, which is neglected by many studies, but it combines low-level semantics and global semantics well, optimizing the overall segmentation effect while positioning.

[0027] Finally, after further optimization by the Cascade Fusion Module (CFM), the network comprehensively refines the aforementioned features and outputs an accurate pixel-level segmentation map. The entire process of step 3 ensures that each input image can obtain a segmentation result that is strictly aligned with the ground truth, effectively identifying the entire polyp and highlighting its boundary details.

[0028] Step 5: Set the optimizer and loss function;

[0029] Select a suitable optimizer and loss function based on the complexity of the model obtained in step 4 and the requirements of the endoscopic image polyp segmentation task. The specific solution is as follows:

[0030] AdamW combines the adaptive gradient update and weight decay mechanism of Adam. Let the model parameter obtained in step 4 be θ. t After the gradient is calculated, the update process is as follows:

[0031] (1) First-order momentum estimation

[0032] m t =β1m t-1 +(1-β1)g t

[0033] (2) Second-order momentum estimation

[0034]

[0035] (3) Deviation correction

[0036]

[0037] (4) Parameter update formula

[0038]

[0039] Where η represents the learning rate (e.g. 1×10 -4 ), λ w represents the weight attenuation coefficient, ∈ is a small constant to prevent division by zero;

[0040] Loss calculation, after feature extraction, uses the cross entropy loss function K CE To calculate Y pretrain and the true label Y train The error between, the cross entropy loss function is one of the commonly used loss functions in segmentation problems, and the loss function is expressed as:

[0041]

[0042] Among them, y i represents the true category label, Represents the predicted probability, M is the number of samples, and the accuracy of the model prediction can be quantified by calculating the loss function;

[0043] Optimize model parameters. Use the Adam optimizer to update the model weights and combine it with the exponential decay learning rate scheduler to optimize model parameters. The learning rate adjustment formula is as follows:

[0044] η t =η0·exp(-λ·t)

[0045] Among them, η t is the learning rate of the tth round, η0 is the initial learning rate, and λ is the decay rate parameter;

[0046] Step 6: PVT-MSFF model training and saving;

[0047] In the training phase, the endoscopic images and corresponding labels after preprocessing and data augmentation in step 2 are used. The labels are single-channel binary masks. The PVT-MSFF model is trained with the parameters in step 5 to enable it to learn effective features and make accurate predictions. The trained PVT-MSFF model is saved for further testing.

[0048] The training loop is an iterative process, repeating from i = 1 to N until the model parameters reach their optimal state. The training loop is the core of the model training process. Through continuous iterative optimization, the model gradually learns the inherent patterns and characteristics of the data. In each iteration, the model undergoes a complete forward and backward propagation process, gradually adjusting its internal parameters and improving prediction accuracy. Specifically, at the beginning of each epoch, the data loader randomly perturbs the training set (shuffle = True) and divides the samples into several data blocks according to the preset batch_size. For small sample sizes, full batch gradient descent can be used; for large datasets, stochastic gradient descent (SGD) is implemented, with parallel data preloading enabled via torch.utils.data.DataLoader. During the forward propagation phase, the model performs sliding convolution kernel calculations, nonlinear transformations of activation functions, and spatial pooling of feature maps layer by layer, ultimately outputting a prediction tensor. Loss functions (such as cross entropy and mean squared error) compare the predicted values with the true labels to generate a scalar loss value, which represents the degree to which the current model parameters deviate from the ideal state.

[0049] Save the model. After the training cycle is repeated N times, the model training is completed. At this time, the PVT-MSFF parameters have been adjusted to the optimal state, and the model's prediction performance has reached the expected target. The optimal model is saved. The optimal model at this moment will include the specific weights of the perception of the data and record the specific process of training. The weights can be called later for testing.

[0050] Step 7: PVT-MSFF model testing phase

[0051] The test set sample x test Input into the trained model in step 6. In this stage, only forward propagation is performed, and back propagation and parameter update are not required. In step 7, the model outputs the predicted feature vector y for each test sample. predict By comparing the feature vector with the segmentation threshold, the model generates the corresponding predicted label; the predicted label obtained in step 7 is compared with the true label y of the test set. test Comparison, used to evaluate the performance of the model.

[0052] Specifically, after the model training and validation phases are complete, the system enters the test set performance evaluation phase. At this point, the preprocessed test set sample data, xtest, is loaded into memory in batches and fed through a data pipeline into the neural network model whose parameters have been optimized in step 6. During this inference phase, the model's computational graph maintains only the forward propagation path. The gradient calculation module is explicitly disabled using the torch.no_grad() context manager, which avoids the computational resource consumption associated with backpropagation and ensures that model parameters are not accidentally updated by test data. After obtaining the test examples in step 7, the input data is sequentially processed through feature extraction using the convolution kernel, nonlinear transformation using the activation function, and spatial mapping using the fully connected layer. Finally, the output layer generates a feature vector, ypredict, whose dimensions match the number of classification categories. For the endoscopic polyp segmentation task, a sigmoid function is used to compress the feature values to the [0, 1] range. Binary prediction labels are generated by comparing them against a preset threshold (default 0.5).

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] Performance evaluation experiments on five public polyp segmentation datasets demonstrate that, compared to existing state-of-the-art CNN and Transformer-based methods, PVT-MSFF achieves state-of-the-art performance on the Kvasir-SEG dataset, achieving 0.921 mDice and 0.870 mIoU, representing improvements of 1.0% / 1.7% and 0.4% / 0.6% over HSNet and Polyp-PVT, respectively, in mDice and mIoU. On the CVC-ClinicDB dataset, PVT-MSFF further achieves new records with 0.949 mDice and 0.902 mIoU, demonstrating improvements of 0.5% / 1.2% and 0.4% / 1.3% over HSNet and Polyp-PVT, respectively, validating its ability to model typical polyp structures. Furthermore, generalization testing on unseen datasets demonstrates significant advantages, including promising results on the ColonDB dataset, which is subject to significant colonoscopy noise. On the ColonDB dataset, mDice reaches 0.813, an improvement of 1.4% and 1.6% over HSNet and Polyp-PVT, respectively; mIoU is 0.734, an improvement of 0.6% and 1.1% over HSNet and Polyp-PVT (0.728 and 0.723, respectively). On the ETIS dataset, our method leads with 0.809mDice and 0.731mIoU, a 2.2% and 2.5% improvement over Polyp-PVT, and 2.1% and 1.5% improvement over HSNet, demonstrating its advantage in small object boundary segmentation. Our method's mDice (0.878) and mIoU (0.814) improves by 0.6% and 1.0% over Polyp-PVT, and by 0.8% and 1.4% over HSNet, respectively. In Table 5, our method leads with 0.809mDice and 0.731mIoU, which are 2.2% and 3.5% higher than Polyp-PVT and 2.1% and 1.5% higher than HSNet, indicating its significant advantages in small objects and fuzzy boundary scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a multi-scale feature hybrid pyramid converter model based on the present invention;

[0056] Figure 2 is the multi-scale feature fusion module (MSFM) of the present invention;

[0057] Figure 3 is the Feature Enhancement Module (FEM) of the present invention;

[0058] Figure 4 is the cascaded feature fusion module (CFM) of the present invention;

[0059] Figure 5is the Global Context Sensing (GCS) of the present invention;

[0060] Figure 6 This is a visualization of the comparative results of the present invention on five public datasets. DETAILED DESCRIPTION

[0061] The specific technical solutions of the present invention are further described below to facilitate further understanding of the present invention by those skilled in the art, but do not constitute a limitation to their rights.

[0062] In one embodiment, see Figure 1 , Multi-scale feature aggregation endoscopic polyp segmentation method Example 1, endoscopic image polyp segmentation method based on multi-scale feature hybrid pyramid transformer (A Pyramid Vision Transformer with Multi-Scale Feature Fusion, PVT-MSFF):

[0063] The system implementing the method consists of a pyramid vision transformer encoder (Pyramid Vision Transformer Encoder), a feature enhancement module (FEM), a multi-scale feature fusion module (MSFM), a global context sensing (GCS) and a cascaded fusion module (CFM).

[0064] 1.1 Pyramid Visual Transform Encoder

[0065] Existing research shows that Transformer-based models have significant advantages in image segmentation tasks. The core of Transformer-based models is to establish long-range dependencies between pixels in the entire image through the self-attention mechanism, thereby achieving dynamic modeling of global contextual information. Compared with the local inductive bias characteristics of traditional CNN architectures, this type of method can more effectively capture the topological structure and boundary correlation of the lesion area. Based on this, this paper adopts Pyramid Vision Transformer V2 (PVTv2) as the basic feature extractor, and makes adaptive improvements for medical polyp segmentation tasks. Figure 1 As shown, the multi-scale feature group {X1, X2, X3, X4} generated by its hierarchical pyramid architecture is retained. The architecture achieves a progressive transformation of feature granularity through a step-by-step downsampling strategy (step size = 4, 8, 16, 32). Shallow features Encode high-resolution detail information (such as polyp edges, texture patterns), and its spatial resolution maintains a 1 / 4 ratio with the input image, which can accurately locate the lesion boundary. Deep features {X2, X3, X4}. Corresponding to and The multi-head self-attention mechanism captures cross-region semantic associations and forms discriminative high-level semantic representations. This design enables the model to have both local sensitivity and global perception capabilities, providing the foundation for subsequent feature fusion.

[0066] 1.2 Multi-scale feature fusion module

[0067] See also Figure 2 To address the issues of static weight allocation and channel redundancy in multi-scale feature fusion, this study proposes a Multi-Scale Fusion Module (MSFM) based on a channel attention mechanism. Its core submodule is the Squeeze-and-Excitation Feature Fusion (SE-FF) unit. Traditional multi-scale methods (such as Inception and dilated convolutional pyramids) fuse features from different receptive fields through a fixed branch structure, but the contribution weights of each branch must be manually preset, resulting in the model's inability to adapt to the input content. Furthermore, the direct use of large kernel convolutions significantly increases the number of parameters, while multi-scale feature concatenation easily introduces redundant channels, reducing feature effectiveness. MSFM decomposes large kernel convolutions into lightweight 3×3 convolution chains and embeds the SE-FF submodule to achieve dynamic channel weight allocation, enhancing the expressive power of multi-scale features while controlling the number of parameters. Building on the SENet concept, SE-FF uses global statistics to model channel dependencies, addressing the issue of multi-scale feature redundancy and improving the model's sensitivity to key semantics.

[0068] In the experiment, the MSFM module k is set to 4. Figure 2 As shown, the input feature map X∈R H×W×CFirst, it is compressed to a low-dimensional space (C→C / 4) through 1×1 convolution and then input into four parallel SE-FF channels. Each SE-FF channel consists of a stacked 3×3 convolution chain with expansion rates set to d, [d / 4]+1, [d / 2]+1, d+1, respectively, to capture local details, mid-range and global context information. The output of each channel is processed by the SE module, and channel statistics are generated by global average pooling. The nonlinear dependencies between channels are learned through two fully connected layers (dimensionality reduction ratio r=16), dynamic weights are generated and multiplied with the feature map channel by channel to suppress redundant features. After the four branch features are processed by SE-FF, the hierarchical feature fusion (HFF) strategy is used to cascade and add them layer by layer instead of directly splicing them to eliminate the checkerboard noise caused by cross-scale jump connections. The fused features are restored to the input channel dimension through 1×1 convolution and residually connected with the original input X to retain the underlying detail information. Finally, the output Y∈R is generated through ReLU activation. H×W×C .

[0069] 1.3 Feature Enhancement Module

[0070] Wu et al.

[20] found that low-level features are limited by their higher spatial resolution and require more computing resources, resulting in a smaller contribution to overall performance under the same circumstances. However, low-level features often contain rich features, which helps improve the performance of the model in boundary segmentation. Based on this, this study proposes a feature fusion module based on dual-branch collaborative enhancement. Its core design concept is to extract the rich edge information in low-level semantic features through the channel attention mechanism and discard the spatial attention module to optimize computational efficiency.

[0071] See also Figure 3 , input features After the split operation, two groups of sub-features are obtained The left branch uses the channel attention mechanism to enhance semantic expression. Specifically, it generates the channel statistic S through global average pooling (GAP). The calculation formula is as follows:

[0072]

[0073] in, For the left branch sub-feature, the GAP operation compresses the spatial information of each channel into a scalar, reflecting the global response strength of the channel. And through the fully connected layer and Sigmoid activation to generate the weight vector The specific formula is as follows:

[0074] w=σ(W2δ(W1S)),

[0075] in, Used to compress the channel statistics S to a low-dimensional space, where r is set to 16, δ is a ReLU or function, It is used to restore the channel to its original channel dimension, σ is the sigmoid activation function. Finally, the enhanced features are obtained by channel weighting The specific calculation formula is as follows:

[0076] U=w⊙f c .

[0077] The right branch extracts nonlinear spatial features through 3×3 convolution and Tanh activation The specific calculation formula is as follows:

[0078] Z=|tanh(Conv(f i ))|,

[0079] Conv is the convolution operation, the convolution layer padding is set to 1, tanh is the activation function, and the absolute value is used to prevent negative values. To achieve the interaction of dual-branch features, the module designs two sets of Hadamard product operations: first, the original features of the left branch are Multiply element-by-element with the right branch spatial feature Z to generate a feature f containing the original channel information and spatial details c '=Z⊙f c The second interaction is to convolve the weighted feature U obtained by the left branch with the convolution result Conv(f i ) Calculate the Hadamard product and get f i '=U⊙Conv(f i ). Finally, the original number of channels is restored by splicing the 1×1 convolution kernel in the channel dimension, and the output feature Output can be expressed as

[0080] Output=Conv 1×1 (Concat(f c ',f i ')).

[0081] 1.4 Cascade Fusion Module

[0082] See also Figure 4 , a cascaded fusion module (CFM) is used to achieve a balance between computational efficiency and feature representation capabilities. Specifically, the module adopts a two-stage feature fusion mechanism, and each stage is built based on a standardized convolution unit. Each convolution unit F(·) consists of a 3×3 convolution kernel (with 1 pixel padded at the edge), batch normalization

[35] , and a ReLU activation function

[36] . The specific implementation process is described as follows:

[0083] (1) The highest-level feature map X'4 is upsampled to the spatial resolution of X3' by bilinear interpolation, and the feature tensors X are generated by parallel convolution branches F1(·) and F2(·). 14 and X 24 The feature interaction relationship is established as shown in formula (1): First calculate X 14 The Hadamard product of X3' is then concatenated with the product along the channel axis. 24 Finally, feature regularization is performed through F3(·) to output the fusion feature

[0084] X 34 =F3(Concat(F(X4'))⊙X3',F2(X4'))),

[0085] Where “⊙” represents the Hadamard product, and Concat(·) is the concatenation operation along the channel dimension.

[0086] (2) X'4, X3' and X 34 The uniform upsampling is performed to the resolution of X'2, and the feature optimization is performed by F4(·), F5(·) and F6(·). The composite feature relationship is established as shown in formula (2): First, the ternary Hadamard product of F4(Upsampling(X4')), F5(Upsampling(X3')) and X'2 is calculated, and then the result is concatenated with F6(Upsampling(X 34 )), feature compression is implemented through cascade convolution F7(·) and F8(·), and the final output is

[0087] T1=F8(F7(Concat(F4(X4')⊙F5(X3')⊙X2',F6(X 34 ))))

[0088] 1.5. Global Context Sensing (GCS)

[0089] See also Figure 5 In order to solve the key problems of missing global context information and insufficient preservation of local details in cross-level feature fusion, this module aims to fuse high-level features F x and low-level features F y ,Through the dual-stream fusion layer, shallow features and deep features are fused, combining the dual advantages of global context guidance and local detail constraints, so as to more accurately integrate the semantic information and edge features of polyps. The specific formula is as follows:

[0090] F=Concat(F x ,Downsample(Fy )),

[0091] Where F is the merged feature, F x and F Y These are the main features and auxiliary features, respectively. Next, the module is divided into two branches: global perception and local perception. The global perception branch uses global average pooling (GAP) to extract global features, adjusts the number of channels through 1×1 convolution, and then broadcasts them back to the original size. The local perception branch directly retains the original feature information. The two are added together and then activated by the Sigmoid function to generate the dynamic calibration weight S. The specific formula is as follows:

[0092] S=σ(Broadcast(GAP(f)))+F

[0093] Where σ is the sigmoid function, GAP is the global pooling operation, and Broadcast is used to restore the original size of the image. Then it is multiplied element by element with F to adjust the importance of the feature. Then, the adjusted feature is reduced in dimension by 1×1 convolution and combined with the input F. x Residual connections are made to retain the original feature information and enhance the stability of the model. The final output features are consistent with the input F x With the same dimension, the final output result is obtained. The specific formula is as follows:

[0094]

[0095] 1.6 Loss Function

[0096] The cross entropy loss function is one of the commonly used loss functions in segmentation problems. It measures the difference between the predicted probability distribution and the true label distribution. Specifically, the loss function can be expressed as:

[0097]

[0098] Among them, yi represents the true category label, Denotes the predicted probability, and M is the number of samples. By calculating the loss function, we can quantify the accuracy of the model's predictions and provide a basis for subsequent optimization steps. Furthermore, to ensure the model's stability and generalization capabilities, regularization methods are used to avoid overfitting.

[0099] 1.7 Model Optimization

[0100] The Adam optimizer is an adaptive learning rate optimization algorithm that combines the advantages of both momentum and RMSprop. During the optimization process, the Adam optimizer adjusts the learning rate of each parameter based on the variance and mean of historical gradient estimates, ensuring that the model converges to the optimal solution more quickly and stably. Furthermore, to avoid exploding or vanishing gradients, methods such as learning rate decay are used to further fine-tune the optimization process.

[0101] 2.1. Dataset Introduction

[0102] The Kvasir-SEG dataset is an enhanced version of the Kvasir dataset, providing more refined pixel-level segmentation annotations for detection tasks, validated by professional physicians. Kvasir-SEG consists of 1,000 polyp images extracted from colonoscopy videos, with image resolutions ranging from 332×487 to 1920×1072. The ClinicDB dataset is a polyp segmentation dataset provided by the Barcelona Hospital Clinic.

[0103] It first captures 31 frames of polyp tissue from 23 colonoscopy videos. Then, it retains 25 frames of good quality after screening. Finally, it obtains 612 polyp images. The ColonDB dataset contains 380 polyp images extracted from 15 colonoscopy videos, with a fixed image resolution of 500×574.

[0104] Each image has a corresponding binary mask label to highlight the polyp area. The CVC-300 dataset is a 300-image dataset widely used for polyp segmentation, with a fixed image resolution of 500×574.

[0105] ETIS dataset: ETIS is an early polyp segmentation dataset consisting of 196 images with a resolution of 1225×996.

[0106] 2.2.1 Hyperparameters

[0107] For each method, the Adam optimizer was used to update network parameters, with a learning rate of 5e-4. Training was stopped when the training loss stabilized. The number of training epochs was uniformly set to 100, under which the training loss was stable. The batch size was set to 32. All comparison methods used publicly available code from the original authors and were re-tested on the same hardware configuration to ensure experimental consistency.

[0108] 2.2.2 Training Samples

[0109] To evaluate the segmentation performance of the model under limited sample conditions, the number of samples used in the experiment was divided. The Kvasir-SEG dataset selected 900 training images, and the ClinicDB dataset contained 548 training images. All training samples were randomly selected to ensure the stability of the algorithm. Except for the training samples, the remaining ones were used for testing.

[0110] 2.2.3 Operating Platform and Indicators

[0111] For fair comparison, we adopted the same data distribution as used in the Polyp-PVT and PraNet experiments. Specifically, we employed the Kvasir-SEG and ClinicDB datasets to validate the feature modeling capabilities of the proposed framework. Kvasir-SEG consists of 900 training images and 100 test images, while ClinicDB consists of 548 training images and 64 test images. Similarly, we validated the generalization performance of our approach on the unseen ColonDB, ETIS, and CVC-300 datasets.

[0112] The algorithm was deployed using the PyTorch framework. For hardware, two NVIDIA 3090TI GPUs were used to accelerate the training of the segmentation model, with a mini-batch size of 16 images. Following common practice, the AdamW

[38] optimizer was used with an initial learning rate of 1x le-4. The segmentation model was trained for a total of 200 epochs. During training, images were uniformly set to 352x352 pixels.

[0113] 2.3 Experimental Analysis

[0114] We validate the feature modeling capabilities of our framework by training it on the Kvasir-SEG and ClinicDB datasets. Furthermore, we present comparative results against state-of-the-art approaches, including UNet, UNet++, PraNet, CFPNet, HSNet, Polyp-PVT, and TransNetR.

[0115] See also Figure 6 ,To verify the performance of the model, comparative experiments were conducted on ,seen datasets such as Kvasir-SEG, CVC-ClinicDB, and unseen datasets ,such as ColonDB, ETIS, and CVC-300, with the current ,optimal CNN model HSNet and the hybrid architecture Polyp-PVT.

[0116] In one embodiment, see Figure 6 , a visual comparison is made between the scheme of this study and the five most advanced methods, where the white part is the correctly segmented part and the red part is the incorrectly segmented part.

[0117] In one embodiment, by Figure 6 It can be seen that the method has significant advantages in segmentation accuracy. This method shows good edge continuity in the polyp boundary results. This method effectively distinguishes between real boundaries and artifact interference through the dual-branch interaction mechanism of the feature enhancement module, significantly reducing the "jagged" or "burr-like" mis-segmentation phenomenon common in traditional methods. Since the method adapts to the features of polyps of different sizes and shapes through the dynamic channel weight allocation of the multi-scale feature fusion module, the method shows a high degree of adaptability in different scenarios, and can accurately segment polyps of different shapes and colors. In addition, the global context perception module of this method uses the global-local dual-stream attention mechanism to enhance the main semantics while retaining the key details of the polyp, and uses multi-branch dilated convolution to fuse the global context to ensure the integrity of the lesion body.

[0118] In one example, a comparison of Tables 1 and 2 demonstrates that our method achieves significant performance improvements compared to other methods. Table 1 shows that on the Kvasir-SEG dataset, our method achieves optimal results with 0.921 mDice and 0.870 mIoU, representing improvements of 1.0% / 1.7% and 0.4% / 0.6% over HSNet and Polyp-PVT in mDice and mIoU, respectively. Table 2 shows that our method further achieves new records with 0.949 mDice and 0.902 mIoU, representing improvements of 0.5% / 1.2% and 0.4% / 1.3% over HSNet and Polyp-PVT, respectively, validating its ability to model typical polyp structures.

[0119] In one embodiment, a generalization test was performed on an unseen dataset, and the proposed method showed significant advantages: for the CVC-ColonDB dataset with severe colonoscopy noise interference, the results are shown in Table 3. On the ColonDB dataset, mDice reached 0.813, an increase of 1.4% and 1.6% over HSNet and Polyp-PVT respectively; mIoU was 0.734, an increase of 0.6% and 1.1% over 0.728 and 0.723 of HSNet and Polyp-PVT respectively.

[0120] In one embodiment, as shown in Table 5, on the ETIS dataset, our method leads with 0.809mDice and 0.731mIoU, which is 2.2% and 2.5% higher than Polyp-PVT and 2.1% and 1.5% higher than HSNet, demonstrating its advantage in small object boundary segmentation.

[0121] In one embodiment, in addition, as shown in Table 4, in the CVC-300 dataset, the mDice (0.878) and mIoU (0.814) of the proposed method are respectively improved by 0.6% and 1.0% over Polyp-PVT, and by 0.8% and 1.4% over HSNet.

[0122] In one embodiment, in Table 5, our method leads with 0.809mDice and 0.731mIoU, which is 2.2% and 3.5% higher than Polyp-PVT and 2.1% and 1.5% higher than HSNet, indicating its significant advantages in small objects and fuzzy boundary scenarios.

[0123] Table 1 Comparison of the patents of this invention on the test set Kvasir-SEG

[0124]

[0125]

[0126] Table 2 Comparison of the patents of this invention on the test set CVC-ClinicDB

[0127]

[0128] Table 3 Comparison of the patents of this invention on the test set CVC-ColonDB

[0129]

[0130]

[0131] Table 4 Comparison of the present invention's patents on the test set CVC-300

[0132]

[0133] Table 5 Comparison of the present invention patents in the test set ETIS

[0134]

[0135]

Claims

1. An endoscopic polyp segmentation method based on multi-scale feature aggregation, characterized in that: The following steps are involved: Step 1: Preparation of endoscopic image dataset; There are five endoscopic image datasets, namely the ETIS dataset, the Kvasir-SEG dataset, the CVC-ClinicDB dataset, the CVC-ColonDB dataset, and the CVC-300 dataset. The endoscopic images in each dataset are divided into real endoscopic images and labels of manually marked polyp areas; Step 2: Data preprocessing; The endoscopic images of the Kvasir-SEG dataset and the CVC-ClinicDB dataset in step 1 are combined with the labels of the manually marked polyp areas. The labels mark the polyp areas as 1 and the background as 0 to achieve accurate supervised training. The images and labels are enhanced to enhance the model's performance under different lighting and imaging conditions. The dataset division follows the same standards as Polyp-PVT and PraNet: for example, the Kvasir-SEG dataset is divided into 900 training images and 100 test images, and the CVC-ClinicDB dataset is divided into 548 training images and 64 test images, ensuring the rationality of the training and testing division. At the same time, to verify the model's generalization ability on unseen data, external datasets such as CVC-ColonDB, ETIS, and CVC-300 are used for independent testing. Step 3: Initialize the PVT-MSFF model Based on step 2, a three-stage progressive architecture is used, which consists of a PVT encoder, a Feature Enhancement Module (FEM), a Multi-Scale Fusion Module (MSFM), a Global Context Sensing (GCS), and a cascade fusion module CFM. In addition, a multi-scale fusion module (MSFM) is used to enhance the multi-level feature output from the encoder. After the image is input into the PVT encoder, features are obtained from the four stages of PVT. In PVT encoding, a single image is processed to Four levels of features are obtained, where i∈{1,2,3,4}, and S=2 i+1 and the number of channels C selected from the set {64,128,320,512} i ,The model is divided into three stages. The first stage is the low-level feature extraction stage, the second stage is the high-level semantic aggregation stage, and the third stage is the cross-level feature fusion stage; In the low-level feature extraction stage, the high-resolution X1 with rich edge and texture information is passed through the MSFM module to obtain X′1 after multi-scale feature fusion, and then processed by the FEM module to obtain features full of boundary information. In the high-level semantic aggregation stage, the number of channels {X2, X3, X4} is first adjusted to 32 channels, and then input into MSFM respectively to obtain {X′2, X′3, X′4} with multi-scale features. The high-level semantic features and low-level semantic features are input into CFM for fusion to obtain In the cross-level feature fusion stage, the output of the first stage is and the second stage output After global context awareness GCS, we get Finally, R1 and R2 are added together to obtain the final prediction result R of the model. In addition, during the training process, the loss of R1 and R2 relative to the ground truth (GT) is added together as the training loss; Step 4: Endoscopic image polyp segmentation task; After initialization in step 3, the endoscopic image preprocessed in step 2 is first input into the Pyramid Vision Transformer (PVT) encoder, which uses its hierarchical pyramid structure to extract multi-scale, low-level detail features, capture edges, and low-level texture information, laying the foundation for subsequent segmentation. Subsequently, the network further processes the features output by the encoder through the Multi-Scale Fusion Module (MSFM) and the Feature Enhancement Module (FEM). The MSFM dynamically weights and fuses information of different scales through a lightweight convolution chain and the SE-FF submodule. Furthermore, FEM focuses on using the channel attention mechanism to enhance edge information extraction, suppress noise, and retain low-level details, thereby improving the ability to capture tiny or fuzzy polyps. Secondly, GCS combines global perception and local perception, and uses global average pooling and local feature preservation to integrate low-level details and high-level semantic information, improving the overall feature expression capability and ensuring that the segmentation results meet high standards in terms of regional coherence and boundary refinement. Finally, after further optimization by the Cascade Fusion Module (CFM), the network comprehensively refines the aforementioned features and outputs an accurate pixel-level segmentation map. The entire process of step 3 ensures that each input image can obtain a segmentation result that is strictly aligned with the ground truth, effectively identifying the entire polyp and highlighting its boundary details. Step 5: Set the optimizer and loss function; Select an appropriate optimizer and loss function based on the complexity of the model obtained in step 4 and the requirements of the endoscopic image polyp segmentation task. The specific solution is as follows: AdamW combines the adaptive gradient update and weight decay mechanism of Adam. Let the model parameter obtained in step 4 be θ. t After the gradient is calculated, the update process is as follows: (1) First-order momentum estimation m t =β1m t-1 +(1-β1)g t (2) Second-order momentum estimation (3) Deviation correction (4) Parameter update formula Where η represents the learning rate (e.g. 1×10 -4 ), λ w represents the weight attenuation coefficient, ∈ is a small constant to prevent division by zero; Loss calculation, after feature extraction, uses the cross entropy loss function K CE To calculate Y pretrain and the true label Y train The error between, the cross entropy loss function is one of the commonly used loss functions in segmentation problems, and the loss function is expressed as: Among them, y i represents the true category label, Represents the predicted probability, M is the number of samples, and the accuracy of the model prediction can be quantified by calculating the loss function; Optimize model parameters. Use the Adam optimizer to update the model weights and combine it with the exponential decay learning rate scheduler to optimize model parameters. The learning rate adjustment formula is as follows: or t =η0·exp(-λ·t) Among them, η t is the learning rate of the tth round, η0 is the initial learning rate, and λ is the decay rate parameter; Step 6: PVT-MSFF model training and saving; In the training phase, the endoscopic images and corresponding labels after preprocessing and data augmentation in step 2 are used. The labels are single-channel binary masks. The PVT-MSFF model is trained in combination with the parameters in step 5, so that step 6 can learn effective features and make accurate predictions. The trained PVT-MSFF model is saved for further preparation for subsequent testing. The training loop is an iterative process. The training loop is repeated from i=1 to N until the model parameters reach the optimal state. The training loop is the core of the model training process. Through continuous iterative optimization, the model can gradually learn the inherent laws and characteristics of the data. In each iteration, the model will go through a complete forward propagation and backpropagation process, thereby gradually adjusting its internal parameters and improving prediction accuracy. Model saving: After the training cycle is repeated N times, the model is trained. At this time, the parameters of PVT-MSFF have been adjusted to the optimal state, and the prediction performance of the model has reached the expected target. The optimal model is saved. Step 7: PVT-MSFF model testing phase The test set sample x test Input into the model trained in step 6. In this stage, only forward propagation is performed, and backpropagation and parameter update are not required. Step 7 The model outputs the predicted feature vector y for each test sample predict By comparing the feature vector with the segmentation threshold, the model generates the corresponding prediction label; The predicted label obtained in step 7 and the true label y of the test set test Comparison, used to evaluate the performance of the model.

2. The method for polyp segmentation in endoscopic images based on a multi-scale feature hybrid pyramid transformer according to claim 1, characterized in that: The MSFM module in step 3 improves multi-scale perception; Pyramid Vision Transformer (PVT) encoder, which extracts multi-scale low-level and high-level features from the input pre-processed endoscopic image; Feature Enhancement Module (FEM), which uses a dual-branch collaborative attention mechanism to weight features through channel attention while preserving low-level detail information (such as edges and textures) to improve the expression of edge information; The Multi-Scale Fusion Module (MSFM) uses a lightweight 3×3 convolution chain and a compression-excitation feature fusion (SE-FF) unit to achieve dynamic fusion and adaptive adjustment of features at different scales, solving the problems of fixed weight allocation and channel redundancy. The Global Context-Sensing Module (GCS) combines global average pooling operations with local feature preservation strategies to achieve cross-layer adaptive fusion of low-level details and high-level semantic information; Cascade Fusion Module (CFM), which is used to integrate the features extracted and fused by the aforementioned modules to generate pixel-level segmentation predictions that are highly corresponding to the polyp area in the endoscopic image; In the PVT-MSFF model, each module works together according to the strategy of deepening and strengthening layer by layer. First, the Pyramid Vision Transformer (PVT) encoder extracts multi-scale low-level and high-level features from the input endoscopic image, capturing edge texture and global semantic information; Subsequently, the Feature Enhancement Module (FEM) uses a dual-branch attention mechanism to enhance edge details in low-level features, improving the perception of tiny polyps and fuzzy boundaries. The Multi-Scale Fusion Module (MSFM) further integrates features of different scales and dynamically adjusts channel weights to address scale imbalance and information redundancy issues in feature fusion. Then, the Global Context Perception Module (GCS) achieves cross-level adaptive fusion of global semantics and local details based on this, taking into account the overall structure and edge accuracy of the polyp; Finally, the cascade fusion module (CFM) uniformly optimizes the features of each layer to generate high-quality pixel-level segmentation prediction maps. PVT-MSFF can effectively cope with the challenges of diverse polyp morphologies, blurred boundaries and noise interference, significantly improving the accuracy and robustness of segmentation.

3. The endoscopic polyp segmentation method based on multi-scale feature aggregation according to claim 2, characterized in that: Channel statistics are extracted through global context perception, and dynamic channel weights are generated through fully connected layers and Sigmoid activation.

4. The endoscopic polyp segmentation method based on multi-scale feature aggregation according to claim 2, characterized in that: The feature enhancement module extracts nonlinear spatial features. Meanwhile, the multi-scale fusion module (MSFM) uses parallel SE-FF units to compress and excite the output of the lightweight 3×3 convolution chain, dynamically adjusting the feature weights of each channel to eliminate the checkerboard noise introduced by cross-scale connections and enhance the ability to capture key semantic information.

Images