Real-time polyp segmentation system based on multi-domain hierarchical attention network

Abstract

The invention discloses a real-time polyp segmentation system based on a multi-domain hierarchical attention network, and relates to the field of medical image processing and computer vision, and the system comprises a dynamic region guide block which is used for partitioning an input image and selecting key region features through a two-stage routing attention mechanism; the potential entropy quantization channel space attention module is used for carrying out channel and space information entropy quantization calculation on the feature map and highlighting fine organization differences; the spatial frequency fusion module is used for performing pixel-level fusion on the spatial domain features and frequency domain features extracted through fast Fourier convolution, and capturing a periodic mode and global structure information; and the patch extension layer and the linear projection are used for layer-by-layer up-sampling and executing quadruple up-sampling in the final stage to recover to the input resolution, and pixel-level segmentation prediction is generated. The boundary precision and the pixel classification performance are both improved, and the real-time performance of polyp segmentation and the clinical application reliability are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision, and in particular to a real-time polyp segmentation system based on a multi-domain hierarchical attention network. Background Technology

[0002] With the widespread clinical application of endoscopy, the early detection and removal of colorectal polyps has become a crucial means of reducing the incidence of colorectal cancer. Traditional manual identification methods rely on the doctor's experience and real-time observation, which can easily lead to visual fatigue during prolonged operation, resulting in a decreased polyp recognition rate. Furthermore, different polyps vary in size, shape, and boundary clarity, making it difficult to detect some small or flat polyps in a timely manner. Therefore, how to utilize computer vision technology to assist endoscopic diagnosis and improve the efficiency and accuracy of polyp detection and segmentation has gradually become a research focus.

[0003] In recent years, deep learning-based medical image segmentation methods have attracted widespread attention. Classical convolutional neural networks can extract local spatial features, but they have limitations in capturing long-range dependencies and global information, and are prone to misjudgments in complex backgrounds. To address this, some studies have introduced attention mechanisms to enhance the response of key regions during feature extraction, thereby improving sensitivity to subtle targets. However, conventional attention mechanisms are computationally expensive when dealing with high-resolution inputs, making them unsuitable for real-time segmentation, and their effectiveness in region selection and detail preservation is limited.

[0004] In polyp segmentation, it is crucial to balance global structural information with local details. Existing methods often prioritize one feature dimension, such as relying solely on spatial domain convolution to capture local patterns or using frequency domain transformations to focus on the overall structure, but they struggle to utilize features from multiple domains simultaneously. Furthermore, the downsampling process inevitably leads to the loss of spatial details, resulting in blurred boundaries for small polyps or their inclusion in the background, which can affect the reliability of diagnosis in clinical applications. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, this invention provides a real-time polyp segmentation system based on a multi-domain hierarchical attention network to solve the problems of high computational overhead, insufficient boundary recognition, and low segmentation accuracy in existing polyp segmentation methods in terms of real-time region selection, preservation of downsampling details, and fusion of spatial frequency features.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a real-time polyp segmentation system based on a multi-domain hierarchical attention network, comprising: a dynamic region guidance block for partitioning the input image and selecting key region features through a two-level routing attention mechanism to suppress irrelevant background features;

[0009] The latent entropy quantization channel spatial attention module is used to quantify and calculate the channel information entropy and spatial information entropy of the feature map after processing by the dynamic region guide block, highlighting the subtle organizational differences of key channels and spatial regions.

[0010] The spatial frequency fusion module is used to perform pixel-level fusion of spatial domain features with frequency domain features extracted by fast Fourier convolution to capture periodic patterns and global structural information.

[0011] Output: Through multiple patch expansion layers and linear projection, the resolution of multi-scale features is restored step by step during the layer-by-layer upsampling process. The last patch expansion layer performs a four-fold upsampling to restore the H×W resolution of the input image, and pixel-level segmentation prediction is generated through linear projection.

[0012] As a preferred embodiment of the real-time polyp segmentation system based on a multi-domain hierarchical attention network described in this invention, the dynamic region guidance block includes: establishing an initial feature representation through a 1×1 convolution and a 3×3 depth convolution for position encoding, and performing a nonlinear transformation by combining the SE activation function with the 1×1 convolution;

[0013] An LN layer is set before the two-stage routing attention mechanism to normalize the input features;

[0014] The features are expanded and refined by combining three residual connections with a multilayer perceptron with an inflation factor.

[0015] The query and key regional averages are calculated through region-to-region routing, and an adjacency matrix is ​​constructed. The matrix is ​​then used to optimize the information flow and select the top-k relevant regions.

[0016] Perform token-to-token sparse attention at a fine-grained level to focus on key feature regions and suppress irrelevant information.

[0017] As a preferred embodiment of the real-time polyp segmentation system based on a multi-domain hierarchical attention network described in this invention, the two-level routing attention mechanism includes dividing the feature map into S×S non-overlapping regions and generating linear projections of query Q, key K, and value V at the region scale respectively.

[0018] The average value of the query and key is calculated by region-to-region routing to obtain the region query and region key, and the adjacency matrix A is constructed by matrix multiplication.

[0019] By combining matrix I to optimize the information flow, the top-k relevant regions are selected for each query, and the relevant regions are determined by row-by-row selection.

[0020] Collect key and value tensors within the selected relevant regions, and perform token-to-token attention at a fine-grained level, generating attention results using depthwise convolution parameters.

[0021] As a preferred embodiment of the real-time polyp segmentation system based on a multi-domain hierarchical attention network described in this invention, the latent entropy quantization channel space attention module includes independently applying global normalization on each channel of the input feature map to obtain a statistical measure of information distribution.

[0022] The normalized results of each channel are used to calculate the two-dimensional information entropy, and a sequence of entropy values ​​connected along the channel axis is formed.

[0023] Channel weights are determined by batch normalization and the Sigmoid activation function, and then multiplied element-wise with the initial feature map to generate a channel-weighted feature map.

[0024] As a preferred embodiment of the real-time polyp segmentation system based on a multi-domain hierarchical attention network described in this invention, the latent entropy quantization channel spatial attention module includes extracting local spatial features using two consecutive 7×7 convolutional layers based on the channel weighted feature map, and generating spatial feature representations through global average pooling.

[0025] Spatial attention weight map is computed using a convolutional network and normalized using a sigmoid activation function;

[0026] The spatial attention weight map and the channel weighted feature map are multiplied element-wise to obtain the final weighted feature map, which is used to compensate for the loss of spatial details during the downsampling process.

[0027] As a preferred embodiment of the real-time polyp segmentation system based on a multi-domain hierarchical attention network according to the present invention, the spatial frequency fusion module includes: a spatial convolution branch for extracting local spatial features;

[0028] Fast Fourier convolution branch is used to extract global frequency features;

[0029] The fusion of local spatial features and global frequency features is achieved through pixel-level attention guidance.

[0030] As a preferred embodiment of the real-time polyp segmentation system based on a multi-domain hierarchical attention network described in this invention, the fast Fourier convolution branch includes performing a two-dimensional fast Fourier transform on the spatial features to obtain a frequency domain representation.

[0031] Perform convolution operations in the frequency domain to extract frequency features;

[0032] By using the inverse two-dimensional fast Fourier transform, frequency features are mapped back to the spatial domain, thus achieving the corresponding fusion of frequency information and spatial information.

[0033] As a preferred embodiment of the real-time polyp segmentation system based on a multi-domain hierarchical attention network described in this invention, the training process employs a combination of Dice loss and cross-entropy loss, specifically including:

[0034] By constraining the boundaries of the segmented regions using Dice loss, the boundary accuracy of the segmentation results can be improved.

[0035] The classification performance is optimized by using cross-entropy loss to improve the accuracy of pixel-level classification.

[0036] We set weighting coefficients to balance the Dice loss and cross-entropy loss, thereby improving both boundary segmentation accuracy and classification performance.

[0037] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in the first aspect of the present invention.

[0038] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in the first aspect of the present invention.

[0039] The beneficial effects of this invention are as follows: By introducing a two-level routing attention mechanism in the dynamic region guide block, key features are selected at the regional scale and combined with token-level sparse attention, improving the real-time performance and effectiveness of feature extraction; by using a latent entropy quantization channel spatial attention module, entropy quantization is used in the channel dimension to enhance the response to subtle differences, and convolution and global average pooling are used in the spatial dimension to generate a spatial weight map, effectively compensating for the loss of detail caused by downsampling; by using a spatial frequency fusion module, local spatial features and frequency domain features are fused at the pixel level, enabling the model to capture both local texture and global structural information; and by combining the balance optimization of Dice loss and cross-entropy loss during training, the segmentation results are improved in both boundary accuracy and pixel classification performance, thereby improving the real-time performance and reliability of polyp segmentation in clinical scenarios. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is the overall architecture diagram of MDHA-Net.

[0042] Figure 2 This is a network structure diagram of the Dynamic Region Bootstrap Block (DRGBlock).

[0043] Figure 3 for Figure 3 A demonstration diagram of the Latent Entropy Quantized Channel Spatial Attention Module (LECSA).

[0044] Figure 4 This is a framework diagram of the Spatial Frequency Fusion Module (SFFM).

[0045] Figure 5 A comparison of visual segmentation results from different methods on the Kvasir-SEG and CVC-ColonDB datasets.

[0046] Figure 6 A comparison of visual segmentation results of different methods on the CVC-ClinicDB, CVC-300, and ETIS-Larib datasets.

[0047] Figure 7 The image shows a comparison of visual segmentation results from ablation experiments on the Kvasir-SEG dataset. In this image, (I) represents the original image, (II) represents the ground reality, (III) shows the segmentation results of UNet, (IV) illustrates the segmentation results of the backbone, (V) shows the results of integrating SFFM into the backbone, (VI) describes the segmentation after adding LECSA to the backbone, and (VII) represents the final segmentation results achieved by MDHA-Net. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0051] Example 1, referring to Figures 1-4 This is one embodiment of the present invention, which provides a real-time polyp segmentation system based on a multi-domain hierarchical attention network, including the following steps:

[0052] This invention proposes a multi-domain hierarchical attention network (MDHA-Net) for polyp segmentation, the specific architecture of which is as follows: Figure 1 As shown, the entire framework consists of multiple modules, including the Dynamic Region Guiding Block (DRG block), the Latent Entropy Quantized Channel Spatial Attention Module (LECSA), and the Spatial Frequency Fusion Module (SFFM). These modules centrally and hierarchically integrate multi-domain information from the channel, spatial, and frequency domains to improve the accuracy of polyp segmentation.

[0053] Given an input image, it is initially segmented into overlapping regions. The feature dimension of each region is projected onto an arbitrary dimension C through patch embedding. In a continuous hierarchical structure, Dynamic Region Guided Blocks (DRG blocks) sequentially partition the features, prioritizing important regions and discarding trivial background, thereby generating a more complete feature representation. As the hierarchy increases, the feature map resolution gradually decreases, while the number of channels increases accordingly. Figure 1 In levels I to III, the features extracted by the dynamic region guide block are further input into the latent entropy quantized channel spatial attention module (LECSA) for processing. This module quantifies the importance of the channel and the richness of the context information in the channel domain through two-dimensional information entropy, and optimizes the context expression by combining cross-channel connection information entropy. It captures local details and enhances key region features in the spatial domain, while suppressing irrelevant background information, thereby improving the accuracy of polyp region localization and identification.

[0054] exist Figure 1 In Level IV, the Spatial-Frequency Fusion (SFFM) module jointly processes spatial and frequency domain information. Under pixel-level attention guidance (Pag), this module fuses the features output by the spatial convolution branch and the fast Fourier convolution branch, enabling precise correspondence and complementarity between local spatial information and global frequency features, thereby enhancing the ability to capture periodic patterns and global structural information.

[0055] exist Figure 1In the final stage, the features are upsampled and reduced in dimensionality layer by layer through patch expansion layers, gradually backtracking to a high-resolution representation. During this process, multi-scale features from dynamic region guiding blocks and contextual features from the latent entropy quantized channel spatial attention module are integrated, effectively compensating for the spatial information loss caused by downsampling. Finally, the last patch expansion layer performs a fourfold upsampling, restoring the input image to H×W resolution, and generates pixel-level segmentation prediction results through linear projection.

[0056] In related embodiments, the visual transducer (ViT) performs well in capturing long-range dependencies at different feature scales, but its computational complexity increases with sequence length, affecting computation time and memory overhead. Inspired by the sparse attention concept in BiFormer, a dynamic region guiding block (DRG block) was designed to focus the model's attention on polyp regions through efficient feature selection and dynamic attention mechanisms. Figure 2 As shown, the DRG block begins with a 1×1 convolution, followed by a 3×3 depthwise convolution for positional encoding, then an SE activation function followed by another 1×1 convolution. An LN layer is placed before the two-stage routing attention and integrated with three residual connections. A multilayer perceptron (MLP) with an inflation factor e=3 is used to further expand and refine the feature representation. The core of the DRG block is the two-stage routing attention mechanism, which is dynamic and query-oriented sparsity. The two-stage routing attention operates at the region scale, acting as a selective sieve to discard semantically irrelevant key-value pairs, retaining only the subset most critical to subsequent fine-grained attention.

[0057] Specifically, the feature map is divided into S×S non-overlapping regions, and the derived query Q, key K, and value V are linearly projected onto the regions to obtain region representations. Region-to-region routing obtains the region query Q by calculating the regional average of the query and key. r With regional bond K r Furthermore, an adjacency matrix A is constructed through matrix multiplication to optimize the information flow and select the most relevant region for each query. A matrix I is introduced to represent the index of the selected region. This selection is achieved through a row-by-row top-k operation, implemented by the function topkIndex(), with the formula:

[0058] I r =topkIndex(Q r (K r ) T )

[0059] Among them, I r Represents the region index matrix; topkIndex(·) represents the row-by-row top-k selection function; Q r Represents the region query matrix; K r Represents the region bond matrix; (K r )T Q represents the transpose of the region key matrix; r (K r ) T This represents the region similarity matrix.

[0060] The processing flow of topkIndex() is as follows: Query the region Q r With regional bond K r Linear layer embeddings are used to learn more meaningful representations; the scaled dot product of the embedding query and the embedding key is calculated to obtain the attention logarithm, which reflects the similarity between regions; row-wise top-k selection is applied to the attention logarithm along the last dimension to obtain the top k highest similarity values ​​and their corresponding indices; softmax is applied to the top k attention logarithm tensors to obtain the region weights used to weight the selected regions, thereby focusing the model's attention on the selected top k regions and effectively weakening the influence of regions with low relevance.

[0061] In obtaining the distributed routing area index I r Then, the corresponding terms are collected from the original key tensor K and value tensor V, denoted as:

[0062] K g =gather(K,I) r ),V g =gather(V,I) r ).

[0063] Subsequently, token-to-token attention is performed on the tokens at a fine-grained level, calculated as follows:

[0064]

[0065] Where C represents the feature channel dimension, LCE(·) is the local context enhancement function supporting this process, and LCE(·) is instantiated by the depthwise convolution parameters to provide additional local context information enhancement; K g V represents the collected key matrix; g The matrix represents the collected values; gather(·) represents the collection operation; K represents the original key matrix; V represents the original value matrix; O represents the final output feature matrix; softmax(·) represents the softmax activation function; Q represents the original query matrix. Indicates the scaling factor; This represents the scaled dot product attention score matrix.

[0066] The above steps work together within the DRG block to achieve efficient screening and fine-grained focus on key regions, thereby improving the ability to characterize polyp regions while maintaining computational efficiency.

[0067] like Figure 3As shown, the distribution of contextual information in feature maps plays a crucial role in accurate attention localization, while existing mechanisms often overlook the changes in pixel value frequency. Feature maps typically consist of multiple channels, and the channel domain refers to the information carried by each channel in these multi-channel feature maps. Channel domain information can reflect the importance of different features. By quantifying the information entropy of channels, we can identify the channels that are more critical in polyp segmentation tasks, thereby reducing the interference of irrelevant features. Based on this, we propose the Latent Entropy Quantized Channel Spatial Attention Module (LECSA). This module describes the distribution of contextual information among feature maps through information entropy calculation and compensates for the spatial information loss caused by downsampling in the spatial domain, thereby achieving more effective feature capture at both the local and global structural levels.

[0068] This process begins with the input feature map, denoted as:

[0069]

[0070] Where W, H, and C represent the width, height, and number of channels of the input feature map, respectively. LECSA first applies F... input Each channel is independently subjected to global normalization to obtain a statistical measure of information distribution. For the i-th channel, the normalization result is expressed as...

[0071]

[0072] i∈[1,C]

[0073] The specific calculation method is as follows:

[0074]

[0075] in f represents the i-th channel of the input feature map. max with f min These represent the maximum and minimum values ​​within the channels, respectively. This normalization operation adjusts the relative magnitudes of the values ​​in different channels, thereby ensuring that the attention mechanism can more effectively highlight key features.

[0076] Based on this, the expression for calculating the two-dimensional information entropy is as follows:

[0077]

[0078] in, The entropy value for the i-th channel represents the magnitude of information distribution in that channel. The entropy values ​​of all channels are connected along the channel axis to form an overall information entropy description. Subsequently, channel weights are obtained through batch normalization (BN) and the sigmoid activation function. These weights are then used to perform element-wise weighting on the initial feature map to obtain the channel attention-weighted feature map F.X To further model the spatial context, LECSA... X Two consecutive 7×7 convolutional layers are applied to extract local spatial features. Then, global average pooling is used to aggregate the extracted features, and a spatial attention weight map is calculated using a small convolutional network. The obtained attention weights are normalized using a sigmoid activation function and then compared with F... X Element-wise multiplication yields the final weighted feature map F. output .

[0079] like Figure 4 As shown, abnormal growth of polyps causes periodic changes in tissue structure, thus affecting specific frequency components. The Spatial-Frequency Fusion (SFFM) module utilizes two-dimensional Fast Fourier Transform (FFT) to perform frequency domain analysis on the image structure in order to capture overall information in the frequency domain. Fast Fourier Convolution (FFC) helps the model detect and recognize subtle features that are difficult to observe directly in the spatial domain. The SFFM module consists of two basic branches: an upper branch using FFC and a lower branch using conventional spatial convolution operations. The outputs of the two branches are fused through pixel-level attention guidance (Paq) to achieve dynamic coordination between spatial and frequency domain features, thereby improving the discriminative power of the final output.

[0080] Specifically, the input feature map F input It is simultaneously transmitted to both the spatial domain branch and the frequency domain branch. The spatial domain branch is denoted as X. spatial This branch contains a 3×3 convolutional layer at both ends, and each convolutional layer combines batch normalization (BN) and the LeakyReLU activation function to extract local spatial features. The frequency domain branch is denoted as X. frequency This branch first maps conventional spatial features to the frequency domain using a two-dimensional fast Fourier transform (FFT) to capture remote context information, and then reprojects the frequency domain features back to the spatial domain using an inverse two-dimensional fast Fourier transform.

[0081] During the fusion process, the components from X will be respectively... frequency With X spatial pixel-to-vector representation and Apply the function mapping f to each of the two vectors. f (·) and f s (·), and then the fusion coefficients are obtained through the Sigmoid activation function:

[0082]

[0083] Here, δ represents the probability that two pixels belong to the same object. A higher δ value indicates that the frequency domain feature X... frequencyThe higher the confidence level, the richer the semantic information and the higher the discrimination accuracy of this branch. Conversely, if the δ value is low, the spatial domain feature X... spatial Their contributions are even more important.

[0084] Where Sigmoid(·) represents the Sigmoid activation function, which maps input values ​​to the interval [0, 1]; f f (·) denotes the mapping function of the frequency domain branch, for Perform nonlinear transformation; Indicates that it comes from the frequency domain branch X frequency The pixel-corresponding vector; f s (·) denotes the mapping function for the spatial domain branch, for Perform nonlinear transformation; From spatial domain branch X spatial The pixel-to-vector.

[0085] Finally, the output of the Pag module can be formalized as:

[0086]

[0087] Among them, F output This represents the final output feature map after fusion; 1-δ represents the pixel vector from the spatial domain branch; 1-δ represents the weight factor of the frequency domain branch.

[0088] Through the aforementioned fusion mechanism, the SFFM module, while ensuring the fidelity of spatial details, fully utilizes the global expressive power of frequency domain features to achieve robust detection and segmentation of anomalous structures.

[0089] During the training phase, optimization is achieved by combining Dice loss and Cross-Entropy loss to balance performance in both segmentation and classification tasks. Dice loss measures the overlap between the predicted and ground truth regions, thus enhancing segmentation accuracy; Cross-Entropy loss measures the difference in classification probabilities between the predicted and ground truth labels, thereby improving classification accuracy. By weighting and combining these two loss functions, a more balanced performance can be obtained in boundary recognition and class discrimination. The mathematical expression for the Dice loss function is:

[0090]

[0091] The mathematical expression for the cross-entropy loss function is:

[0092]

[0093] The weighted combination of the two loss functions is as follows:

[0094] L=λL dice +(1-λ)L ce

[0095] Where N represents the total number of pixels, and K represents the total number of classes. G(k,i)∈(0,1) represents the ground truth label of the k-th class at pixel i, and P(k,i)∈[0,1] represents the predicted probability of the k-th class at pixel i. The parameter ω... k The weights corresponding to the k-th class satisfy the condition ∑ k ω k =1 to ensure balance between different categories. The parameter λ is a weighting coefficient used to adjust the relative contributions of Dice loss and Cross-Entropy loss to the overall loss function.

[0096] Among them, L dice Represents Dice loss; k represents the class index; ω k Let P(k,i) represent the weight of the k-th class; let i represent the pixel index; P(k,i) represent the predicted probability of the k-th class at pixel i; G(k,i) represent the ground truth label of the k-th class at pixel i; L ce λ represents the cross-entropy loss; L represents the total loss; and λ represents the weighting coefficient.

[0097] In medical image segmentation tasks, traditional methods typically rely on a single loss function for optimization, such as cross-entropy loss alone. This often leads to inaccurate identification of small targets or boundary regions when there is class imbalance. This approach introduces a joint optimization strategy of Dice loss and cross-entropy loss. Dice loss emphasizes the overlap between the predicted and ground truth regions, effectively addressing the issues of blurred target boundaries and the easy neglect of small regions. Meanwhile, cross-entropy loss maintains stable performance in class differentiation, resulting in a more balanced overall segmentation outcome.

[0098] When processing polyps or other tissues in medical images, the complex boundary morphology and subtle local features make traditional methods prone to edge recognition breaks or discontinuities. This approach, by introducing Dice loss, makes the model more sensitive to the weights of boundary regions during backpropagation, ensuring the integrity of fine-grained structures. Compared to single pixel-based classification, this strategy significantly reduces the false positive rate for edge regions and improves the ability to capture complex contours.

[0099] Traditional methods are susceptible to background noise interference when faced with different patient samples and varying imaging conditions, leading to unstable segmentation results. This invention achieves a reasonable allocation of weights for each category through weight constraints, enabling the model to adaptively balance the importance of various features during training and reducing the negative impact of class imbalance. This design ensures stable model performance under different sample distributions, improving overall robustness.

[0100] The joint loss function dynamically adjusts the relative contributions of Dice loss and cross-entropy loss through weighting coefficients, playing a complementary role in tasks where background and target regions coexist. Compared with traditional single optimization objectives, this composite optimization approach can simultaneously ensure boundary accuracy and class discrimination ability, thereby achieving higher sensitivity and specificity in real-world applications and significantly improving the practical value of medical image segmentation.

[0101] Example 2, as Figures 5-7 As one embodiment of the present invention, a real-time polyp segmentation system based on a multi-domain hierarchical attention network is provided. To verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0102] To verify the effectiveness of our proposed MDHA-Net, experiments were conducted using the five benchmark datasets in Table 1: CVC-ClinicDB, CVC-300, CVC-ColonDB, ETIS-Larib, and Kvasir-SEG.

[0103] CVC-ClinicDB: It contains 612 images of colon polyps, also known as CVC-612, obtained from 25 colonoscopy videos. The image size is 288×384. CVC-300: It is a subset of the EndoScene dataset, consisting of 300 images taken from colonoscopy videos. CVC-ColonDB: CVCColonDB is a small database containing 380 images. ETISLarib: ETIS-Larib is an older and challenging dataset consisting of 196 polyp images. The image size is 1255×966. KvasirSEG: Kvasir-SEG is a recently released and challenging dataset consisting of 1000 polyp images. The image sizes in the Kvasir-SEG dataset vary, ranging from 332×487 to 1920×1072. Generally, 90% of the dataset is used for training and 10% for testing.

[0104] Table 1 Benchmark Data Set

[0105] Dataset Train Test ImageSize Number Kvasir-SEG √ √ 332×487-1920×1072 1000 CVC-ColonDB √ √ 500×574 380 CVC-ClinicDB × √ 288×384 612 ETIS-Larib × √ 966×1225 196 CVC-300 × √ 500×574 60

[0106] To ensure a fair comparison, this study used a training set consisting of 1,242 images sampled from the Kvasir-SEG and CVC-ColonDB datasets, while retaining the remaining 138 images from Kvasir-SEG and CVC-ColonDB as a test set. Other datasets were used to evaluate the generalization ability of MDHA-Net.

[0107] A comprehensive comparison was conducted using five widely accepted evaluation metrics: Dice, mIoU, Accuracy, Precision (P), and Recall (R). These metrics are divided into two groups. Dice and mIoU measure regional similarity and evaluate segmented objects. Recall, Precision, and Accuracy are pixel-level metrics used to quantify the difference between predicted segmentation and ground truth annotation. For clarity, we use the following notation conventions: TP represents true positives, TN represents true negatives, FP represents false positives, and FN represents false negatives. Therefore, the formulas for calculating these five performance metrics are provided below:

[0108]

[0109] The higher the evaluation score for these metrics, the better the segmentation performance.

[0110] The MDHA-Net model and several ablation variants of this invention were trained on an Nvidia GeForce RTX 4090 graphics card equipped with 24GB of memory. The method of this invention was implemented using Python 3.10 and PyTorch 2.1.0. All images were resized to a uniform resolution of 256×256 pixels. The Adam optimizer was used to train all models for a total of 200 epochs, with a batch size set to 16. A cosine annealing learning rate scheduler was employed, with an initial learning rate of 5e-4. To enhance the diversity of the training data, data augmentation techniques were applied, including horizontal flipping, vertical flipping, rotation, and cropping, each with a probability of 0.25. Specific details of the hyperparameters are shown in Table 2.

[0111] Table 2 Hyperparameter Table

[0112]

[0113]

[0114] The proposed MDHA-Net was compared with classic models such as UNet, Unet++, and PraNet, as well as state-of-the-art methods: UACANet, CFANet, SBANet, and CTNet. All methods were trained on the Kvasir-SEG and CVC-ColonDB datasets and then tested on their respective test subsets.

[0115] As shown in Table 3, MDHA-Net outperforms other methods in both mIoU and Dice coefficients on the two datasets. Notably, on the Kvasir-SEG dataset, MDHA-Net significantly improves mIoU by 3.89% and Dice coefficient by 2.78% compared to the suboptimal model CTNet. MDHA-Net improves accuracy by 1.21% and recall by 0.32%. MDHA-Net provides higher quality segmentation results in accurately identifying the shape and boundaries of polyps. The enhancements are also significant on the CVC-ColonDB dataset, with mIoU improved by 2.78% and Dice coefficient by 2.88%. MDHA-Net demonstrates more substantial improvements across all three metrics: accuracy improved by 1.64%, recall by 1.69%, and precision by 2.74%. These results indicate that MDHA-Net exhibits higher accuracy and reliability in polyp identification and segmentation.

[0116] Table 3. Segmentation performance comparison between the Kvasir-SEG dataset and the CVC-ColonDB dataset.

[0117]

[0118]

[0119] When comparing segmentation performance, CFANet and SBANet are strong competitors, but they still fall short of MDHA-Net's segmentation capabilities. While CTNet excels at capturing multi-scale information, it cannot match the segmentation accuracy of MDHA-Net. Our model's strength lies in its ability to analyze subtle differences between polyps and surrounding tissue in the channel space domain. The inclusion of the Spatial Frequency Fusion (SFFM) module is crucial for providing frequency attributes that are essential for distinguishing polyp tissue from its surroundings, and is vital for enhancing segmentation.

[0120] It is worth noting that the present invention MDHA-Net can achieve a final result of less than 31.79 FPS on an RTX4090 GPU, which ensures that MDHA-Net can be used for colonoscopy video.

[0121] Figure 5 The qualitative results visually demonstrate the segmentation performance of MDHA-Net. To highlight segmentation differences, we use green to represent correctly segmented regions, red to represent over-segmented regions, and yellow to represent under-segmented regions. Figure 5In rows 1-3, the polyps exhibit high texture similarity to the background. Algorithms like CFANet and CTNet often over-segment. The segmentation of the polyp tissue by these algorithms is also discontinuous, such as... Figure 5 As shown in rows 1-3, columns 3-5. In contrast, the MDHA-Net of this invention demonstrates superior performance in accurately segmenting polyp tissue. Figure 5 In rows 4-6, these pedunculated or subpedunculated polyps have different internal colors. In these cases, the comparison method cannot distinguish polyp regions and is more prone to misidentification due to the lack of global information, such as... Figure 6 As shown in rows 4-6, columns 3-6. However, the model of this invention performs well because it can obtain global information about polyps from both the frequency and spatial domains.

[0122] To evaluate generalization ability, we conducted experiments on three unseen datasets: CVC-ClinicDB, CVC-300, and ETIS-Larib. The results for these datasets are detailed in Table 4.

[0123] Specifically, on the CVC-ClinicDB dataset, our method demonstrates a significant advantage in mIoU (90.96%), Dice coefficient (94.47%), and accuracy (98.40%). This represents a substantial leap forward in both mIoU and Dice metrics, achieving increases of 1.96% and 0.64%, respectively, in surpassing our closest competitor, CTNet. This advantage is primarily attributed to MDHA-Net's unique architectural design, particularly the DRG block, which prioritizes essential functionality while hiding irrelevant information. While CTNet slightly leads with a 98.65% precision, MDHA-Net closely follows, dominating with a 97.83% recall.

[0124] Table 4. Segmentation performance comparison of CVC-ClinicDB dataset, CVC-300 dataset, and ETIS-Larib dataset.

[0125]

[0126]

[0127] On the CVC-300 dataset, MDHA-Net demonstrates a more significant advantage, achieving an mIoU of 92.62%, a Dice coefficient of 96.16%, and an accuracy of 99.80%, showing a clear advantage over models such as CTNet. Furthermore, our method achieves a precision of 98.35% and a recall of 98.12% on this dataset. MDHA-Net's superior performance is attributed to its powerful feature extraction and fusion capabilities. The LECSA module effectively quantifies the information entropy of different channels and spatial locations, enhancing the model's ability to capture subtle changes in polyp tissue, thus resulting in more accurate segmentation results. The SFFM module further improves the recognition of complex structures and periodic patterns by analyzing frequency patterns and spatial texture, enabling the model to excel in handling various polyp morphologies present in the dataset.

[0128] For the ETIS-Larib dataset, MDHA-Net outperformed the comparison methods on all metrics. Compared to the second-best model, CTNet, MDHA-Net improved mIoU by 2.80%, Dice coefficient by 0.20%, Accuracy by 2.63%, Precision by 0.29%, and Recall by 1.37%. The ETIS-Larib dataset is extremely challenging due to color variations, lighting effects, motion blur, and reflections. In this complex environment, MDHA-Net stands out due to its multi-domain hierarchical attention architecture. The DRG block uses dynamic attention to sift key features, enabling rapid localization of potential polyp regions in complex backgrounds. The LECSA module enhances the understanding of polyp tissue by quantifying information entropy, accurately distinguishing polyps from surrounding tissue. The SFFM module analyzes frequency characteristics and spatial texture, capturing subtle periodic patterns to improve segmentation accuracy.

[0129] Green indicates a true positive (correct region segmentation), red indicates a false positive (oversegmentation of regions), and yellow indicates a false negative (undersegmentation of regions). Figure 6 The visualizations are particularly compelling because they demonstrate MDHA-Net's ability to accurately locate and segment polyps on challenging and diverse datasets. For polyps with indistinct boundaries, such as flat polyps or sessile polyps, as... Figure 6 As shown in rows 3-6, the polyp's color is similar to the surrounding background. Under these conditions, most models struggle to produce accurate segmentation results, such as... Figure 6 As shown in rows 3-6, columns 3-5. However, MDHA-Net's multi-domain hierarchical attention mechanism effectively extracts the channel, spatial, and frequency features of polyps, thus enabling accurate segmentation of polyp tissue, such as... Figure 6As shown in line 7. This demonstrates strong generalization ability and efficient processing of unseen data.

[0130] To validate the effectiveness of the proposed LECSA and SFFM modules, we conducted ablation studies on the Kvasir-SEG and CVC-ColonDB datasets to evaluate their respective contributions. A simple UNet was used as the benchmark for comprehensive comparison. Starting with a Backbone model consisting only of DRG modules (referred to as MDHANet-0), the LECSA and SFFM modules were progressively integrated. As shown in Table 5, including these modules significantly improved performance. Specifically, MDHA-Net outperformed UNet on all metrics on both the Kvasir-SEG and CVC-ColonDB datasets. In particular, compared to the Backbone model, MDHA-Net achieved significant improvements of 2.84% and 2.42% in mIoU and Dice, respectively, on the Kvasir-SEG dataset. Similarly, on the CVC-ColonDB dataset, MDHA-Net demonstrated an mIoU of 52% and a Dice of 2.74%. These results highlight the significant contributions of the LECSA and SFFM modules to improving segmentation accuracy and overall model performance across different datasets. The substantial improvements in five key metrics further validate MDHA-Net’s superior performance in prediction quality and overall consistency.

[0131] Table 5. Comparison of ablation experiment segmentation performance on Kvasir-SEG and CVC-ColonDB datasets.

[0132]

[0133]

[0134] The effect of the LECSA module: As shown in Table 5, integrating the LECSA module into the network significantly improved the model's performance. The LECSA module quantifies the information entropy across channels and spatial locations, enhancing contextual understanding in feature maps. This is crucial for distinguishing subtle changes between polyps and normal tissue.

[0135] The SFFM module significantly improves segmentation performance, as shown in Table 5. Operating in both the spatial and frequency domains, it facilitates multi-domain fusion of polyp features. Frequency domain analysis captures global structure and periodic patterns. This synergistic effect enhances the recognition of both local texture and overall structure, improving segmentation accuracy for polyp morphologies and sizes.

[0136] Figure 7This section displays visualizations of the modules applied to the Kvasir-SEG dataset. Columns show: original polyp image, ground truth, UNet model segmentation, Backbone model segmentation, Backbone+SFFM, Backbone+LECSA, and MDHA-Net segmentation results. The image sequence from the original polyp image to the final MDHA-Net segmentation result clearly illustrates the progressive refinement of segmentation quality. Initially, the UNet model not only exhibited undersegmentation, such as... Figure 7 As shown in the second column of the third row, oversegmentation is also evident, as shown in columns 1, 3, and 4 of the third row. This indicates that the UNet model struggles to accurately locate lesions. Next, the Backbone model can fully identify the lesion area, but it still exhibits oversegmentation, as shown in... Figure 7 As shown in line 4. By combining SFFM, it utilizes spatial and frequency information to capture the structure of polyps. This model reduces missegmentation around adjacent tissue and improves focus on lesions. Adding the LECSA module further improves boundary clarity by quantifying latent entropy information in the channel domain, thereby reducing incorrect segmentation. When both modules are used simultaneously, the boundaries become clearer, and the missegmented areas are significantly reduced.

[0137] Ablation studies confirm the substantial and complementary contributions of the LECSA and SFFM modules to the MDHA-Net architecture. Consistent improvements across different metrics and datasets validate the effectiveness of our proposed method. Visual evidence further reinforces the quantitative results, highlighting the robustness and accuracy of MDHA-Net in segmenting polyps in medical images. These results suggest that MDHA-Net may be a valuable tool for clinical applications, providing greater accuracy and reliability in polyp detection and segmentation.

[0138] MDHA-Net integrates information from the channel, spatial, and frequency domains in a hierarchical manner, which significantly improves the segmentation accuracy of traditional baselines and high-level models. Key features include:

[0139] (1) Significance of automated polyp segmentation: MDHA-Net achieves a real-time segmentation speed of 31.79 frames per second (FPS), meeting the real-time feedback requirements during endoscopic examinations. This enables immediate detection and localization of polyps. By reducing the variability associated with manual colonoscopy, it improves the polyp detection rate, thereby enhancing the accuracy and efficiency of diagnosis.

[0140] (2) Novelty and effectiveness of the proposed method: MDHA-Net integrates DRG block, LECSA and SFFM module, and surpasses the state-of-the-art models such as CTNet

[55] in terms of mIoU, Dice, Accuracy, Precision and Recall.

[0141] (3) Improved boundary segmentation accuracy: The DRG block employs a two-level routing attention approach to prioritize salient features while suppressing less relevant information, enabling the network to focus on critical segment regions. The LECSA module focuses on subtle polyp tissue differences, while the SFFM module integrates frequency attributes and spatial texture, thereby improving boundary delineation and segmentation accuracy.

[0142] (4) Multi-domain distribution design: Levels I to III (green background) process information from both the spatial and channel domains, transitioning from coarse to fine feature representations. This multi-scale approach helps capture details across various scales. In Level IV (blue background), feature resolution is minimized while the number of feature channels is maximized, indicating a shift in focus to fine-grained feature representations and increased channel diversity. The SFFM module at this level combines spatial convolution with Fast Fourier Convolution (FFC) to preserve spatial details and analyze feature variations across frequencies.

[0143] (5) Limitations and Future Directions: MDHA-Net was designed specifically to address the unique challenges of polyp image segmentation. SFFM aims to capture the periodic patterns and structural information of polyp tissue. In the future, we will develop a more efficient and adaptive frequency attention mechanism that can dynamically adapt to the frequency information of different medical images, thereby enhancing the versatility of MDHA-Net in other medical image applications.

[0144] This structured approach highlights the advancements MDHA-Net has made in automated polyp segmentation and its potential to enhance clinical outcomes in colorectal cancer detection.

[0145] In this work, we propose an innovative multi-domain hierarchical attention network for polyp segmentation, called MDHA-Net. Specifically, we first integrate multi-domain information from the channel, spatial, and frequency domains in a hierarchical manner. This innovative fusion strategy breaks through the limitations of traditional segmentation methods that rely solely on single-domain information, fully leveraging the complementarity of different domains. A Dynamic Region Guiding Block (DRG block) serves as the base layer, introducing a dynamic mechanism for region feature selection and analysis. It uses two layers of routed attention to prioritize salient features while suppressing less relevant information, enabling the network to focus on key segmentation regions. Following the DRG block, a Latent Entropy Quantized Channel Spatial Attention Module (LECSA) enhances the specificity of feature maps by quantizing subtle organizational variations in the channel and spatial domains. The Spatial-Frequency Fusion Module (SFFM), based on the quantized channel and spatial features depicted by LECSA, combines spatial domain features with frequency domain analysis to capture complex structures and patterns. This hierarchical fusion method of multi-domain information enables MDHA-Net to comprehensively understand polyp images from multiple dimensions, forming richer and more representative feature representations. This approach comprehensively represents polyps and improves the segmentation accuracy of different polyp morphologies.

[0146] Extensive experiments on five polyp segmentation datasets demonstrate the superiority of our MDHA-Net compared to other state-of-the-art segmentation methods. These rich and detailed experimental data fully demonstrate the superior performance of MDHA-Net in practical applications, providing a solid experimental foundation for its promotion and use in clinical practice. In the future, we will develop a more efficient and adaptive frequency attention mechanism that can dynamically adjust the frequency information of various medical images, thereby enhancing the versatility of MDHA-Net for other medical imaging applications.

[0147] This embodiment also provides a computer device applicable to a real-time polyp segmentation system based on a multi-domain hierarchical attention network, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the real-time polyp segmentation system based on a multi-domain hierarchical attention network as proposed in the above embodiment.

[0148] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0149] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements the real-time polyp segmentation system based on a multi-domain hierarchical attention network as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0150] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A real-time polyp segmentation system based on a multi-domain hierarchical attention network, characterized in that, include: Dynamic region guidance blocks are used to partition the input image and select key region features through a two-level routing attention mechanism to suppress irrelevant background features; The latent entropy quantization channel spatial attention module is used to quantify and calculate the channel information entropy and spatial information entropy of the feature map after processing by the dynamic region guide block, highlighting the subtle organizational differences of key channels and spatial regions. The spatial frequency fusion module is used to perform pixel-level fusion of spatial domain features with frequency domain features extracted by fast Fourier convolution to capture periodic patterns and global structural information. Output: Through multiple patch expansion layers and linear projection, the resolution of multi-scale features is restored step by step during the layer-by-layer upsampling process. The last patch expansion layer performs a four-fold upsampling to restore the H×W resolution of the input image, and pixel-level segmentation prediction is generated through linear projection.

2. The real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in claim 1, characterized in that: The dynamic region guidance block includes establishing an initial feature representation through a 1×1 convolution and a 3×3 depth convolution for position encoding, and performing a nonlinear transformation by combining the SE activation function with the 1×1 convolution; An LN layer is set before the two-stage routing attention mechanism to normalize the input features; The features are expanded and refined by combining three residual connections with a multilayer perceptron with an inflation factor. The query and key regional averages are calculated through region-to-region routing, and an adjacency matrix is ​​constructed. The matrix is ​​then used to optimize the information flow and select the top-k relevant regions. Perform token-to-token sparse attention at a fine-grained level to focus on key feature regions and suppress irrelevant information.

3. The real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in claim 2, characterized in that: The two-level routing attention mechanism includes dividing the feature map into S×S non-overlapping regions and generating linear projections of query Q, key K and value V at the region scale respectively. The average value of the query and key is calculated by region-to-region routing to obtain the region query and region key, and the adjacency matrix A is constructed by matrix multiplication. By combining matrix I to optimize the information flow, the top-k relevant regions are selected for each query, and the relevant regions are determined by row-by-row selection. Collect key and value tensors within the selected relevant regions, and perform token-to-token attention at a fine-grained level, generating attention results using depthwise convolution parameters.

4. The real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in claim 3, characterized in that: The latent entropy quantization channel spatial attention module includes independently applying global normalization to each channel of the input feature map to obtain a statistical measure of information distribution. The normalized results of each channel are used to calculate the two-dimensional information entropy, and a sequence of entropy values ​​connected along the channel axis is formed. Channel weights are determined by batch normalization and the Sigmoid activation function, and then multiplied element-wise with the initial feature map to generate a channel-weighted feature map.

5. The real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in claim 4, characterized in that: The latent entropy quantization channel spatial attention module includes extracting local spatial features using two consecutive 7×7 convolutional layers based on the channel weighted feature map, and generating spatial feature representations through global average pooling. Spatial attention weight map is computed using a convolutional network and normalized using a sigmoid activation function; The spatial attention weight map and the channel weighted feature map are multiplied element-wise to obtain the final weighted feature map, which is used to compensate for the loss of spatial details during the downsampling process.

6. The real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in claim 5, characterized in that: The spatial frequency fusion module includes: a spatial convolution branch to extract local spatial features; Fast Fourier convolution branch is used to extract global frequency features; The fusion of local spatial features and global frequency features is achieved through pixel-level attention guidance.

7. The real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in claim 6, characterized in that: The fast Fourier convolution branch includes performing a two-dimensional fast Fourier transform on the spatial features to obtain a frequency domain representation; Perform convolution operations in the frequency domain to extract frequency features; By using the inverse two-dimensional fast Fourier transform, frequency features are mapped back to the spatial domain, thus achieving the corresponding fusion of frequency information and spatial information.

8. The real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in claim 1, characterized in that: During training, a combination of Dice loss and cross-entropy loss is used, specifically including: By constraining the boundaries of the segmented regions using Dice loss, the boundary accuracy of the segmentation results can be improved. The classification performance is optimized by using cross-entropy loss to improve the accuracy of pixel-level classification. We set weighting coefficients to balance the Dice loss and cross-entropy loss, thereby improving both boundary segmentation accuracy and classification performance.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the real-time polyp segmentation system based on a multi-domain hierarchical attention network as described in any one of claims 1 to 7.

Images