Polyp image segmentation method and system based on parallel double-encoder feature fusion

Abstract

The application provides a polyp image segmentation method and system based on parallel double-encoder feature fusion, the scheme improves the convolution feedforward network of the Convformer model by introducing two layers of transposed fully connected layers combined with softmax activation, not only replaces the original point convolution operation, but also effectively fuses local and global information, significantly improves the performance of the Convformer model; at the same time, the improved Convformer module and Swin-Transformer module constitute a double-encoder structure, through the feature fusion of the two encoders, the efficiency and accuracy of information extraction are significantly improved, and the accuracy of image segmentation is improved; at the same time, the scheme proposes a feature fusion strategy based on the self-attention mechanism, effectively fuses the features of two independent encoders Convformer and Swin-Transformer, and improves the performance of image segmentation.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, and in particular relates to a polyp image segmentation method and system based on parallel dual encoder feature fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Colonic polyps, as precursors to colorectal cancer, play an indispensable role in cancer prevention through timely identification and accurate segmentation. Currently, with the advancement of medical imaging technology, the use of computer-aided diagnostic systems for the detection and segmentation of colonic polyps has become a popular research topic.

[0004] Deep learning-based models have demonstrated immense potential in medical image segmentation because they can learn complex image features and deliver accurate segmentation results in a variety of tasks, from segmenting specific anatomical structures to identifying pathological regions. For example, in the image segmentation of colon polyps, traditional polyp segmentation methods are increasingly being replaced by deep learning-based approaches, which have achieved significant improvements in segmentation accuracy. Convolutional Neural Networks (CNNs), one of the core technologies of deep learning, have played a crucial role in this field. For instance, CNN-based models such as U-Net, ResUnet, and PraNet are highly effective at capturing detailed polyp information. Compared to the U-Net image segmentation method, the DeepLab model, through its unique dilated convolution and spatial pyramid pooling (ASPP) techniques, increases the model's receptive field while maintaining high-resolution features. However, the receptive field of CNNs is limited, and capturing the overall shape of polyps remains a challenge.

[0005] The introduction of Transformer technology has revolutionized image processing, particularly medical image analysis. This technology significantly enhances the capture of global information by establishing long-distance connections between image pixels through its unique attention mechanism. In particular, the emergence of Swin-Transformer, with its window-based attention mechanism, effectively alleviates the dependence of traditional Transformers on large datasets while balancing the processing of local details and global information. This balance makes Swin-Transformer perform exceptionally well in high-precision medical image segmentation tasks. However, when processing images within fixed-size image patches, Swin-Transformer may miss some detailed features while capturing global information. Furthermore, in the field of medical imaging, the efficiency of Transformers largely depends on large, high-quality datasets. However, due to the limited availability of medical datasets and the tendency of networks combining CNNs and Transformers to converge to the learning patterns of CNNs, there is a risk of attention collapse, which may limit further improvements in model performance. Meanwhile, the innovative Convformer model cleverly combines the local perception capabilities of CNNs with the global processing capabilities of Transformers. The core feature of Convformer is its CNN-style self-attention mechanism, which allows the model to customize the convolutional kernel for each pixel. This approach enhances the model's sensitivity to local features and strengthens its ability to process global information. However, the point convolutional structure used in the Convolutional Feedforward Network (CFFN) may have certain limitations when integrating information from different regions of the image. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a polyp image segmentation method and system based on parallel dual-encoder feature fusion. The scheme improves the convolutional feedforward network of the Convformer model by introducing two transposed fully connected layers combined with softmax activation. This not only replaces the original point convolution operation but also effectively fuses local and global information, significantly improving the performance of the Convformer model. Simultaneously, the improved Convformer module and the Swin-Transformer module form a dual-encoder structure. Through feature fusion of the two encoders, the efficiency and accuracy of information extraction are significantly improved, thereby enhancing the accuracy of image segmentation.

[0007] According to a first aspect of the present invention, a polyp image segmentation method based on parallel dual encoder feature fusion is provided, comprising:

[0008] Obtain the polyp image to be segmented;

[0009] Based on the polyp image, a pre-trained deep learning-based image segmentation network model is used to obtain the polyp image segmentation result; wherein, the image segmentation network model specifically performs the following processing steps:

[0010] Based on the polyp image, feature extraction is performed using a preset neural network model to obtain a basic feature map; the basic feature map is used as input to a first encoder and a second encoder respectively to obtain a first feature map and a second feature map; based on the first feature map and the second feature map, feature fusion is performed using a self-attention mechanism to obtain a fused feature map; based on the obtained fused feature map, an image segmentation result is obtained using a preset decoder; wherein, the first encoder adopts an improved Convformer encoder, and the point convolution in the convolutional feedforward network of the improved Convformer encoder is replaced by: sequentially passing through a transposed fully connected layer and an activation function before entering the point convolution.

[0011] Furthermore, the improved Convformer encoder incorporates two transposed fully connected layers and a softmax activation function to fuse local and global information.

[0012] Furthermore, the transposed fully connected layer is specifically obtained by performing a linear algebraic transpose operation on the weight matrix of a traditional fully connected layer.

[0013] Furthermore, the feature fusion through a self-attention mechanism to obtain a fused feature map is specifically as follows:

[0014] Based on the first feature map, generate its corresponding key and value;

[0015] Based on the second feature map, generate its corresponding query;

[0016] Based on the key and query, an attention weight is generated;

[0017] Based on the attention weights and values, and combined with the first feature map, a fused feature map is obtained.

[0018] Furthermore, the second encoder is a Swing-Transformoer encoder.

[0019] Furthermore, the preset decoder employs a CNN network.

[0020] Furthermore, the basic feature map is obtained by extracting features through a preset neural network model, specifically using a CNN neural network.

[0021] According to a second aspect of the present invention, a polyp image segmentation system based on parallel dual encoder feature fusion is provided, comprising:

[0022] Image acquisition unit, which is used to acquire images of polyps to be segmented;

[0023] The segmentation unit is used to obtain polyp image segmentation results based on the polyp image using a pre-trained deep learning-based image segmentation network model; wherein the image segmentation network model specifically performs the following processing steps:

[0024] Based on the polyp image, feature extraction is performed using a preset neural network model to obtain a basic feature map; the basic feature map is used as input to a first encoder and a second encoder respectively to obtain a first feature map and a second feature map; based on the first feature map and the second feature map, feature fusion is performed using a self-attention mechanism to obtain a fused feature map; based on the obtained fused feature map, an image segmentation result is obtained using a preset decoder; wherein, the first encoder adopts an improved Convformer encoder, and the second encoder adopts a Swin-Transformer encoder, and the point convolution in the convolutional feedforward network of the improved Convformer encoder is replaced by: sequentially passing through a transposed fully connected layer and an activation function before entering the point convolution.

[0025] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the polyp image segmentation method based on parallel dual encoder feature fusion.

[0026] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the polyp image segmentation method based on parallel dual encoder feature fusion.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] (1) This invention provides a polyp image segmentation method and system based on parallel dual encoder feature fusion. The scheme improves the convolutional feedforward network of the Convformer model by introducing two transposed fully connected layers combined with softmax activation. This not only replaces the original point convolution operation, but also effectively fuses local and global information, significantly improving the performance of the Convformer model. At the same time, the improved Convformer module and the Swin-Transformer module form a dual encoder structure. Through feature fusion of the two encoders, the efficiency and accuracy of information extraction are significantly improved, thus improving the accuracy of image segmentation.

[0029] (2) This invention provides a feature fusion strategy based on self-attention mechanism, which effectively fuses the features of two independent encoders, Convformer and Swin-Transformer, and improves the performance of image segmentation.

[0030] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0031] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0032] Figure 1 These are schematic diagrams illustrating three typical challenge scenarios in the current polyp segmentation described in this embodiment of the invention;

[0033] Figure 2 This is a schematic diagram of the network architecture used in the polyp image segmentation method based on parallel dual encoder feature fusion described in this embodiment of the invention.

[0034] Figure 3 This is a schematic diagram of the improved Convformer encoder structure described in an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram of the feature fusion process described in an embodiment of the present invention.

[0036] Figure 5 This is a schematic diagram of the polyp segmentation results in five datasets as described in an embodiment of the present invention.

[0037] Figure 6 This is a schematic diagram of the segmentation results of different layers of Swin-transformer described in the embodiments of the present invention, where "Swin" represents Swin-transformer, "L" represents Layer, and the number represents the number of layers.

[0038] Figure 7 The results are the test results using the Kvasir dataset as described in the embodiments of the present invention.

[0039] Figure 8 This is a schematic diagram of the heat map analysis results described in an embodiment of the present invention.

[0040] Figure 9 This is a schematic diagram of polyp segmentation results for images selected from multiple datasets in different environments, as described in an embodiment of the present invention.

[0041] Figure 10This is a schematic diagram of the heat map analysis results under different encoders as described in the embodiments of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0046] Terminology Explanation:

[0047] A fully connected layer (FC) is a basic layer in a neural network, where every node between the input and output is fully connected.

[0048] A transposed fully connected layer refers to performing a linear algebraic transpose operation on the weight matrix of a traditional fully connected layer. This operation aims to enhance feature extraction and processing capabilities by changing the structure of the weight matrix.

[0049] Example 1:

[0050] The purpose of this embodiment is to provide a polyp image segmentation method based on parallel dual encoder feature fusion.

[0051] like Figure 1 The images illustrate three typical challenging scenarios in polyp segmentation, where "Image" refers to the original colonoscopy image and "Mask" refers to the standard overlay labeled by the physician based on clinical experience. (a) shows a case where the polyp boundaries are blurred, making segmentation in image processing more difficult; (b) depicts a complex scenario with multiple polyps, requiring precise differentiation and segmentation of each polyp; and (c) shows a case of a complex and varied sampling environment.

[0052] To address the problems existing in the prior art, this embodiment mainly adopts the following technical concept: A comprehensive sequence-to-sequence network architecture (hereinafter referred to as DTFNet) is designed as the image segmentation network model. This network architecture first uses a pre-defined neural network model (CNN in this embodiment) to perform preliminary processing on the input image, aiming to extract basic features. Subsequently, to enhance the fusion of local and global information, a key improvement is made to the Convolutional Feedforward Network (CFFN) part of the Convformer module. The traditional point convolution is replaced with a transposed fully connected layer, followed by softmax activation, and finally point convolution. It should be noted that the transposed fully connected layer, by transposing the weight matrix of the fully connected layer, can change the way features are passed and combined, thereby enhancing the model's expressive power. The input of this network architecture first undergoes preliminary feature extraction through a feature extractor, outputting a feature map; then, the feature map is simultaneously input into the improved Convformer and Swin-Transformer, which act as opposing encoders. Subsequently, a method similar to self-attention is employed to achieve more effective fusion of these two types of features, during which the size of the feature map remains unchanged. Finally, the feature map is input into the decoder to generate a segmentation mask. The structure and working principle of the entire network are as follows: Figure 2 As shown. Figure 2 As shown, CBR represents a combination of convolutional, batch normalization, and rectified linear unit (ReLU) activation functions. A multilayer perceptron (MLP) consists of two fully connected layers and Gaussian error linear units (GeLUs). GeLU is a non-linear activation function typically used in hidden layers to non-linearly transform the input.

[0053] Specifically, a polyp image segmentation method based on parallel dual encoder feature fusion performs the following processing steps:

[0054] Step 1: Obtain the image of the polyp to be segmented;

[0055] Step 2: Based on the polyp image, obtain the polyp image segmentation result using a pre-trained deep learning-based image segmentation network model;

[0056] The image segmentation network model specifically performs the following processing steps:

[0057] Based on the polyp image, feature extraction is performed using a preset neural network model to obtain a basic feature map; the basic feature map is used as input to a first encoder and a second encoder respectively to obtain a first feature map and a second feature map; based on the first feature map and the second feature map, feature fusion is performed using a self-attention mechanism to obtain a fused feature map; based on the obtained fused feature map, an image segmentation result is obtained using a preset decoder; wherein, the first encoder adopts an improved Convformer encoder, and the second encoder adopts a Swin-Transformer encoder, and the point convolution in the convolutional feedforward network of the improved Convformer encoder is replaced by: sequentially passing through a transposed fully connected layer and an activation function before entering the point convolution.

[0058] In its implementation, the overall architecture of the Convformer encoder is as follows: First, a pooling module is used to extract detailed local information, and downsampling is used to reduce computational cost, thus preparing for the Channel Spatial Attention (CSA) module in the CNN Attention section of the Convformer encoder. The CSA module then creates an adaptive receptive field for each pixel, enhancing its ability to capture global information through custom convolutional kernels. For input features... Figure X For each pixel in the convolution kernel, the kernel size is calculated using the following formula:

[0059] First, a learnable projection matrix E is used. q and E k The surrounding pixel neighborhood is processed. This calculation produces Q with local spatial information. (i,j) and K (i,j) :

[0060]

[0061] Where Q is the query vector in the self-attention mechanism, K is the key vector in the self-attention mechanism, l, g, and g′ represent the preset range, and x is the pixel.

[0062] In this embodiment, the learnable projection matrix is ​​defined as a fixed-dimensional matrix and filled with learnable parameters. The projection matrix can be optimized according to data characteristics during model training, thereby improving the model's expressive power and performance. The learnable parameters can be updated through backpropagation.

[0063] After initial processing, the Q of the input image is calculated. (i,j) and K (i,j)The cosine similarity between components determines their matching degree. Cosine similarity refers to the cosine value of the angle between two non-zero vectors in multidimensional space, which is used to determine the similarity between the two vectors. The value of cosine similarity ranges from -1 to 1, where -1 indicates that the two vectors are opposite, 0 indicates that they are not related (orthogonal), and 1 indicates that the vectors are in the same direction (i.e., completely similar).

[0064]

[0065] Among them, C q denoted as embedding, and m and n represent the indices of the two-dimensional convolution.

[0066] To dynamically determine the convolution kernel size for different pixels, a Gaussian matrix M is introduced, where the standard deviation and M(i,j) = I(i,j) control the trend of receptive field size variation. The unique final convolution kernel size for each pixel is given by the following formula.

[0067]

[0068] Where d is the size of the convolution kernel, H is the height of the feature map, W is the width of the feature map, θ is a learnable parameter used to control the size of the receptive field, and α is a parameter used to control the scaling of the receptive field size.

[0069] Based on the above design, the dynamic adjustment of the convolution kernel size for each pixel is achieved through the interaction of Q and K and the calculation of cosine similarity, thereby better capturing local and global information and improving the network's performance in image segmentation tasks.

[0070] Inspired by the MLP-Mixer, this embodiment innovatively improves the CFFN module in the traditional Convformer to optimize the segmentation of colonic polyps. While the traditional CFFN module performs well in capturing local information through two layers of point convolution, it often fails to adequately capture the global information that is equally important for polyp segmentation. To address this issue, we propose a method combining MLP and convolution. Our network uses MLP modules as the core component for feature transformation, with each module containing two fully connected layers. The specific steps are as follows:

[0071] First, input feature map x∈R n×c×h×w Reshape as X = reshape(X) T The input is defined as (n, h × w, c), where n is the batch size (the number of images input to the model at one time), h and w represent the height and width of the model image, and c represents the number of channels. Then, it is processed by the designed MLP module X = MlpBlock(X) and reshaped back to its original input form. Afterwards, the channel dimension X = reshape(X) is reshaped. TThe feature transformation is performed on (n×h×w,c), and another MLP operation is executed before restoring the initial shape. The processed features... Figure X The result S is obtained by normalization using the Softmax function and then element-wise multiplied with the original input X = S ⊙ X. Finally, the output is obtained by point convolution Y = Conv2d(X). Subsequent experiments show that this method can more effectively integrate global and detailed information, thereby improving model performance. The specific architecture is as follows: Figure 3 As shown.

[0072]

[0073] Among them, z i `i` is the i-th element in the input feature map, representing the raw, unnormalized score of each category or feature before the Softmax operation. The expression represents the summation of the exponential operations on all input features. Here, `K` is the total number of elements in the input feature map. This step calculates the sum of all exponential operations, used to normalize the value of each feature.

[0074] In practical implementation, the introduction of the Swin-Transformer encoder plays a crucial role in improving polyp segmentation performance. As an efficient Transformer architecture, Swin-Transformer emphasizes sensitivity to detail through its unique window partitioning mechanism and hierarchical feature extraction design. This characteristic is particularly complementary to Convformer. This deep focus on local details is essential for detecting small and structurally complex polyps, making Swin-Transformer an important addition to our model when tackling the challenges of medical image segmentation.

[0075] In specific implementation, the feature fusion through the self-attention mechanism is as follows:

[0076] This is achieved by operating on two independent encoders: Convformer and Swin-Transformer. For the feature map generated by the Convformer... A We apply C 1×1 Generate keys and values. Correspondingly, for the feature map generated by the Swin-Transformer... B We generate queries. Using the dot product and SoftMax function, we compute attention weights and then apply them to the values. In the final stage of the fusion process, we combine the weighted feature maps with the original feature maps. AElement-wise addition effectively combines these two features. This fusion strategy allows the model to integrate the strengths of both encoders. Through extensive experiments and tuning, we observed that this variant of the self-attention mechanism performs exceptionally well in integrating global vision and local details, thus significantly improving the overall performance of the model. Specifically, it is represented as follows:

[0077] Q = Conv2d query (Feature B ),

[0078] K,V=Conv2d (k,v) (Feature A ),

[0079]

[0080] Out = Attention ⊙V,

[0081] FusedMap = feature A +Out.

[0082] Where, d k This is the scaling factor.

[0083] In its implementation, the decoder employs a classic CNN network architecture, consisting of multiple convolutional layers, batch normalization layers, and Rectified Linear Unit (ReLU) activation functions. This effectively reduces the number of channels in the feature map while gradually increasing the spatial dimension. The decoder reconstructs the spatial resolution through upsampling and transposed convolution operations to match the original image, thus laying the foundation for final detail reconstruction. Finally, a specially designed convolutional layer outputs the segmentation map, enabling the model to effectively distinguish between polyp and non-polyp regions. Despite its simplicity, this structure ensures efficient and accurate polyp segmentation in the overall model.

[0084] Furthermore, to verify the effectiveness of the solution described in this embodiment, the following related experiments were conducted:

[0085] (1) Experimental Dataset

[0086] In this embodiment, the combined datasets of Kvasir-SEG and CVC-ClinicDB were used, as mentioned in the PraNet experimental setup. This dataset comprises two subsets: Kvasir-SEG (900 training samples) and CVC-ClinicDB (550 training samples). For benchmarking, this embodiment selected five datasets as shown in Table 1: Kvasir-SEG, CVC-ClinicDB, ETIS, CVC300, and CVC-ColonDB. Of these five datasets, all except Kvasir-SEG are out-of-distribution datasets. For research and learning purposes, this embodiment divided the combined dataset into two parts: one for training and the other for validation. Training and validation accounted for 80% and 20%, respectively. This data partitioning method was used for evaluation before benchmarking our model on various datasets.

[0087] Table 1 introduces the relevant datasets used in the experiment.

[0088]

[0089] (2) Data Augmentation

[0090] To enhance the model's robustness to changes in image orientation and position, we applied two dynamic data augmentation methods in this study. The first method involves random rotation and flipping. An integer K is randomly selected from 0 to 3 and used to rotate the image and label by 90 degrees. Subsequently, an axis (horizontal or vertical) is randomly selected, and the image and label are flipped. The second method involves random angle rotation, allowing the image and label to rotate randomly between -20 and 20 degrees, thereby increasing the angular diversity of the training samples. Each image has a 50% probability of having one of these methods applied. By introducing variations in orientation and position, this strategy aims to improve the model's adaptability to changes in real-world clinical images, thereby enhancing its performance and generalization ability in the colon polyp segmentation task.

[0091] (3) Loss function setting

[0092] In our research, for the polyp segmentation task, we employed a loss function combining 0.5 times the Dice loss and 0.5 times the binary cross-entropy loss. This strategy leverages the advantages of the Dice loss in addressing class imbalance and the accuracy of the binary cross-entropy loss in probabilistic prediction. The combination of these methods enhances the model's generalization ability when handling complex polyp segmentation datasets. The application of this strategy demonstrates our rigor in selecting the loss function, which is significant for improving the performance of the polyp segmentation model.

[0093]

[0094] Where y represents the probability that the true label is 0 or 1. This represents the probability of predicting a label of 0 or 1.

[0095]

[0096] Where X is the result of model segmentation, and Y is the ground truth mask.

[0097]

[0098] (4) Experimental setup

[0099] In this experiment, a loss function combining 0.5 times the Dice loss and 0.5 times the binary cross-entropy loss was employed for the polyp segmentation task. This strategy leverages the advantages of the Dice loss in addressing class imbalance and the accuracy of the binary cross-entropy loss in probability prediction. The combination of these methods enhances the model's generalization ability when handling complex polyp segmentation datasets. The application of this strategy demonstrates the rigor in selecting the loss function, which is significant for improving the performance of the polyp segmentation model.

[0100] (5) Evaluation indicators

[0101] In this study, the Intersection over Union (IOU), Dice coefficient, and Mean Absolute Error (MAE) were used to evaluate the model proposed in this embodiment. These metrics were used to assess the consistency between the predicted mask and the true mask on the test dataset. Higher consistency indicates better model performance.

[0102]

[0103] Intersection over Union (IOU) is evaluated by calculating the ratio of the overlapping area between the predicted and ground truth masks to the merged area. A higher IOU value indicates a greater degree of overlap between the predicted and ground truth masks, reflecting the model's high accuracy in image segmentation tasks.

[0104]

[0105] Mean Absolute Error (MAE) is evaluated by calculating the absolute difference between the predicted and ground truth masks at each pixel and then averaging these values. A lower MAE value indicates a smaller difference between the predicted and ground truth masks, thus suggesting higher prediction accuracy. This provides an intuitive and effective metric for evaluating model performance.

[0106]

[0107] (6) Experimental Results

[0108] 1) Qualitative results

[0109] To comprehensively evaluate the performance and capabilities of the model proposed in this embodiment, a detailed comparative analysis was conducted with previous models. Specifically, as shown in Table 2, the model was first internally tested on the Kvasir-SEG and CVC-ClinicDB datasets, yielding encouraging results. Furthermore, to further investigate the model's generalization ability, extensive external tests were performed using the CVC-300, ETIS, and CVC-ColonDB datasets, as shown in Tables 3, 4, and 5. These test results demonstrate the performance of the scheme described in this embodiment on different datasets, while also proving its strong generalization ability.

[0110] Table 2 shows the experimental results on the Kvasir-SEG and CVC-ClinicDB datasets.

[0111]

[0112] Table 3 shows the experimental results on the CVC-ColonDB dataset.

[0113]

[0114]

[0115] Table 4 shows the experimental results on the CVC-300 dataset.

[0116]

[0117] Table 5 shows the experimental results on the ETIS dataset.

[0118]

[0119]

[0120] In particular, the model described in this embodiment demonstrates satisfactory performance on test sets beyond the Kvasir-SEG dataset, especially excelling in key performance metrics such as mean intersection-over-union ratio (mIoU) and mean Dice coefficient (mDice). These results indicate that the model possesses superior ability to segment small and irregularly shaped polyps. However, it is worth noting that our model scores relatively high on the MAE metric. The increase in MAE suggests that the model may be overly sensitive to detail in certain specific cases, particularly in the accurate prediction of boundary pixels.

[0121] Analysis of the data in Tables 6-1 and 6-2 shows that when using only the Swin-Transformer as the encoder, the model performs similarly across the datasets. However, performance significantly degrades when the number of encoder layers increases to three; we hypothesize that this phenomenon may stem from a breakdown in the attention mechanism. To further investigate this hypothesis, this embodiment analyzes heatmaps of Swin-Transformer models with different numbers of layers. The analysis shows that with one or two encoder layers, the model can accurately locate polyps. However, when the number of layers increases to three, an abnormal attention distribution occurs, causing the model to lose its ability to recognize polyps. Considering the balance between the number of parameters and model performance, this experiment selects a single-layer Swin-Transformer for subsequent experiments to achieve the optimal balance between model efficiency and accuracy.

[0122] Table 6-1 Segmentation results when only the Win-Transformer is used as the encoder.

[0123]

[0124] Table 6-2 Segmentation results when only the Win-Transformer is used as the encoder.

[0125]

[0126] To determine the optimal number of encoder layers, this experiment progressively increased the number of layers in the enhanced MConvformer, recording the model's performance and computational cost. Test results using the Kvasir dataset are shown below. Figure 7 As shown, the relationship between the model's mDice score and the number of parameters is illustrated. The results clearly show that increasing the number of layers does not significantly improve encoder performance. However, the model achieves the optimal balance between performance and parameter count when using only a single-layer MConvformer. Figure 7 This shows that limiting the model to a single layer can effectively control the parameter size while maintaining model performance.

[0127] In addition, heatmap analysis was performed in this experiment, such as... Figure 8 As shown in the figure, although the MConvformer alleviates the attention mechanism collapse problem to some extent, the heatmap distribution indicates that increasing the number of layers may still lead to attention collapse. Therefore, we decided to use a single layer of Swin-Transformer and MConvformer in parallel as the encoder in the subsequent polyp segmentation task.

[0128] Furthermore, the scheme described in this embodiment performs heatmap analysis on the image. Although Convformer alleviates the attention mechanism failure problem to some extent, as... Figure 7As shown in the heatmap distribution, the attention collapse problem can still occur as the number of layers increases.

[0129] (2) Ablation results

[0130] In this ablation study, this embodiment provides a comprehensive analysis and evaluation of five encoder configurations. These include:

[0131] 1. The classic Convformer encoder (denoted as Conf)

[0132] 2. Optimized Convformer encoder (MConf)

[0133] 3. Integration of traditional Convformer and Swin-Transformer (Conf+Swin)

[0134] 4. Existing models used for benchmark comparison (MConf+Swin)

[0135] In the following discussion, we will use the abbreviations of these models to demonstrate their performance advantages and features.

[0136] Tables 7-1 and 7-2 detail the performance of various models on the datasets, clearly demonstrating the performance differences introduced by the ablation experiments. The results show that the fusion model successfully combines the advantages of the Swin and Conf architectures, thus surpassing their performance when used individually as encoders. Particularly noteworthy is our innovative improvement to the CFFN layer of Conf, which enabled our model to outperform Conf+Swin on most test sets, except for the ETIS dataset. Ultimately, our developed MConf+Swin model achieved satisfactory results on both known and unknown datasets.

[0137] To comprehensively and intuitively evaluate our model, we selected images from multiple datasets across different environments. This selection took into account factors such as whether the images were cropped, their regularity, and their contrast levels. Figure 9The performance of the model described in this embodiment is further demonstrated. First, it is clear that using Swing as the sole encoder yields relatively poor results. Compared to the traditional Conf model, our improved MConf shows a significant improvement in the clarity of polyp boundary recognition, effectively preventing oversegmentation. In eliminating irrelevant tissue in the background, our designed feature fusion mechanism effectively combines the feature maps of Conf and Swing, thus surpassing the performance of Conf and Swing models used individually. More importantly, the MConf model exhibits higher recognition accuracy in these challenging scenarios. By integrating the features of MConf and Swing, we effectively address the challenges faced by Conf+Swin in segmenting polyps and tissues with similar colors, thereby significantly improving the model's performance. This fusion enhances the model's ability to handle complex images and strengthens its practical value in various clinical settings.

[0138] Table 7-1 Ablation Experiment Results

[0139]

[0140] This embodiment provides a more intuitive heatmap analysis of the effects of various encoders, and the results are as follows: Figure 10 As shown in the figure, the analysis reveals that our optimized Convformer (MConf) is more effective at capturing both global details and local features of polyps compared to the original Convformer (Conf), thus reducing segmentation errors to some extent. Furthermore, through our designed feature fusion mechanism, the parallel operation of the dual encoders can more accurately handle the detection of multiple polyps and more precisely depict their edges. This result validates that our feature fusion strategy effectively integrates the advantages of both encoders, achieving accurate polyp segmentation while minimizing errors.

[0141] Table 7-2 Ablation Experiment Results

[0142]

[0143] (7) Experimental Conclusion

[0144] This embodiment introduces an innovative dual-encoder network, DTFNet, specifically designed for colon polyp segmentation. This model is optimized for various challenges in polyp segmentation, such as the diversity of polyp shapes and sizes, the similarity of tissue and polyp colors, and contrast variations during image acquisition. DTFNet integrates an improved Convformer and Swin-Transformer as parallel dual encoders, effectively achieving deep fusion of local and global features. This model significantly improves the accuracy and robustness of polyp segmentation.

[0145] The performance of the proposed solution on various datasets was evaluated through comprehensive testing of DTFNet on five current mainstream datasets. Multiple comparative experiments and ablation analyses further confirmed DTFNet's superior performance in accurate segmentation of colon polyps. These experimental results demonstrate the efficient segmentation capabilities of the proposed model and highlight its potential applications in clinical image segmentation solutions.

[0146] Example 2:

[0147] The purpose of this embodiment is to provide a polyp image segmentation system based on parallel dual encoder feature fusion.

[0148] A polyp image segmentation system based on parallel dual encoder feature fusion includes:

[0149] Image acquisition unit, which is used to acquire images of polyps to be segmented;

[0150] The segmentation unit is used to obtain polyp image segmentation results based on the polyp image using a pre-trained deep learning-based image segmentation network model; wherein the image segmentation network model specifically performs the following processing steps:

[0151] Based on the polyp image, feature extraction is performed using a preset neural network model to obtain a basic feature map; the basic feature map is used as input to a first encoder and a second encoder respectively to obtain a first feature map and a second feature map; based on the first feature map and the second feature map, feature fusion is performed using a self-attention mechanism to obtain a fused feature map; based on the obtained fused feature map, an image segmentation result is obtained using a preset decoder; wherein, the first encoder adopts an improved Convformer encoder, and the second encoder adopts a Swin-Transformer encoder, and the point convolution in the convolutional feedforward network of the improved Convformer encoder is replaced by: sequentially passing through a transposed fully connected layer and an activation function before entering the point convolution.

[0152] Furthermore, the relevant technical details of the system described in this embodiment have been described in detail in Embodiment 1, and therefore will not be repeated here.

[0153] In further embodiments, the following is also provided:

[0154] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0155] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0156] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0157] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0158] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0159] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0160] The polyp image segmentation method and system based on parallel dual encoder feature fusion provided in the above embodiments can be implemented and has broad application prospects.

[0161] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A polyp image segmentation method based on parallel dual encoder feature fusion, characterized in that, include: Obtain the polyp image to be segmented; Based on the polyp image, a pre-trained deep learning-based image segmentation network model is used to obtain the polyp image segmentation result; wherein, the image segmentation network model specifically performs the following processing steps: Based on the polyp image, feature extraction is performed using a preset neural network model to obtain a basic feature map; the basic feature map is used as input to a first encoder and a second encoder respectively to obtain a first feature map and a second feature map; based on the first feature map and the second feature map, feature fusion is performed using a self-attention mechanism to obtain a fused feature map; based on the obtained fused feature map, an image segmentation result is obtained using a preset decoder; wherein, the first encoder adopts an improved Convformer encoder, and the point convolution in the convolutional feedforward network of the improved Convformer encoder is replaced by: sequentially passing through a transposed fully connected layer and an activation function before entering the point convolution; The improved Convformer encoder introduces two transposed fully connected layers and combines them with the softmax activation function to fuse local and global information. The transposed fully connected layers are specifically obtained by performing a linear algebraic transpose operation on the weight matrix of the traditional fully connected layer.

2. The polyp image segmentation method based on parallel dual encoder feature fusion as described in claim 1, characterized in that, The feature fusion through a self-attention mechanism to obtain a fused feature map is specifically as follows: Based on the first feature map, generate its corresponding key and value; Based on the second feature map, generate its corresponding query; Based on the key and query, an attention weight is generated; Based on the attention weights and values, and combined with the first feature map, a fused feature map is obtained.

3. The polyp image segmentation method based on parallel dual encoder feature fusion as described in claim 1, characterized in that, The second encoder is a Swing-Transformer encoder.

4. The polyp image segmentation method based on parallel dual encoder feature fusion as described in claim 1, characterized in that, The preset decoder uses a CNN network.

5. The polyp image segmentation method based on parallel dual encoder feature fusion as described in claim 1, characterized in that, The basic feature map is obtained by extracting features through a preset neural network model, specifically using a CNN neural network.

6. A polyp image segmentation system based on parallel dual encoder feature fusion, characterized in that, include: Image acquisition unit, which is used to acquire images of polyps to be segmented; The segmentation unit is used to obtain polyp image segmentation results based on the polyp image using a pre-trained deep learning-based image segmentation network model; wherein the image segmentation network model specifically performs the following processing steps: Based on the polyp image, feature extraction is performed using a preset neural network model to obtain basic feature maps. These basic feature maps are then used as inputs to a first encoder and a second encoder to obtain a first feature map and a second feature map, respectively. Based on the first and second feature maps, feature fusion is performed using a self-attention mechanism to obtain a fused feature map. Based on the obtained fused feature map, an image segmentation result is obtained using a preset decoder. The first encoder uses an improved Convformer encoder, and the second encoder uses a Swin-Transformer encoder. In the improved Convformer encoder, the point convolutions in the convolutional feedforward network are replaced with sequential passes through a transposed fully connected layer and an activation function before entering the point convolutions. The improved Convformer encoder introduces two transposed fully connected layers and combines them with the softmax activation function to fuse local and global information. The transposed fully connected layers are specifically obtained by performing a linear algebraic transpose operation on the weight matrix of the traditional fully connected layer.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements a polyp image segmentation method based on parallel dual encoder feature fusion as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a polyp image segmentation method based on parallel dual encoder feature fusion as described in any one of claims 1-5.

Images