A polyp segmentation method and system based on a double-branch feature fusion network
By using a dual-branch feature fusion network that combines the advantages of CNN and Transformer, the limitations of colonoscopy image segmentation models in global and local feature extraction are solved, achieving fine polyp segmentation and improving segmentation accuracy and generalization ability.
Patent Information
- Application Number
- CN202311315813.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-11
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2043-10-11
AI Technical Summary
Existing colonoscopy image segmentation models based on CNN and Transformer have limitations in extracting global and local features of images. CNN models lack global contextual information, while Transformer models are not good at modeling local information, resulting in poor segmentation performance.
A dual-branch feature fusion network is adopted, which extracts local fine features of the image through the CNN branch and extracts global features of the image through the Transformer branch. The features are then fused through the BiFusion feature fusion module. Combined with the SFA local progressive upsampling module and deep supervision strategy, a dual-weighted feature loss function is designed for training.
It improves the accuracy and generalization ability of polyp segmentation, eliminates semantic gaps, reduces feature loss, and significantly improves the segmentation accuracy of polyps in images.
Smart Images
Figure CN117237641B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a polyp segmentation method and a polyp segmentation system based on a dual-branch feature fusion network. Background Technology
[0002] Colorectal cancer (CRC) is the second most common cancer among women and the third most common cancer among men. By 2023, approximately 153,020 people were diagnosed with colorectal cancer, and 52,550 were projected to die from the disease, with 19,550 cases and 3,750 deaths occurring in individuals under the age of 50. Polyps are abnormal tissue growths on the lining of the gastrointestinal tract and can sometimes become cancerous. Furthermore, research indicates that adenomatous polyps are a major cause of colorectal cancer. Therefore, early diagnosis of polyps is crucial.
[0003] With the development of machine learning, especially deep learning, a series of deep learning-based solutions have been proposed to achieve automatic segmentation of colonoscopy images. These models can be roughly divided into two categories: CNN-based models represented by U-Net, and Transformer-based models represented by VisionTransformer.
[0004] However, both types of models have certain limitations: CNN-based models, which build hierarchical network models by training end-to-end networks, are very powerful in extracting fine local features of images. But due to the limitation of the receptive field, they lack the ability to capture global contextual information. To overcome this problem, the network needs to be continuously deepened, but this introduces new problems, such as gradient vanishing. Transformer-based models have recently become popular in semantic segmentation and have shown superior performance compared to CNN-based models. They obtain the similarity of all patch pairs through the dot product between patch vectors, thereby adaptively extracting and fusing features across all patches. This gives Transformers a more efficient global receptive field and better generalization ability. The first purely Transformer-based image recognition model was VisionTransformer, which converted images into a series of patches for serialization and achieved state-of-the-art (SOTA) results at the time. However, segmentation networks based solely on Transformers do not achieve satisfactory performance due to their insufficient ability to model local information and the need for training on large amounts of data. Summary of the Invention
[0005] To address the aforementioned problems, this invention provides a polyp segmentation method and system based on a dual-branch feature fusion network. Considering the limited sample size and difficulty in distinguishing foreground and background in the Kvasir-SEG dataset, a spatially exclusive pasting data preprocessing method is used for data augmentation. A dual-branch parallel network architecture is employed, using a CNN branch to extract local fine features from the image and a Transformer branch to extract global features. A BiFusion feature fusion module is designed to fuse the features extracted from the two branches to reduce semantic discrepancies. A cascaded upsampling module using SFA local progressive upsampling is then used to reduce feature loss during the upsampling process. Simultaneously, a deep supervision strategy is used to verify the contribution of each module to the final segmentation result, and a dual-weighted feature loss function is designed to accelerate convergence, ultimately yielding a refined semantic segmentation result.
[0006] To achieve the above objectives, this invention provides a polyp segmentation method based on a dual-branch feature fusion network, comprising:
[0007] Obtain the colonoscopy polyp segmentation dataset and divide it into training, validation, and test sets according to a preset ratio;
[0008] The colonoscopy polyp segmentation dataset was preprocessed using a space-exclusive pasting technique for data augmentation.
[0009] The preprocessed training set, validation set, and test set are input into the TCBFormer segmentation network model. A deep supervision strategy is employed, and the TCBFormer segmentation network model is trained for semantic segmentation using a weighted IoU loss function and a binary cross-entropy loss function. The TCBFormer segmentation network model includes a Transformer branch, a CNN branch, a BiFusion feature fusion module, and an SFA local progressive upsampling module. The Transformer branch extracts global semantic information from the image data, the CNN branch extracts local fine features from the image data, the BiFusion feature fusion module effectively combines the global semantic information and local fine features of the image data to obtain fused features, and the SFA local progressive upsampling module performs cascaded upsampling on the fused features.
[0010] The colonoscopy image to be segmented is input into the trained TCBFormer segmentation network model to obtain the polyp segmentation result.
[0011] In the above technical solution, preferably, the specific process of performing data augmentation preprocessing on the colonoscopy polyp segmentation dataset using the space-exclusive pasting technique includes:
[0012] Each coordinate is initialized using latent values through the latent map generation module;
[0013] The paste module allows you to paste an image onto the same image background in a space-exclusive manner;
[0014] The latent map update module updates the latent values of each coordinate in the image background and uses convolution operations to expand the latent values of each coordinate to the surrounding area.
[0015] The pasting module and the potential map update module are iterated a preset number of times to obtain a new number of combined images, which are then used to augment the original images.
[0016] In the above technical solution, preferably, in the TCBFormer segmentation network, the Transformer branch uses stacked Swin-V2 Blocks to encode the input image in order to extract the global semantic information of the image;
[0017] The CNN branch is based on the ResNet-101 network and uses 4 Conv blocks to downsample the original image of size H×W to size H / 32×W / 32.
[0018] In the above technical solution, preferably, in the TCBFormer segmentation network, the BiFusion feature fusion module combines self-attention and multimodal fusion mechanisms, uses SE-Block to implement channel attention on the features extracted by the Transformer branch, uses CBAM-Block to implement spatial attention on the features extracted by the CNN branch, performs a Haldman product operation on the features of the Transformer branch and the features of the CNN branch, and generates the fused features through residual connections between the SE-Block, the CBAM-Block and the Haldman product operation results;
[0019] The SFA local progressive upsampling module is a cascaded upsampling structure based on attention gates.
[0020] In the above technical solution, preferably, the specific method for training the TCBFormer segmentation network model for semantic segmentation using a deep supervision strategy, based on a weighted IoU loss function and a binary cross-entropy loss function, is as follows:
[0021] The dual-weighted feature loss function, which combines the weighted IoU loss function and the binary cross-entropy loss function, is: in, Indicates the weighted Iou loss. This represents the weighted binary cross-entropy loss;
[0022] Under the deep supervision strategy, the final loss function L is obtained by weighted summation of the loss functions of each deep supervision branch. total =αL T +βL F +γL f ;
[0023] Among them, L total L represents the final loss. T L F L f α, β, and γ represent the deep supervision loss of the Tranformer branch, the feature fusion module, and the final output feature map, respectively. α, β, and γ are weighting coefficients and satisfy α+β+γ=1.
[0024] The TCBFormer segmentation network model is trained end-to-end using the final loss function.
[0025] This invention also proposes a polyp segmentation system based on a dual-branch feature fusion network, applying the polyp segmentation method based on a dual-branch feature fusion network disclosed in any of the above technical solutions, including:
[0026] The data collection module is used to acquire colonoscopy polyp segmentation datasets and divide them into training, validation and test sets according to a preset ratio;
[0027] The data augmentation module is used to perform data augmentation preprocessing on the colonoscopy polyp segmentation dataset using space-exclusive pasting technology;
[0028] The model training module is used to input the preprocessed training set, validation set, and test set into the TCBFormer segmentation network model. Employing a deep supervision strategy, the TCBFormer segmentation network model is trained for semantic segmentation using a weighted IoU loss function and a binary cross-entropy loss function. The TCBFormer segmentation network model includes a Transformer branch, a CNN branch, a BiFusion feature fusion module, and an SFA local progressive upsampling module. The Transformer branch extracts global semantic information from the image data, the CNN branch extracts local fine features from the image data, the BiFusion feature fusion module effectively combines the global semantic information and local fine features of the image data to obtain fused features, and the SFA local progressive upsampling module performs cascaded upsampling on the fused features.
[0029] The image segmentation module is used to input the colonoscopy image to be segmented into the trained TCBFormer segmentation network model to obtain the polyp segmentation result.
[0030] In the above technical solution, preferably, the data enhancement module is specifically used for:
[0031] Each coordinate is initialized using latent values through the latent map generation module;
[0032] The paste module allows you to paste an image onto the same image background in a space-exclusive manner;
[0033] The latent map update module updates the latent values of each coordinate in the image background and uses convolution operations to expand the latent values of each coordinate to the surrounding area.
[0034] The pasting module and the potential map update module are iterated a preset number of times to obtain a new number of combined images, which are then used to augment the original images.
[0035] In the above technical solution, preferably, in the TCBFormer segmentation network, the Transformer branch uses stacked Swin-V2 Blocks to encode the input image in order to extract the global semantic information of the image;
[0036] The CNN branch is based on the ResNet-101 network and uses 4 Conv blocks to downsample the original image of size H×W to size H / 32×W / 32.
[0037] In the above technical solution, preferably, in the TCBFormer segmentation network, the BiFusion feature fusion module combines self-attention and multimodal fusion mechanisms, uses SE-Block to implement channel attention on the features extracted by the Transformer branch, uses CBAM-Block to implement spatial attention on the features extracted by the CNN branch, performs a Haldman product operation on the features of the Transformer branch and the features of the CNN branch, and generates the fused features through residual connections between the SE-Block, the CBAM-Block and the Haldman product operation results;
[0038] The SFA local progressive upsampling module is a cascaded upsampling structure based on attention gates.
[0039] In the above technical solution, preferably, the specific method by which the model training module performs semantic segmentation training on the TCBFormer segmentation network model includes:
[0040] The dual-weighted feature loss function, which combines the weighted IoU loss function and the binary cross-entropy loss function, is: in, Indicates the weighted Iou loss. This represents the weighted binary cross-entropy loss;
[0041] Under the deep supervision strategy, the final loss function L is obtained by weighted summation of the loss functions of each deep supervision branch. total =αL T +βL F +γL f ;
[0042] Among them, L total L represents the final loss. T L F L f α, β, and γ represent the deep supervision loss of the Tranformer branch, the feature fusion module, and the final output feature map, respectively. α, β, and γ are weighting coefficients and satisfy α+β+γ=1.
[0043] The TCBFormer segmentation network model is trained end-to-end using the final loss function.
[0044] Compared with existing technologies, the beneficial effects of this invention are as follows: Addressing the characteristics of the Kvasir-SEG dataset—small sample size and difficulty in distinguishing foreground and background—this invention employs a spatially exclusive pasting data preprocessing method for data augmentation. It utilizes a dual-branch parallel network architecture, extracting local fine features from the image through a CNN branch and global features through a Transformer branch. A BiFusion feature fusion module is designed to fuse the features extracted by the two branches to reduce semantic gaps. A cascaded upsampling module using SFA local progressive upsampling further reduces feature loss during the upsampling process. Simultaneously, a deep supervision strategy is used to verify the contribution of each module to the final segmentation result. A dual-weighted feature loss function is designed to accelerate convergence, ultimately yielding a refined semantic segmentation result. This eliminates semantic gaps, reduces feature loss, exhibits strong generalization ability, and significantly improves the segmentation accuracy of polyps in images. Attached Figure Description
[0045] Figure 1 This is a flowchart illustrating a polyp segmentation method based on a dual-branch feature fusion network, as disclosed in an embodiment of the present invention.
[0046] Figure 2 This is a schematic diagram illustrating the implementation process of space-exclusive pasting in one embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram of the overall architecture of the TCBFormer segmentation network model disclosed in one embodiment of the present invention.
[0048] Figure 4 This is a schematic diagram of the architecture of the Swin-V2 Block disclosed in one embodiment of the present invention;
[0049] Figure 5 This is a schematic diagram of the architecture of the BiFusion feature fusion module disclosed in one embodiment of the present invention;
[0050] Figure 6 This is a schematic diagram of a polyp segmentation system based on a dual-branch feature fusion network, as disclosed in an embodiment of the present invention.
[0051] In the diagram, the correspondence between the components and the reference numerals is as follows:
[0052] 1. Data collection module, 2. Data augmentation module, 3. Model training module, 4. Image segmentation module. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0054] The present invention will now be described in further detail with reference to the accompanying drawings:
[0055] like Figure 1 As shown, a polyp segmentation method based on a dual-branch feature fusion network provided by the present invention includes:
[0056] Obtain the colonoscopy polyp segmentation dataset and divide it into training, validation, and test sets according to a preset ratio;
[0057] Spatial exclusive pasting technique was used to perform data augmentation preprocessing on the colonoscopy polyp segmentation dataset;
[0058] The preprocessed training, validation, and test sets are input into the TCBFormer segmentation network model. A deep supervision strategy is adopted, and the TCBFormer segmentation network model is trained for semantic segmentation under a weighted IoU loss function and a binary cross-entropy loss function. The TCBFormer segmentation network model includes a Transformer branch, a CNN branch, a BiFusion feature fusion module, and an SFA local progressive upsampling module. The Transformer branch extracts global semantic information from the image data, the CNN branch extracts local fine features from the image data, the BiFusion feature fusion module effectively combines the global semantic information and local fine features of the image data to obtain fused features, and the SFA local progressive upsampling module performs cascaded upsampling on the fused features.
[0059] The colonoscopy image to be segmented is input into the trained TCBFormer segmentation network model to obtain the polyp segmentation result.
[0060] In this implementation, addressing the challenges of the Kvasir-SEG dataset's limited sample size and difficulty in distinguishing between foreground and background, a spatially exclusive pasting data preprocessing method is used for data augmentation. A dual-branch parallel network architecture is employed, extracting local fine features from the image through a CNN branch and global features through a Transformer branch. A BiFusion feature fusion module is designed to fuse the features extracted from the two branches to reduce semantic gaps. A cascaded upsampling module using SFA local progressive upsampling further reduces feature loss during the upsampling process. Simultaneously, a deep supervision strategy is used to verify the contribution of each module to the final segmentation result. A dual-weighted feature loss function is designed to accelerate convergence, ultimately yielding a refined semantic segmentation result that eliminates semantic gaps, reduces feature loss, exhibits strong generalization ability, and significantly improves the segmentation accuracy of polyps in images.
[0061] Specifically, during the implementation process, the open-source colonoscopy polyp segmentation dataset Kvasir-SEG was used as the experimental dataset, which contains 880 colonoscopy polyp images and their corresponding ground truth labels, and the training set, validation set and test set were divided in an 8:1:1 ratio.
[0062] like Figure 2 As shown, in the above embodiment, preferably, the specific process of using space-exclusive pasting technology to perform data augmentation preprocessing on the colonoscopy polyp segmentation dataset includes:
[0063] Each coordinate is initialized using latent values through the latent map generation module;
[0064] The paste module allows you to paste an image onto the same image background in a space-exclusive manner;
[0065] The latent map update module updates the latent values of each coordinate in the image background and uses convolution operations to expand the latent values of each coordinate to the surrounding area.
[0066] The paste module and the potential map update module are iterated a preset number of times to obtain a new image with a combined number of images, which are then used to augment the original image.
[0067] During implementation, the space-exclusive pasting technology includes the following steps:
[0068] Step 1: The generation module randomly initializes an image M of the same size as the input image. The coordinates with the lowest potential value are raised to the reference position for pasting. The position r of the responding foreground is manually set to a large value T so that the potential value of the foreground region does not rise.
[0069] Step 2: Paste the cropped foreground into the canvas area. If there is overlap between the foreground to be pasted and the pasting location, cancel the pasting and increase the potential value of the pasting location to the maximum value T. If there is no overlap, replace the candidate area with the following convex combination:
[0070] I[c]=α×I[c]+(1-α)×I f
[0071] L[c]=α×L[c]+(1-α)×L f
[0072] Step 3: After attempting to paste, some potential values will be changed to large values T, meaning they will never be selected as candidate values. Based on the reasonable assumption that nearby coordinates have similar probabilities of avoiding repeated pasting, we extend the potential value of each coordinate to its neighbors using the following convolution operation: M = Conv(M, w)
[0073] Here, w is a mean filter.
[0074] Step 4: Repeat the paste module and potential mapping update module 10 times. Foreground regions may appear in multiple locations; use the obtained combined foreground images to perform data augmentation on the original image.
[0075] like Figure 3 As shown, in the above embodiment, preferably, in the TCBFormer segmentation network, the Transformer branch uses stacked Swin-V2 Blocks to encode the input image in order to extract the global semantic information of the image;
[0076] The CNN branch is based on the ResNet-101 network and uses 4 Conv blocks to downsample the original H×W image to H / 32×W / 32.
[0077] like Figure 4 As shown, the Swin-V2 Block has made several adjustments to the original Swin-Transformer Block:
[0078] 1) Replace the previous pre-norm configuration with res-post-norm;
[0079] 2) Replace the original dot product attention with scaled cosine attention;
[0080] 3) The previous parameterization method was replaced with the relative positional deviation of the logarithmic interval.
[0081] Among them, 1) and 2) make the model easier to extend, and 3) makes the model have a larger transfer capacity.
[0082] The CNN branch uses ResNet-101. Considering that using 5 Conv blocks to map the original H×W image to H / 64×W / 64 would increase the depth of the Transformer branch, resulting in too many model parameters and excessive computational resources, the excellent global feature extraction capabilities of the Transformer branch mean that the CNN branch does not require excessive network depth to obtain a larger receptive field. Therefore, only 4 Conv blocks are used to downsample the image to H / 32×W / 32.
[0083] like Figure 5 As shown, in the above implementation, preferably, in the TCBFormer segmentation network, the BiFusion feature fusion module combines self-attention and multimodal fusion mechanisms. It employs SE-Block to achieve channel attention on features extracted from the Transformer branch, thereby promoting global information from the Transformer branch. It also employs CBAM-Block to achieve spatial attention on features extracted from the CNN branch, thereby enhancing local details and suppressing irrelevant regions. Furthermore, it performs a Haldman product operation on the features of the Transformer branch and the CNN branch, modeling the fine-grained interactions between the features of the two branches to obtain interactive features. b ^ Participation features obtained from i, SE-Block and CBAM-Block ^ t i and g ^ i and interaction features b ^ The i-th elements are connected via residual connections to generate a fused feature f. i .
[0084] In this embodiment, feature fusion is represented as follows:
[0085]
[0086]
[0087] in, This represents the feature map from the Transformer branch. This represents the feature map from the CNN branch. ⊙ is the Hadamard product, and Conv is a 3×3 convolutional layer.
[0088] To reduce feature loss during upsampling and shorten the sentence gap between feature maps from different layers, the SFA local progressive upsampling module is a cascaded upsampling structure based on attention gates. The specific implementation is as follows:
[0089]
[0090] in, This represents the feature map from the i-th layer. This represents the feature map from the (i+1)th layer. The AG stands for Attention Gate, representing the fused feature map, and is implemented using a regular convolution operation.
[0091] In the above implementation, preferably, a deep supervision strategy is adopted. The specific method for training the TCBFormer segmentation network model for semantic segmentation under the weighted IoU loss function and binary cross-entropy loss function is as follows:
[0092] The dual-weighted feature loss function, which combines the weighted IoU loss function and the binary cross-entropy loss function, is:
[0093] To monitor the contribution of different modules to the final segmentation result, a deep supervision strategy is adopted. The loss functions of each deep supervision branch are weighted and summed to obtain the final loss function L. total =αL T +βL F +γL f ;
[0094] The TCBFormer segmentation network model is trained end-to-end using the final loss function. The segmentation result is generated by a simple segmentation head that directly adjusts the input feature map to the original resolution.
[0095] According to the polyp segmentation method based on dual-branch feature fusion network disclosed in the above implementation method, in order to verify the model performance, the performance of TCBFormer in polyp segmentation is first evaluated relative to various state-of-the-art methods, including mDice and mIoU.
[0096] Table 1 Performance comparison of TCBFormer compared to various state-of-the-art methods
[0097]
[0098]
[0099] As shown in Table 1, under the same experimental conditions, TCBFormer achieved the best performance in both mDice and mIoU metrics. On the ISIC-2018 dataset, TCBFormer also achieved the highest mDice, while its mIoU was second only to MSRF-Net.
[0100] like Figure 6 As shown, this invention also proposes a polyp segmentation system based on a dual-branch feature fusion network, applying the polyp segmentation method based on a dual-branch feature fusion network disclosed in any of the above embodiments, including:
[0101] Data collection module 1 is used to acquire colonoscopy polyp segmentation dataset and divide it into training set, validation set and test set according to preset ratio;
[0102] Data augmentation module 2 is used to perform data augmentation preprocessing on the colonoscopy polyp segmentation dataset using space-exclusive pasting technology;
[0103] Model training module 3 is used to input the preprocessed training set, validation set, and test set into the TCBFormer segmentation network model. A deep supervision strategy is employed, using a weighted IoU loss function and a binary cross-entropy loss function to train the TCBFormer segmentation network model for semantic segmentation. The TCBFormer segmentation network model includes a Transformer branch, a CNN branch, a BiFusion feature fusion module, and an SFA local progressive upsampling module. The Transformer branch extracts global semantic information from the image data, the CNN branch extracts local fine features from the image data, the BiFusion feature fusion module effectively combines the global semantic information and local fine features of the image data to obtain fused features, and the SFA local progressive upsampling module performs cascaded upsampling on the fused features.
[0104] Image segmentation module 4 is used to input the colonoscopy image to be segmented into the trained TCBFormer segmentation network model to obtain the polyp segmentation result.
[0105] In the above implementation, considering the small number of samples and the difficulty in distinguishing between foreground and background in the Kvasir-SEG dataset, a spatial exclusive pasting data preprocessing method is used for data augmentation. A dual-branch parallel network architecture is used, extracting local fine features of the image through the CNN branch and global features through the Transformer branch. A BiFusion feature fusion module is designed to fuse the features extracted by the two branches to reduce semantic gap. Then, a cascaded upsampling module is used through SFA local progressive upsampling to reduce feature loss during the upsampling process. At the same time, a deep supervision strategy is used to verify the contribution of each module to the final segmentation result. A dual-weighted feature loss function is designed to accelerate the convergence speed. Finally, a fine semantic segmentation result is obtained, which eliminates semantic gap, reduces feature loss, has strong generalization ability, and greatly improves the segmentation accuracy of polyps in the image.
[0106] In the above embodiments, preferably, the data enhancement module 2 is specifically used for:
[0107] Each coordinate is initialized using latent values through the latent map generation module;
[0108] The paste module allows you to paste an image onto the same image background in a space-exclusive manner;
[0109] The latent map update module updates the latent values of each coordinate in the image background and uses convolution operations to expand the latent values of each coordinate to the surrounding area.
[0110] The paste module and the potential map update module are iterated a preset number of times to obtain a new image with a combined number of images, which are then used to augment the original image.
[0111] In the above embodiments, preferably, in the TCBFormer segmentation network, the Transformer branch uses stacked Swin-V2 Blocks to encode the input image in order to extract the global semantic information of the image;
[0112] The CNN branch is based on the ResNet-101 network and uses 4 Conv blocks to downsample the original H×W image to H / 32×W / 32.
[0113] In the above implementation, preferably, in the TCBFormer segmentation network, the BiFusion feature fusion module combines self-attention and multimodal fusion mechanisms, uses SE-Block to achieve channel attention on the features extracted from the Transformer branch, uses CBAM-Block to achieve spatial attention on the features extracted from the CNN branch, performs a Haldman product operation on the features of the Transformer branch and the features of the CNN branch, and generates fused features through residual connections between the SE-Block, CBAM-Block and Haldman product operation results;
[0114] The SFA local progressive upsampling module is a cascaded upsampling structure based on attention gates.
[0115] In the above embodiments, preferably, the specific method by which the model training module 3 performs semantic segmentation training on the TCBFormer segmentation network model includes:
[0116] The dual-weighted feature loss function, which combines the weighted IoU loss function and the binary cross-entropy loss function, is: in, Indicates the weighted Iou loss. This represents the weighted binary cross-entropy loss;
[0117] Under the deep supervision strategy, the final loss function L is obtained by weighted summation of the loss functions of each deep supervision branch. total =αL T +βL F +γL f ;
[0118] Among them, L total L represents the final loss. T L F L f α, β, and γ represent the deep supervision loss of the Tranformer branch, the feature fusion module, and the final output feature map, respectively. α, β, and γ are weighting coefficients and satisfy α+β+γ=1.
[0119] The TCBFormer segmentation network model is trained end-to-end using the final loss function.
[0120] The polyp segmentation system based on a dual-branch feature fusion network disclosed in the above embodiments has modules whose functions correspond to the steps in the polyp segmentation method based on a dual-branch feature fusion network disclosed in the above embodiments. In the implementation process, the specific methods in the above embodiments are referred to, and will not be repeated here.
[0121] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A polyp segmentation method based on a double-branch feature fusion network, characterized in that, The application relates to a colonoscopy polyp segmentation method based on a TCBFormer segmentation network model. A colonoscopy polyp segmentation dataset is acquired and divided into a training set, a verification set and a test set according to a preset proportion; The colonoscopy polyp segmentation dataset is subjected to data enhancement preprocessing by adopting a spatial exclusive pasting technology; The preprocessed training set, verification set and test set are input into a TCBFormer segmentation network model, deep supervision strategy is adopted, and the TCBFormer segmentation network model is subjected to semantic segmentation training under a weighted IoU loss function and a binary cross-entropy loss function; wherein the TCBFormer segmentation network model comprises a Transformer branch, a CNN branch, a BiFusion feature fusion module and an SFA local progressive up-sampling module, the Transformer branch extracts global semantic information of image data, the CNN branch extracts local fine features of the image data, the BiFusion feature fusion module effectively combines the global semantic information and the local fine features of the image data to obtain fusion features, and the SFA local progressive up-sampling module performs cascaded up-sampling on the fusion features; A colonoscopy image to be segmented is input into the trained TCBFormer segmentation network model to obtain a polyp segmentation result; In the TCBFormer segmentation network, the BiFusion feature fusion module combines self-attention and multi-modal fusion mechanisms, adopts an SE-Block to realize channel attention on features extracted by the Transformer branch, adopts a CBAM-Block to realize spatial attention on features extracted by the CNN branch, performs Hadamard product operation on the features of the Transformer branch and the features of the CNN branch, and generates the fusion features through residual connection between the SE-Block, the CBAM-Block and the Hadamard product operation result; The SFA local progressive up-sampling module is an attention gate-based cascaded up-sampling structure; The specific mode of adopting deep supervision strategy and performing semantic segmentation training on the TCBFormer segmentation network model under the weighted IoU loss function and the binary cross-entropy loss function is that: The double-weighted feature loss function based on the combination of the weighted IoU loss function and the binary cross-entropy loss function is wherein, denotes the weighted IoU loss, denotes the weighted binary cross-entropy loss; Under the deep supervision strategy, the final loss function L is obtained by weighted summation of the loss functions of each deep supervision branch total = aL T + bL F + gL f ; wherein L total represents the final loss, L T , L F , L f respectively represent the deep supervision loss of the Tranformer branch, the feature fusion module and the final output feature map, and α, β, γ are weighting coefficients, and satisfy α+β+γ=1. The TCBFormer segmentation network model is trained end to end by using a final loss function.
2. The polyp segmentation method based on the dual-branch feature fusion network according to claim 1, characterized in that, The specific process of adopting the spatial exclusive pasting technology to perform data enhancement preprocessing on the colonoscopy polyp segmentation dataset comprises the following steps: Each coordinate is initialized by a latent map generation module by using a latent value; An image is pasted into the same image background in a spatial exclusive manner by a pasting module; The latent value of each coordinate of the image background is updated by a latent map updating module, and the latent value of each coordinate is expanded to the surrounding area by using convolution operation; The pasting module and the latent map updating module are iterated for a preset number of times, new images with a combined number are obtained, and the original image is subjected to data enhancement.
3. The polyp segmentation method based on the dual-branch feature fusion network according to claim 1, characterized in that, In the TCBFormer segmentation network, the Transformer branch encodes the input image by using stacked Swin-V2Block to extract global semantic information of the image. The CNN branch is based on a ResNet-101 network and uses four Conv blocks to downsample an image with an original size of HxW to a size of H / 32xW / 32.
4. A polyp segmentation system based on a double-branch feature fusion network, characterized in that, The application discloses a polyp segmentation method based on a double-branch feature fusion network. A data collection module is configured to obtain a colonoscopy polyp segmentation dataset and divide the dataset into a training set, a validation set and a test set according to a preset ratio. A data enhancement module is configured to perform data enhancement preprocessing on the colonoscopy polyp segmentation dataset by using a spatially exclusive pasting technique. A model training module is configured to input the preprocessed training set, validation set and test set into a TCBFormer segmentation network model, perform semantic segmentation training on the TCBFormer segmentation network model under a deep supervision strategy based on a weighted IoU loss function and a binary cross-entropy loss function, wherein the TCBFormer segmentation network model comprises a Transformer branch, a CNN branch, a BiFusion feature fusion module and an SFA local progressive upsampling module, the Transformer branch extracts global semantic information of the image data, the CNN branch extracts local fine features of the image data, the BiFusion feature fusion module effectively combines the global semantic information and the local fine features of the image data to obtain fused features, and the SFA local progressive upsampling module performs cascaded upsampling on the fused features. An image segmentation module is configured to input a colonoscopy image to be segmented into the trained TCBFormer segmentation network model to obtain a polyp segmentation result.
5. The polyp segmentation system based on the dual-branch feature fusion network according to claim 4, characterized in that, The data enhancement module is specifically configured to: Initialize each coordinate by using a latent value by a latent map generation module; Paste the image into the same image background in a spatially exclusive manner by a pasting module; Update the latent value of each coordinate of the image background by a latent map updating module, and expand the latent value of each coordinate to the surrounding area by using convolution operation; Iterate the pasting module and the latent map updating module for a preset number of times to obtain a plurality of new images to perform data enhancement on the original image.
6. The polyp segmentation system based on the dual-branch feature fusion network according to claim 5, characterized in that, In the TCBFormer segmentation network, the Transformer branch encodes the input image by using stacked Swin-V2Block to extract global semantic information of the image. The CNN branch is based on a ResNet-101 network and uses four Conv blocks to downsample an image with an original size of HxW to a size of H / 32xW / 32.
7. The polyp segmentation system based on the dual-branch feature fusion network according to claim 6, characterized in that, In the TCBFormer segmentation network, the BiFusion feature fusion module combines self-attention and multi-modal fusion mechanism, adopts SE-Block to realize channel attention on the features extracted by the Transformer branch, adopts CBAM-Block to realize spatial attention on the features extracted by the CNN branch, performs Hadamard product operation on the features of the Transformer branch and the features of the CNN branch, and generates the fusion features through residual connection between the SE-Block, the CBAM-Block and the Hadamard product operation result. The SFA local progressive up-sampling module is an attention gate-based cascaded up-sampling structure.
8. The polyp segmentation system based on the dual-branch feature fusion network according to claim 7, characterized in that, The specific manner in which the model training module performs semantic segmentation training on the TCBFormer segmentation network model includes: The double-weighted feature loss function based on the combination of the weighted IoU loss function and the binary cross-entropy loss function is wherein, denotes the weighted IoU loss, denotes the weighted binary cross-entropy loss; Under the deep supervision strategy, the final loss function L is obtained by weighted summation of the loss functions of each deep supervision branch total = aL T + bL F + gL f ; wherein L total represents the final loss, L T , L F , L f respectively represent the deep supervision loss of the Tranformer branch, the feature fusion module and the final output feature map, and α, β, γ are weighting coefficients and satisfy α+β+γ=1. The TCBFormer segmentation network model is trained end-to-end by using the final loss function.