CNN and PVT fusion-based colonoscope polyp image segmentation method
By adopting a dual-branch architecture with CNN and PVT fusion in colonoscopic polyp image segmentation, combining a multi-scale boundary enhancement module and a two-stage feature fusion module, the problem of global and local feature balance in the existing technology is solved, and higher segmentation accuracy and robustness are achieved.
Patent Information
- Application Number
- CN202411643752.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art is difficult to balance global semantic features with local detail features in colonoscopic polyp image segmentation, resulting in insufficient segmentation accuracy and robustness, especially when facing polyps and complex backgrounds with large scale changes.
Using a dual-branch architecture based on CNN and PVT fusion, PVT replaces Transformer, its multi-scale feature extraction capability is utilized, and the multi-scale boundary enhancement module and two-stage feature fusion module are enhanced to enhance local feature extraction and feature fusion capabilities to ensure the interaction and balance between global and local features.
It improves the accuracy and robustness of colonoscopic polyp image segmentation, can more accurately capture the boundary details and overall shape of polyp, adapt to different morphology and complex backgrounds, significantly reduces the situation of missegment and missegment, and improves the generalization performance of the model.
Smart Images
Figure CN119600049A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of colonoscopic polyp image segmentation, in particular to a colonoscopic polyp image segmentation method based on CNN and PVT fusion. Background Art
[0002] Existing polyp segmentation methods are divided into two categories: traditional methods and deep learning methods. Traditional polyp segmentation mostly relies on low-level features of polyp graphics, distinguishing lesions from backgrounds based on attributes such as color, texture, and geometric shape. However, due to the high similarity between polyps and surrounding tissues, these methods have a high risk of missed detection or false detection, which limits their accuracy and reliability in actual clinical applications. Deep learning-based methods are mainly divided into two categories: CNN and Transformer. CNN-based methods use convolution operations to extract features from adjacent pixels. This operation method only achieves short-range correlations and ignores long-range dependencies. This makes it excellent in capturing local feature information, but it is difficult to capture the overall shape of the polyp object, affecting the performance and generalization ability of polyp segmentation. Unlike CNN methods, Transformer-based methods use multi-head attention mechanisms and multi-layer perceptrons to simulate the global contextual information of the input patch. Therefore, Transformer shows excellent ability in capturing long-range dependencies, but it is insufficient in learning local features. This makes Transformer lose key local feature information when processing polyp images, and it is difficult to accurately capture the details of the polyp boundary.
[0003] Based on the above situation, although there have been studies combining CNN and Transformer, the traditional Transformer lacks multi-scale feature extraction capabilities and cannot cope with objects with large scale changes such as polyps; moreover, the current single-branch hybrid structure is prone to damage the features extracted in the previous step; the current dual-branch structure does not realize the interaction between global and local features between the two branches, and it is difficult to achieve a balance between global semantic features and local detail features, causing local features to be overwhelmed by the dominant global features; at the same time, continuous upsampling operations cause the loss of some feature detail information, and it is impossible to fully utilize high-level and low-level features of different scales. Therefore, the present application provides a colonoscopic polyp image segmentation method based on CNN and PVT fusion. Summary of the invention
[0004] In order to address the deficiencies in the above-mentioned prior art, the purpose of the present invention is to provide a colonoscopic polyp image segmentation method based on CNN and PVT fusion. This method replaces the Transformer with PVT, and uses its multi-scale feature extraction capability to cope with challenges such as the different shapes and sizes of polyps and the difficulty in distinguishing the differences between lesion tissue and background, thereby enhancing the robustness of the model. On this basis, a dual-branch architecture is used to extract features in parallel to improve segmentation accuracy, and a multi-scale boundary enhancement module is used to strengthen local features to improve edge segmentation accuracy, thereby adapting to the segmentation of polyp images in different morphologies and complex backgrounds.
[0005] The technical solution adopted by the present invention to solve the technical problem is:
[0006] A colonoscopy polyp image segmentation method based on CNN and PVT fusion is provided. Firstly, PVT is used instead of Transformer, and its multi-scale feature extraction capability makes the model more robust when facing challenges such as polyps of different shapes and sizes, and the difficulty in distinguishing the differences between lesion tissue and background. At the same time, the dual-branch architecture is used to extract global and local features of different scales in parallel, which not only improves the segmentation accuracy and robustness, but also effectively reduces the risk of information loss and ensures the comprehensive and accurate use of feature information. Secondly, a multi-scale boundary enhancement module is used to strengthen local features to prevent local features from being overshadowed by dominant global features. Flooding, thereby enhancing the model's ability to capture edge details and improving the edge segmentation accuracy; finally, a two-stage feature fusion module is proposed; this module not only realizes the interaction and balance between the global and local features of the two branches, so that the model can take into account the overall contour structure and edge details of the polyp; it also makes full use of high-level and low-level features of different scales, avoids feature information redundancy caused by direct splicing through innovative strategies, and reduces feature loss that may be caused by continuous upsampling operations; thereby enhancing the feature representation at different scales, enabling the model to accurately capture the complex structure of polyps, and can better adapt to the segmentation of polyp images in different morphologies and complex backgrounds.
[0007] During the above operation, the segmentation accuracy of this technical solution on five challenging datasets Kvasir-SEG, CVC-ClinicD (ClinicDB), CVC-ColonDB (ColonDB), ETIS and EndoScene is better than the current advanced methods. Its advantages are: facing mucus interference, uneven brightness distribution, polyp texture and color similar to surrounding tissues, small polyp size and other problems, our method has shown the best segmentation effect, not only can it accurately locate the position of the polyp, but also can accurately segment the overall contour structure of the polyp; at the same time, the boundary segmented by our proposed method is more precise; this enables our method to help doctors make diagnoses more quickly and accurately, and reduce the risk of misdiagnosis and missed diagnosis; in addition, another advantage of this method is that it has better generalization performance than other methods, and can show accurate polyp segmentation capabilities even on datasets that have never been seen; this provides strong support for medical intelligence and automation.
[0008] Compared with the prior art, the present invention has the following beneficial effects:
[0009] 1. The colonoscopic polyp image segmentation method based on CNN and PVT fusion in the example of the present invention uses PVT to replace Transformer. Its multi-scale feature extraction capability enables the model to show stronger robustness when facing challenges such as polyps of different shapes and sizes, and the difficulty in distinguishing the differences between lesion tissue and background. Secondly, the dual-branch architecture is used to effectively extract global and local features of different scales. On the one hand, this design avoids the lack of segmentation accuracy due to the extraction of a single feature. On the other hand, the parallel dual-branch architecture is used to extract global and local features respectively, reducing the risk of information loss and ensuring the comprehensiveness and accuracy of feature information.
[0010] 2. The colonoscopic polyp image segmentation method based on CNN and PVT fusion in the example of the present invention strengthens the local features extracted by the CNN encoder to obtain more boundary feature detail information, prevents the local features from being overwhelmed by the dominant global features in the feature fusion stage, improves the model's ability to segment boundaries, and reduces mis-segmentation and missed segmentation.
[0011] 3. The colonoscopic polyp image segmentation method based on CNN and PVT fusion in the example of the present invention fuses the global and local features of the same scale of different branches in the first stage to achieve interaction between the two and balance the global and local features, so that the model can take into account the overall contour and boundary details of the polyp, further improving the segmentation accuracy of the model. In the second stage, the mixed features of different scales are fused again to prevent the direct splicing of high-level features and low-level features from causing information redundancy and feature damage. At the same time, continuous upsampling is avoided to damage the feature information, and the continuity of the feature information is maintained to ensure that sufficient detail information is retained while restoring the image resolution. Not only does it enhance the robustness of the model, but it also ensures the segmentation accuracy of the model when facing complex situations. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0013] Figure 1 This is a flow chart of the dual-branch polyp segmentation based on enhanced CNN and PVT in parallel in the present invention;
[0014] Figure 2 It is an overall framework diagram of the dual-branch polyp segmentation model based on CNN and PVT in parallel in the present invention;
[0015] Figure 3 It is a framework diagram of the CNN encoder of the present invention;
[0016] Figure 4 It is a framework diagram of the PVT encoder of the present invention;
[0017] Figure 5 It is a framework diagram of the multi-scale boundary enhancement module of the present invention;
[0018] Figure 6 It is a framework diagram of the cross-branch feature fusion module of the present invention;
[0019] Figure 7 It is a framework diagram of the cross-level feature fusion module of the present invention. DETAILED DESCRIPTION
[0020] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.
[0021] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.
[0022] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.
[0023] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0024] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings.
[0026] In the existing technology, polyp segmentation requires the model to be able to capture both global and local features at the same time, but the existing deep learning models have certain limitations and it is often difficult to strike a balance between the two; CNN performs well in local feature extraction, but it is difficult to capture global shape information; and although Transformer is good at capturing global contextual information, it is insufficient in local feature learning; when combining CNN with Transformer, how to effectively integrate the advantages of the two while avoiding their respective defects is a technical challenge; on the other hand, the scale of polyps varies greatly, requiring the model to be able to process features of different scales; traditional Transformer lacks multi-scale feature extraction capabilities, and existing hybrid structures (such as single-branch and double-branch structures) have their own defects, such as feature damage and insufficient interaction between global and local features.
[0027] The training of deep learning models requires a large amount of data and computing resources; however, it is usually difficult to obtain and label medical image data, especially high-quality, accurately labeled polyp image data; insufficient data and inaccurate labeling will lead to insufficient model training, thereby affecting the accuracy of segmentation; and polyp segmentation datasets are usually small in scale and suffer from data imbalance and uneven labeling quality, which increases the difficulty of model training.
[0028] The proposal and implementation of this application improves the accuracy and reliability of polyp image segmentation; our method can overcome challenges such as mucus interference and uneven brightness, accurately segment polyps, thereby helping doctors make diagnoses more quickly and accurately, and reducing the risk of misdiagnosis and missed diagnosis; more importantly, the implementation of this solution can reduce patients' unnecessary pain and anxiety caused by misdiagnosis or missed diagnosis, improve the quality and efficiency of medical services, and provide patients with a better medical experience.
[0029] Secondly, before surgery, doctors need to know in detail the location, size, and shape of polyps. Our method can provide accurate polyp segmentation results, providing strong support for surgical planning. During surgery, doctors can also use these segmentation results to guide surgical operations and ensure the accuracy and safety of the surgery.
[0030] Finally, by improving the accuracy and efficiency of diagnosis, our method helps reduce unnecessary examinations and surgeries, thereby reducing medical costs; at the same time, accurate segmentation results can also help doctors use medical resources more effectively, such as reasonably arranging surgical time, optimizing surgical procedures, etc., thereby improving the utilization rate of medical resources.
[0031] The present invention proposes a double-branch polyp segmentation method based on enhanced CNN and PVT, and the specific steps are as follows:
[0032] Step 1: Divide the data set and perform preprocessing operations:
[0033] First, the acquired polyp dataset is divided into training set, validation set and test set in the ratio of 0.8:0.1:0.1. Then the images and segmentation maps of the training set and validation set are resized to 352×352, and the images are enhanced by random flipping, affine transformation, Gaussian blur and color jitter, while the segmentation maps are enhanced by random flipping and affine transformation. Finally, the images of the test set are resized to 352×352 without any post-processing optimization strategy.
[0034] Step 2: Use CNN and PVT in parallel to build the backbone network. The overall architecture of the model is shown in the figure below: Figure 2 As shown:
[0035] The CNN and Transformer methods alone cannot take into account both global and local features, so we choose to combine the two. In order to prevent the series connection from damaging the features extracted in the previous step, we choose to combine the two in parallel. At the same time, in order to extract global features of different scales, we choose to use the pyramid vision Transformer to replace the original Transformer.
[0036] Therefore, the present invention uses CNN and PVT encoder to ensure that rich local and global features can be obtained at different levels through layer-by-layer extraction. Specifically, given an image I∈R H×W×3 , use the PVT encoder to extract four pyramid features Where Ci∈{64,128,320,512}, i∈{1,2,3,4}. And use CNN encoder to extract 6 local features where j∈{1,2,3,4,5,6}.
[0037] CNN encoder such as Figure 3 As shown. It contains 6 convolution modules and 5 downsampling modules. Each convolution module is composed of two identical convolution sub-modules connected in series. Each sub-module contains three parts: 3×3 convolution, Batch Normalization and LeakyReLU activation function. The purpose of using two identical convolution sub-modules in succession is to enhance the expressiveness and depth of the model by stacking more nonlinear layers and feature extraction layers. The downsampling module contains a 3×3 convolution operation with a stride of 2 and a padding of 1, a Batch Normalization and a LeakyReLU activation function. This module is used to halve the H and W of the image. The features of convolution modules at different levels are used as features of different scales of the CNN encoder. Output is used for subsequent enhancement operations, where j∈{3,4,5,6}.
[0038] PVT encoder Figure 4As shown, the present invention uses the ImageNet pre-trained Pyramid Visual Transformer v2 (PVTv2) instead of Transformer. PvTv2 consists of 4 stages, each of which consists of an embedding layer and Li layer Transformer encoder. Specifically, in each stage, PVTv2 first decomposes the input feature map into multiple image blocks (patches) of the same size; then uses the embedding layer to linearly project the patch to obtain an embedded patch; then the embedded patch is added to the position embedding and passed through the Li layer Transformer encoder, and finally converted into a three-dimensional state and sent to the next stage. In order to better suit the polyp segmentation task, 4 features of different scales generated at 4 different levels are selected. As output, where Ci∈{64,128,320,512}, i∈{1,2,3,4}.
[0039] Step 3: Construct a multi-scale boundary enhancement module to enhance the CNN branch features;
[0040] In order to better identify the edges of polyps and prevent local features from being overwhelmed by dominant global features in the fusion stage, the present invention introduces a multi-scale boundary enhancement module to enhance the local features output by the CNN encoder, so as to strengthen the edge details of the polyp object.
[0041] The present invention designs a boundary enhancement module, such as Figure 5 The specific process is as follows: Input features First, through the two branches of 1×1 convolution and N×N convolution, the former retains edge information through a small convolution kernel, and the latter expands the receptive field through a large convolution kernel. Subsequently, the features obtained in the 1×1 convolution are subtracted from the features obtained in the N×N convolution to obtain the enhanced detail features. Then, the enhanced features are sent to another 1×1 convolution to perform a nonlinear transformation on the enhanced detail features and further refine these features. Next, the output is added to the output of the N×N convolution to obtain the boundary enhancement features for different convolution kernels. Finally, Batch Normalization and ReLU are used to obtain the final enhanced boundary feature Y j N , where j∈{3,4,5,6}, N∈{3,5,7,9,11}. At the same time, in order to prevent overfitting, the entire enhancement process is connected with residual links. The process can be summarized as follows:
[0042] Y j N =BR(C1(C1(Y j )C N (Y j ))+CN (Y j ))+CBR(Y j )
[0043] Among them, CN represents the convolution operation with kernel size N, BR represents the combined module of Batch Normalization and ReLU, CBR represents the combined module of 1×1 convolution, Batch Normalization and ReLU, j∈{3,4,5,6}, N∈{3,5,7,9,11}.
[0044] In order to solve the problem that polyps have different shapes and sizes, the present invention combines five boundary enhancement modules with different convolution kernel sizes into a multi-scale boundary enhancement module for enhancement. Figure 5 As shown in the lower half of the area. For the input feature Yj, it is first sent to five boundary enhancement modules to obtain five enhanced features Y j N , then Y j N Combine them by splicing, and finally, use 1×1 convolution to adjust the number of channels to obtain the final output features. Where j∈{3,4,5,6}, N∈{3,5,7,9,11}. The process can be summarized as follows:
[0045]
[0046] Among them, C1 represents the convolution with a convolution kernel size of 1, EP N Represents a boundary enhancement module with a large convolution kernel of N. represents the concatenation operation, j∈{3,4,5,6}, N∈{3,5,7,9,11}.
[0047] Step 4: Construct a two-stage feature fusion module to make full use of global and local features of different branches and scales;
[0048] Directly splicing features of the same scale or different scales of different branches is not a good fusion method. The former will cause local features to be submerged by the dominant global features, making the segmented polyp boundaries blurred, and the latter may cause the original feature information to be destroyed, thus affecting the accuracy of model segmentation and reducing the generalization ability of the model. In order to make full use of global and local features of different branches and scales, a two-stage feature fusion module is designed. It consists of four cross-branch feature fusion modules and three cross-level feature fusion modules. The cross-branch feature fusion module generates a feature map by fusing global and local features of the same scale under two different branches, preventing local features from being submerged by the dominant global features, and promoting the complementarity of features between different branches, thereby improving the model's recognition of the overall shape of polyps and the positioning of polyp boundaries. The cross-level feature fusion module fuses mixed features of different scales to avoid continuous upsampling that destroys the original feature information, and also avoids information conflicts caused by direct splicing of high-level features and low-level features, thereby enhancing feature representation at different scales.
[0049] Cross-branch feature fusion module such as Figure 6 As shown, the PVT characteristics of the input With CNN features (where Ci∈{64,128,320,512}, i∈{1,2,3,4}, j∈{3,4,5,6}). First, the features X from the two branches are i , Y j 'Concatenate and adjust the number of channels through 1×1 convolution to obtain two mixed features A with different numbers of channels i , B j Then, the feature X i Send it to the channel Attention module, through which X is kept i The precise location information of the feature is preserved while retaining its long-distance dependencies. j 'Sent to the Spatial Attention module to enhance its ability to represent boundary features and strengthen spatial location information, thereby suppressing the background information of colon polyps. Next, the features X obtained by the above two modules are i ', Y j "Apply the Sigmoid activation function to obtain the pixel-level probability map, and compare it with the mixed feature A of the corresponding number of channels i , B j Multiply them together to obtain A with global feature enhancement i 'With local feature enhancement B j '. Finally, feature A i 'After 1×1 convolution, adjust the number of channels and feature B j'same, and add the two together to obtain the final mixed feature that takes into account both local features and global features Where i∈{1,2,3,4}. The process can be summarized as follows:
[0050]
[0051] Among them, C1 represents the convolution with a convolution kernel size of 1, Sig represents the Sigmoid function, CA represents the channel attention operation, and SA represents the spatial attention operation. Represents a splicing operation, Represents a multiplication operation, i∈{1,2,3,4}, j∈{3,4,5,6}.
[0052] Cross-level feature fusion modules such as Figure 7 As shown, given the high-level feature T i With low-level features T i-1 First, the high-level features T i The size is adjusted by transposing convolution. Then, the significant pixel information is obtained by global pooling. Batch Normalization and Sigmoid activation function are performed on it to obtain the weight of the high-level feature. Next, the low-level feature LF is adjusted to have its number of channels through the convolution layer and multiplied with the weight of the high-level feature to obtain the enhanced low-level feature. Finally, the enhanced low-level feature is spliced with the initial high-level feature to obtain the final fusion feature T. i-1 , where i∈{2,3,4}. The process can be summarized as follows:
[0053]
[0054] Among them, TC1 represents deconvolution with a convolution kernel size of 1×1, C3 represents convolution with a convolution kernel size of 3×3, Avg represents average pooling, BN represents Batch Normalization, and Sig represents Sigmoid function. Represents a splicing operation, Represents the multiplication operation, i∈{2,3,4}.
[0055] Step 5: Train and test the constructed model to obtain the final segmentation result;
[0056] The preprocessed training set and validation set are input into the segmentation model for training and validation. The ReduceLROnPlateau scheduler is used to automatically adjust the learning rate. When the performance of the validation set does not improve within 10 epochs, the learning rate is reduced by 2 times, which can help the model get rid of the local optimal solution. This process will continue until the learning rate is reduced to the set minimum threshold to ensure that the learning rate will not decrease indefinitely, thereby avoiding gradient vanishing or learning stagnation. If the performance of the validation set improves, the model is saved after this epoch to obtain the highest performance model achieved during the training process. Finally, the test set is input into the trained segmentation model for testing to obtain the final segmentation result.
[0057] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.
[0058] Except for the technical features described in the specification, the remaining technical features are known technologies to those skilled in the art. In order to highlight the innovative features of the present invention, the remaining technical features will not be described here in detail.
Claims
1. A colonoscopic polyp image segmentation method based on CNN and PVT fusion, characterized in that: The following steps are involved: Step 1: Divide the data set and perform preprocessing operations; Step 2: Use CNN and PVT in parallel to build the backbone network; Step 3: Construct a multi-scale boundary enhancement module to enhance the CNN branch features; Step 4: Construct a two-stage feature fusion module to make full use of global and local features of different branches and scales; Step 5: Train and test the constructed model to obtain the final segmentation result.
2. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 1, characterized in that: In step 1, the specific steps of dividing the data set and performing preprocessing operations are as follows: Step 1: divide the obtained polyp dataset into training set, validation set and test set in the ratio of 0.8:0.1:0.1; Step 2: The images and segmentation maps of the training set and validation set are resized to 352×352, and the images are enhanced by random flipping, affine transformation, Gaussian blur and color jitter, while the segmentation maps are enhanced by random flipping and affine transformation. Step 3: resize the images of the test set to 352×352 without any post-processing optimization strategy.
3. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 1, characterized in that: In the step 2, CNN and PVT encoders are used to extract layer by layer to ensure that rich local and global features can be obtained at different levels; Given an image I∈R H×W×3 , use the PVT encoder to extract four pyramid features 4. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 3, characterized in that: The CNN encoder includes 6 convolution modules and 5 downsampling modules; Each convolution module is composed of two identical convolution sub-modules connected in series. Each sub-module contains three parts: 3×3 convolution, BatchNormalization and LeakyReLU activation function. The purpose of using two identical convolutional submodules in succession is to enhance the expressiveness and depth of the model by stacking more nonlinear layers and feature extraction layers; The downsampling module contains a 3×3 convolution operation with a stride of 2 and a padding of 1, a BatchNormalization and a LeakyReLU activation function; this module is used to halve the H and W of the image; the features of convolution modules at different levels are used as features of different scales of the CNN encoder Output for subsequent enhancement operations.
5. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 3, characterized in that: The PVT encoder uses the ImageNet pre-trained Pyramid Visual Transformer v2 (PVTv2), PvTv2 consists of 4 stages, each of which consists of an embedding layer and a Li layer Transformer encoder; Specifically, at each stage, PVTv2 first decomposes the input feature map into multiple image blocks (patches) of the same size; then uses the embedding layer to linearly project the patch to obtain the embedded patch; then the embedded patch is added to the position embedding and passed through the Li layer Transformer encoder, and finally converted into a three-dimensional state and sent to the next stage; in order to better suit the polyp segmentation task, four features of different scales generated at four different levels are selected as output.
6. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 1, characterized in that: In step 3, a multi-scale boundary enhancement module is introduced to enhance the local features output by the CNN encoder, so as to strengthen the edge details of the polyp object. The specific steps are as follows: Step 1: Input features Through the two branches of 1×1 convolution and N×N convolution, the former retains edge information through a small convolution kernel, and the latter expands the receptive field through a large convolution kernel; Step 2: Subtract the features obtained in the N×N convolution from the features obtained in the 1×1 convolution to obtain the enhanced detail features; Step 3: Send the enhanced features to another 1×1 convolution to perform nonlinear transformation on the enhanced detail features and further refine these features; Step 4: Add the output to the output of the N×N convolution to obtain the boundary enhancement features for different convolution kernels; Step 5: Use Batch Normalization and ReLU to obtain the final enhanced boundary feature Y j N .
7. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 1, characterized in that: In the step 4, the two-stage feature fusion module is composed of four cross-branch feature fusion modules and three cross-level feature fusion modules; The cross-branch feature fusion module generates a feature map by fusing global and local features of the same scale under two different branches, preventing local features from being overwhelmed by dominant global features, and promoting the complementarity of features between different branches, thereby improving the model's recognition of the overall shape of the polyp and the positioning of the polyp boundary; The cross-level feature fusion module fuses mixed features of different scales to avoid continuous upsampling from destroying the original feature information, and also avoids information conflict caused by direct concatenation of high-level features and low-level features, thereby enhancing feature representation at different scales.
8. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 7, characterized in that: The steps to use the cross-branch feature fusion module are as follows: Step 1: Input PVT features With CNN features Step 2: The features X from the two branches i , Y j 'Concatenate and adjust the number of channels through 1×1 convolution to obtain two mixed features A with different numbers of channels i , B j ; Step 3: transform feature X i Send it to the channel Attention module, through which X is kept i The precise location information of features while preserving their long-range dependencies; Step 4: Set feature Y j 'Send it to the SpatialAttention module to enhance its ability to represent boundary features and strengthen spatial location information, thereby suppressing the background information of colon polyps; Step 5: The features X obtained from the above two modules i ', Y j "Apply the Sigmoid activation function to obtain the pixel-level probability map, and compare it with the mixed feature A of the corresponding number of channels i , B j Multiply them together to obtain A with global feature enhancement i 'With local feature enhancement B j '; Step 6: Feature A i 'After 1×1 convolution, adjust the number of channels and feature B j 'same, and add the two together to obtain the final mixed feature that takes into account both local features and global features 9. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 7, characterized in that: The steps to use the cross-level feature fusion module are as follows: Step 1: Given high-level features T i With low-level features T i-1 ; Step 2: The high-level feature T i Resize it by transposing the convolution; Step 3: Obtain significant pixel information through global pooling; and perform Batch Normalization and Sigmoid activation functions on it to obtain the weights of high-level features; Step 4: The low-level feature LF is adjusted through the convolution layer to have its channel number, and multiplied by the weight of the high-level feature to obtain the enhanced low-level feature; Step 5: Concatenate the enhanced low-level features with the initial high-level features to obtain the final fusion feature T i-1 .
10. The colonoscopic polyp image segmentation method based on CNN and PVT fusion according to claim 1, characterized in that: In the step 5, the pre-processed training set and validation set are input into the segmentation model for training and validation; the ReduceLROnPlateau scheduler is used to automatically adjust the learning rate; When the performance of the validation set does not improve within 10 epochs, the learning rate is reduced by a factor of 2, which can help the model get rid of the local optimal solution; this process will continue until the learning rate is reduced to the set minimum threshold to ensure that the learning rate does not decrease indefinitely, thereby avoiding gradient vanishing or learning stagnation; If the performance of the validation set improves, the model is saved after this epoch to obtain the highest performance model achieved during the training process; finally, the test set is input into the trained segmentation model for testing to obtain the final segmentation result.