Polyp image segmentation method based on pyramid vision transformer

Abstract

The present application relates to a kind of polyp image segmentation method based on pyramid vision Transformer, belong to digital medical technology field.The method includes: by pyramid vision Transformer module capture global context information, obtain feature map of different resolution from multiple extraction stages;By cyclic cross attention module, extract detail information from shallow feature map of first stage, obtain shallow feature;By multi-scale feature fusion perception module, the feature map extracted in multiple stages is fused and processed, and deep feature is extracted, and shallow feature is fused after deep feature and is used for image segmentation;Design weighted binary cross-entropy loss and weighted IoU loss to train the foregoing process, after training, it is used for polyp image segmentation.The present application is through pyramid vision Transformer module, cyclic cross attention module and multi-scale feature fusion perception module, significantly improves global information learning ability, detail information capture ability and segmentation performance.

Description

Technical Field

[0001] The invention belongs to the field of digital medical technology and relates to a polyp image segmentation method based on pyramid vision Transformer. Background Art

[0002] Colorectal cancer (CRC) is the third most common cancer worldwide and has become a serious threat to human health in recent years. According to the 2020 World Cancer Report published by the World Health Organization's International Agency for Research on Cancer (IARC), cancer has gradually become the second leading cause of death worldwide, after heart disease. Colorectal cancer accounts for 10% of all cancer diagnoses, or approximately 1.93 million cases, with a mortality rate of 9.4%, ranking second only to lung cancer. Colorectal polyps are abnormal tissue growths in the colorectal mucosa and are precursors to colorectal cancer. Although polyps are initially benign, they can develop into colorectal cancer over time if left untreated. Polyps can be categorized as serrated adenomas, adenomatous polyps, and hyperplastic polyps. Adenomatous polyps are the direct cause of colorectal cancer. Clinically, every 1% increase in adenoma detection is associated with a 3% decrease in CRC risk.

[0003] Currently, colonoscopy is the most effective method for diagnosing colorectal diseases. Numerous studies have shown that early examination and timely, effective treatment for colorectal cancer patients can significantly reduce mortality. However, colonoscopy is highly dependent on the judgment of the endoscopist, and nearly 25% of polyps may be overlooked during the colonoscopy due to the clinician's skill and subjectivity. Accurately locating the outline of polyps is crucial for colonoscopy. Computer-assisted polyp segmentation systems can segment and mark the location of polyps in colonoscopy videos in real time, effectively reducing the probability of missed and misdiagnoses. Therefore, using deep learning-based image processing methods to improve existing colonoscopy technology is undoubtedly one of the most effective methods.

[0004] For the segmentation of colon polyp images, researchers have proposed a large number of different image segmentation methods, which can be mainly divided into two categories: colorectal polyp image segmentation algorithms based on manual features such as texture, geometric features, color, or gradient histograms. Ameling et al. proposed a texture analysis method based on Color Wavelet Covariance (CWC); Bernal et al. used the watershed algorithm as a texture-based technique to segment potential polyp areas; Tajbakhsh et al. used the Canny edge detection algorithm to extract edge features to distinguish polyp areas; Mamonov et al. used a shape-based method to detect polyp areas. Because polyps have similar color and texture to surrounding tissues and fuzzy boundaries increase the difficulty of segmentation, and models designed based on low-level features have poor generalization capabilities, these early solutions have a high false positive rate and risk of missed detection in polyp extraction, making them difficult to apply on a large scale.

[0005] Due to the tremendous success of convolutional neural networks (CNNs) in image analysis, deep learning-based algorithms have also been applied to medical image analysis. For the medical task of polyp segmentation, Akbari first proposed a fully convolutional network (FCN) model. FCNs replace the fully connected layers in CNNs with fully convolutional layers, achieving an end-to-end segmentation network. While segmentation accuracy significantly improved compared to traditional methods, it still fell short of clinical application. Olaf et al. proposed that the U-Net architecture exhibited significant superiority in medical image segmentation, leading to the development of numerous U-Net-based variants. The UNet++ network proposed by Zhou et al. uses a densely connected skip connection scheme to integrate multi-scale features, fuse features from different layers, and reduce the semantic differences between the encoder and decoder feature maps. Building on ResUet, Debesh Jha et al. proposed the ResUet++ model, systematically integrating ASPP, attention, and residual architectures into each layer of the network to improve segmentation accuracy. ACSNet utilizes local context attention and global context modules to extract more contextual features of the target region, and learns to aggregate contextual information through an adaptive selection module. These networks significantly improve the performance of U-Net-based networks. While the encoder-decoder structure of the U-Net has certain advantages in extracting multi-scale features, it often loses feature detail after a series of convolutions and downsampling, which is particularly disadvantageous for the segmentation of small and irregular polyps. Attention, derived from its role in human perception, can filter out unimportant information while retaining key information. Following the successful application of attention mechanisms in natural language processing and natural image processing, attention has also been introduced into medical image processing. Fan et al. designed a parallel reverse attention network (PraNet). The parallel decoder (PPD) aggregates high-level features to generate a guidance map to predict the rough polyp area, and uses a reverse attention module to mine the detailed information of the polyp boundary. However, it lacks bottom-up feature sharing. UACANet uses parallel axial attention in the encoder network for bottom-up feature sharing, which effectively compensates for the shortcomings of PraNet.

[0006] All of the aforementioned methods are based on CNNs. Due to the inherent limitations of convolutional operations, CNNs are unable to model long-range dependencies. Compared to CNNs, Transformers incorporate long-range dependencies and use self-attention mechanisms to capture global context. However, they lack spatial awareness when modeling local information, resulting in limited localization capabilities. To fully leverage the strengths of CNNs and Transformers, a number of segmentation methods have innovatively combined Transformers with CNNs.

[0007] Although many colonoscopic polyp segmentation methods have been developed, accurate segmentation of polyps remains a challenging task for two main reasons. On the one hand, most polyps have similar color and texture to normal tissues. At the same time, polyps have fuzzy boundaries, different sizes, irregular shapes, etc., which increase the difficulty of segmentation. On the other hand, the network has a complex structure, which increases the complexity of the network. Summary of the Invention

[0008] In view of this, the purpose of this paper is to provide a polyp image segmentation method based on the Pyramid Vision Transformer, which effectively improves the accuracy of network polyp segmentation while maintaining a simple structure. The method mainly consists of the Pyramid Vision Transformer (PVT), the Recurrent Criss-Cross Attention module (RCCA), and the Multi-scale Feature Fusion Perception module (MSFFP).

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A polyp image segmentation method based on pyramid vision Transformer, which includes the following steps:

[0011] S1,captures global context information through the pyramid vision Transformer module,obtaining feature maps of different resolutions from multiple extraction stages;

[0012] S2, extracting detail information from the shallow feature map of the first stage through the cyclic cross attention module to obtain shallow features;

[0013] S3, fuse the feature maps extracted in multiple stages through the multi-scale feature fusion perception module to extract deep features, and fuse the deep features with the shallow features for image segmentation;

[0014] S4. Design weighted binary cross entropy loss and weighted IoU loss to train the above process. After training, they are used to segment polyp images.

[0015] Furthermore, in step S1, the pyramid vision Transformer module includes multiple stages of feature encoders, each of which includes a Patch Embedding module and a Transformer Encoder module, and outputs a feature pyramid of {F1, F2, F3, F4} by adjusting the stride;

[0016] Given an input image size Several feature maps are obtained through the feature backbone network Among them C i ∈{64,128,320,512}, i∈{1,2,3,4}.

[0017] Further, in step S2, the first stage output feature F1 of the pyramid vision Transformer module is fed into the cyclic cross attention module to extract the shallow feature F containing detail information A , the process is:

[0018] Given features After linear transformation and 1×1 convolution, we get the Q, K, and V matrices, where

[0019] At each position u in the spatial dimension of Q, Q u , K u The relationship matrix of different elements is obtained through the inner product operation; the Softmax normalization is performed to obtain the attention weight map of each element

[0020] The weights are summed with the corresponding values ​​and added to the original input F u After adding, output the result F u ′ , which is expressed as follows:

[0021]

[0022] where · represents the inner product, represents the dimension of each u position K, i∈{1,…,H+W-1}, u represents the position;

[0023] A cross-attention module is cascaded after the first-level cross-attention module. Based on the feature F′ extracted by the original module, the context information of elements that are not in the same row and column is further aggregated, and the complete context information is finally learned. The expression is as follows:

[0024] F″=C Att (C Att (F1)

[0025] F A =conv 1×1 (F″)

[0026] Among them C Att represents cyclic cross attention and conv represents convolution.

[0027] Furthermore, in step S3, F4 is input into the multi-scale feature fusion perception module for processing. The multi-scale feature fusion perception module consists of four convolutional layers. The convolution kernels of different branches have different sizes, which are respectively composed of standard convolution with a convolution kernel size of K∈{1,3,5,7} and dilated convolution with a convolution kernel size of 3×3, and the dilation rate is R∈{1,3,5,7}.

[0028] The receptive field of the feature map in each branch is expanded by dilated convolution operations, and K×K convolution is implemented in K×1 and 1×K modes. The outputs of the four branches are concatenated and sent to point convolution to adjust the number of channels of the model, and then added to the initial input and the final result is output through the ReLU activation function.

[0029] Furthermore, the final result of the multi-scale feature fusion perception module is first adjusted through a 1×1 convolution layer, F H Then, the feature F is obtained by upsampling eight times through bilinear interpolation. M , and finally the feature map F A With F M After concatenation, the image is passed through two 3×3 standard convolutions and then upsampled to four times the original resolution using bilinear interpolation to output the segmentation mask.

[0030] F M =Up ×8 (F H )

[0031] F O =concat(F M, F A )

[0032] out=Up ×4 (Conv(Conv(F O )))

[0033] Among them, Up stands for upsampling, concat stands for concatenation, and Conv stands for convolution;

[0034] Furthermore, in step S5, the weighted binary cross entropy loss is used as part of the total loss, and the total loss function is expressed as:

[0035]

[0036] in, and They represent weighted binary cross entropy loss and weighted IoU loss, respectively, and their expressions are:

[0037]

[0038] Among them, y i represents the true label, represents the predicted probability and N represents the total number of pixels in the image.

[0039] The beneficial effects of the present invention are:

[0040] This paper uses the Pyramid Vision Transformer module (PVT) to more effectively capture global contextual information, significantly enhancing its global information learning capabilities compared to traditional CNNs. Furthermore, the introduction of the Multi-Scale Feature Fusion Perception Module (MSFF) enables the model to fuse multi-scale features, fully utilizing both high-level semantic information and low-level detail information, thereby improving segmentation accuracy.

[0041] To further enhance the ability to capture detailed information, the present invention uses a recurrent crisscross attention (RCCA) module after the first stage of the feature encoder to precisely capture details such as polyp boundaries, enabling accurate segmentation of subtle polyp boundaries. Furthermore, the RCCA module collects contextual information horizontally and vertically, modeling the full-image dependencies of local features to further optimize contextual information.

[0042] In terms of loss function design, the present invention adopts weighted binary cross entropy loss and weighted IoU loss to balance the learning of target and background areas, and improve the model's sensitivity to polyp boundaries and segmentation accuracy.

[0043] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0045] Figure 1 Schematic diagram of the overall process of the polyp image segmentation method of the present invention;

[0046] Figure 2Schematic diagram of the cross attention process in the cyclic cross attention module of the present invention;

[0047] Figure 3 Schematic diagram of the flow of cyclic cross attention in the cyclic cross attention module of the present invention;

[0048] Figure 4 Schematic diagram of the process of the multi-scale feature fusion perception module of the present invention;

[0049] Figure 5 Schematic diagram of the data set segmentation process under the embodiment;

[0050] Figure 6 This is an example of polyp image segmentation based on different data sets according to the present invention. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0053] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0054] See also Figures 1-6 , which is a polyp image segmentation method based on pyramid vision Transformer.

[0055] Example

[0056] This embodiment provides a specific implementation of a polyp image segmentation method based on a pyramid vision Transformer, which includes at least the following steps:

[0057] S1,captures global context information through the pyramid vision Transformer module,obtaining feature maps of different resolutions from multiple extraction stages;

[0058] S2, extracting detail information from the shallow feature map of the first stage through the cyclic cross attention module to obtain shallow features;

[0059] S3, fuse the feature maps extracted in multiple stages through the multi-scale feature fusion perception module to extract deep features, and fuse the deep features with the shallow features for image segmentation;

[0060] S4. Design weighted binary cross entropy loss and weighted IoU loss to train the above process. After training, they are used to segment polyp images.

[0061] Compared to CNNs, this feature encoder has a stronger ability to learn global information. Following the design philosophy of DeepLabv3+, the MSFF module is introduced in the final PVT layer to strengthen the deep features learned from the PVT backbone network. Generally speaking, features at different levels in the encoder contain different contextual information, and high-level features contain more semantic information, which helps locate objects. The multi-scale feature fusion perception module strengthens the learning of semantic information in high-level features while fusing multi-scale features. Low-level features contain rich detailed information, such as the size, shape, and boundaries of polyps, which helps capture the subtle boundaries of polyps. Therefore, the feature map output by the first stage of the feature encoder is passed through the recurrent cross attention module (RCCA) to obtain detailed information such as the boundaries of polyps.

[0062] In step S1 of this embodiment, the pyramid vision Transformer module includes multiple stages of feature encoding. By adjusting the stride to 4, 8, 16 and 32, the output feature pyramid is {F1, F2, F3, F4}. Given an input image size Four feature maps are obtained through the feature backbone network Among them C i ∈{64,128,320,512}, i∈{1,2,3,4}. That is, using PVTv2 to capture global context information and obtain stronger multi-scale feature representation, four feature maps of different resolutions are obtained, denoted as F1 to F4. Specifically, it includes the following process:

[0063] Phase 1 (i=1):

[0064] First, the given image is segmented into multiple patches by the Patch Embedding module, and these patches are mapped to a high-dimensional space. This step not only reduces the spatial resolution but also increases the feature dimension for subsequent processing.

[0065] Next, the high-dimensional features are transformed by the first Transformer Encoder module to capture global dependencies in the image, resulting in the first feature map F1.

[0066] Second stage (i = 2):

[0067] The feature map F1 output by the first stage is further segmented and mapped to a higher-dimensional feature space by the second Patch Embedding module. i The second Transformer Encoder module continues to transform the features to generate the second scale feature map F2.

[0068] The second Transformer Encoder module continues to transform the features to generate the second scale feature map F2.

[0069] Third stage (i = 3):

[0070] Similarly, the feature map F2 output by the second stage is processed by the third Patch Embedding module and the Transformer Encoder module to generate the third scale feature map F3.

[0071] Fourth stage (i = 4):

[0072] Finally, the feature map F3 output by the third stage is processed by the fourth Patch Embedding module and the Transformer Encoder module to generate the highest scale feature map F4.

[0073] In each stage, the Patch Embedding module is responsible for segmenting the image into fixed-size patches and increasing the channel dimension through linear projection. The Transformer Encoder module uses self-attention mechanisms to transform the features and capture global context information in the image.

[0074] In step S2 of the present embodiment, the RCCA removes background noise from the shallow encoder feature map F1 to enhance the representation of detailed information, which mainly includes texture, fine edges, and tiny structures. The first stage output feature map F1 of the pyramid visual Transformer module is denoted as F1. The feature F with boundary information is obtained after the convolution layer with a convolution kernel size of 1, batch normalization and a Relu activation function A Cross attention collects context information from horizontal and vertical directions, thereby realizing modeling of full image dependency on local feature representation. As shown in Figure 2 , given the feature , Q, K and V matrices are obtained after linear change and dimension reduction through 1x1 convolution, where Q u , K u At each position u in the spatial dimension of Q, Q The weight is weighted and summed with the corresponding value, and the original input F u is added to output the result F' u , which has the following expression form:

[0075]

[0076] Where · represents inner product, represents the dimension of K at each u position, i∈{1,…,H+W-1}, and u represents the position.

[0077] Since the polyp is similar in appearance to the color of the mucosa, in order to obtain key detail information such as the polyp edge, the feature tensor output by the first stage of PVTv2 is constructed through a recurrent cross attention module to build semantic association between the polyp and the background area, wherein each pixel collects effective context information from the semantic related pixels of the entire image. The cross attention module only aggregates context information in the vertical and horizontal directions of each position, for example, the blue block only aggregates context information of elements in the same row and column. This feature is sparse and incomplete, which is extremely disadvantageous for polyp boundary detail segmentation. In order to obtain dense context information, as shown in Figure 3 , a recurrent cross attention module is further proposed, that is, a cross attention module is cascaded after the first-level cross attention module, and the context information of elements not in the same row and column is further aggregated on the basis of the original module extracting the feature F'. Finally, complete context information is learned. As shown in Figure 3 , the feature F is output as F" after the recurrent cross attention (RCCA). The expression form is as follows:

[0078] F" = C Att C Att C

[0079]

[0080] Among them CC Att represents cyclic cross attention and conv represents convolution.

[0081] In step S3 of this embodiment, F4 is input into the MSFFP module to capture richer semantic information, such as Figure 4 The output of the deep feature F4 in the PVTv2 encoder is input into the multi-scale feature fusion perception module to extract multi-scale features and further enhance feature expression. This module consists of four convolutional layers. The convolution kernel sizes of different branches vary. They are composed of standard convolutions with kernel sizes of K∈{1,3,5,7} and dilated convolutions with kernel sizes of 3×3 and dilation rates of R∈{1,3,5,7}. The dilated convolution operation expands the receptive field of each branch, facilitating the capture of global contextual information across multiple receptive fields. K×K convolutions are implemented using K×1 and 1×K methods, significantly reducing the number of parameters. The outputs of the four branches are concatenated and fed into a point convolution to adjust the number of channels in the model. They are then added to the initial input and activated by the ReLU function to output the final result.

[0082] The final result of the MSFF module is first passed through a 1×1 convolution layer to adjust the channel, and then upsampled eight times by bilinear interpolation to obtain the feature F M , and finally the feature map F A With F M After concatenation, the image is passed through two 3×3 standard convolutions and then upsampled to four times the original resolution using bilinear interpolation to output the segmentation mask.

[0083] In step S4 of this embodiment, due to the large ratio of pixels between the polyp and background regions, the polyp image segmentation method of the present invention focuses on learning the background region. At the same time, the pixels near the boundary are very similar, which can easily lead to misclassification. In order to make this method treat the target and background regions equally, a weighted binary cross entropy loss is used as part of the total loss. Therefore, the total loss function can be written as:

[0084]

[0085] in, and Represent the weighted binary cross entropy loss and weighted IoU loss respectively. Their expressions are as follows:

[0086]

[0087] Among them, y i represents the true label, represents the predicted probability and N represents the total number of pixels in the image.

[0088] This example implements the network model on the Pytorch framework and conducts experiments on an NVIDIA RTX 3090 24GB GPU. Before training, the resolution of all images is uniformly adjusted to 352×352. During training, we follow a multi-scale training strategy and scale the images to [0.75, 1, 1.25]. In addition, data augmentation methods such as image cropping and random flipping are used to reduce the probability of model overfitting. End-to-end optimization is performed using the AdamW optimizer, with an initial learning rate of 1×10 -4 Weight decay is 1e -4 , batch_size is set to 8, and the model is trained for 100 epochs. During the test phase, the image resolution is also adjusted to 352×352.

[0089] This example uses five public polyp datasets as shown in Table 1 for experiments. First, the datasets Kvasir-SEG and CVC-ClinicDB are randomly divided into training sets and test sets in a ratio of 9:1. Specifically, 900 images from Kvasir-SEG and 550 images from CVC-ClinicDB are used for model training, and the remaining datasets together constitute the test set. The dataset division diagram is shown in the following figure. Figure 5 .

[0090] Table 1

[0091]

[0092] In order to quantitatively analyze and evaluate the polyp image segmentation method of the present invention, mean intersection over union (mIoU) and mean Dice coefficient (mDice) are used as the main evaluation indicators of the segmentation model. In addition, since many methods regard polyp segmentation as a salient object detection task, four other indicators commonly used in salient object detection tasks are also used to evaluate the performance of the model, namely, weighted Dice coefficient, Similarity metric (S α ), enhanced alignment metrics Mean absolute error (MAE). Among the six evaluation indicators, except for the indicator MAE, the smaller the better, the larger the other five indicators are, the better. The calculation formulas for Dice and IoU are as follows:

[0093]

[0094]

[0095] Among them, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

[0096]

[0097] Where PR represents precision, RE represents recall, and β represents the weight coefficient.

[0098] Assuming the output size is h×w:

[0099]

[0100] Among them, Pred represents the prediction result, GT represents the true ground label, and N = h × w represents the number of pixels.

[0101] In this example, the polyp image segmentation method of the present invention is named PMNet and compared with six advanced polyp segmentation models, namely UNet, ResNet++, PraNet, MSNet, HarDMSEG, and Polyp-PVT. The final results of the experiment on the Kvasir-SEG dataset are shown in Table 2:

[0102] Table 2

[0103]

[0104] As shown in Table 2, the PMNet of our invention achieves an mDice of 92% and an mIOU of 87.2% on the Kvasir-SEG dataset, which are 10.2% higher than the classic model UNet and 12.6% higher than the widely used PraNet.

[0105] Table 3 shows the mDice and mIOU indicators of PMNet of the present invention and other existing solutions on different data sets:

[0106] Table 3

[0107]

[0108] Comparing the mDice and mIOU metrics across different datasets reveals that our method achieves results of 0.920 and 0.872 on the Kvasir-SEG training set, and 0.939 and 0.895 on the CVC-ClinicDB training set, outperforming the other six models. The test results in Table 3 show that our PMNet achieves state-of-the-art performance on all four datasets, demonstrating its superior learning and generalization capabilities.

[0109] The above results show that the present invention achieves accurate segmentation with a simple encoder-decoder architecture. The inference results of the model PPNet on five public datasets are shown in Figure 2. Figure 6The raw data in rows 1 through 5 come from the Kvasir-SEG, CVC-ColonDB, ETIS, CVC-ClinicDB, and CVC300-T datasets, respectively. The PMNet model output is shown in the ninth column. Compared to the outputs of the other six methods, our model achieves better output margins and higher prediction accuracy.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A polyp image segmentation method based on pyramid vision Transformer, characterized by: It includes the following steps: S1,captures global context information through the pyramid vision Transformer module,obtaining feature maps of different resolutions from multiple extraction stages; S2, extracting detail information from the shallow feature map of the first stage through the cyclic cross attention module to obtain shallow features; In step S2, the first stage output features of the pyramid vision Transformer module are Feed into the recurrent cross-attention module to extract shallow features containing detailed information , the process is: Given features , after linear change After convolution dimensionality reduction, we get Q, K, and V matrices, where ; At each position u in the spatial dimension of Q, Q u , K u Obtain the relationship matrix of different elements through the inner product operation; Normalize to get the attention weight map of each element ; The weights are summed with the corresponding values ​​and added to the original input Output the result after addition , which is expressed as follows: in, represents the inner product, represents the dimension of each u position K, , u represents the position; A cross attention module is cascaded after the first-level cross attention module to extract features in the original module. On the basis of further aggregating the context information of elements that are not in the same row and column, we can finally learn the complete context information. , expressed as follows: in represents cross attention, conv represents convolution; S3, processing the feature maps extracted in multiple stages through the multi-scale feature fusion perception module, extracting deep features, and fusing the deep features with the shallow features for image segmentation; S4. Design weighted binary cross entropy loss and weighted IoU loss to train the above process. After training, they are used to segment polyp images.

2. The polyp image segmentation method based on pyramid vision Transformer according to claim 1, characterized in that: In step S1, the pyramid vision Transformer module includes multiple stages of feature encoders. Each stage of the feature encoder includes a Patch Embedding module and a Transformer Encoder module. By adjusting the stride, the feature pyramid output is ; Given an input image size Several feature maps are obtained through the feature backbone network ,in , .

3. The polyp image segmentation method based on pyramid vision Transformer according to claim 2, characterized in that: In step S3, The input is processed by the multi-scale feature fusion perception module, which consists of four convolutional layers. The convolution kernels of different branches have different sizes, which are respectively The convolution kernel is composed of a standard convolution and a dilated convolution kernel with a size of 3×3, and the dilation rate is ; The receptive field of each branch is expanded by dilated convolution operations. Convolution uses and The outputs of the four branches are concatenated and fed into point convolution to adjust the number of channels of the model, and then added to the initial input and the final result is output through the Relu activation function.

4. The polyp image segmentation method based on pyramid vision Transformer according to claim 3, characterized in that: The final result of the multi-scale feature fusion perception module is first adjusted through a 1×1 convolution layer, and then upsampled eight times by bilinear interpolation to obtain the feature , and finally we get the feature map and After concatenation, the image is passed through two 3×3 standard convolutions and then upsampled to four times the original resolution using bilinear interpolation to output the segmentation mask.

5. The polyp image segmentation method based on pyramid vision Transformer according to claim 4, characterized in that: In step S5, the weighted binary cross entropy loss is used as part of the total loss, and the total loss function is expressed as: in, and They represent weighted binary cross entropy loss and weighted IoU loss, respectively, and their expressions are: in, represents the true label, represents the predicted probability and N represents the total number of pixels in the image.

Images