A self-attention endoscopic polyp segmentation method based on PVT improvement

Abstract

The application discloses a self-attention endoscope polyp segmentation method based on PVT improvement, and relates to the technical field of medical image segmentation.The method comprises the following steps: collecting a plurality of gastrointestinal tract images, pre-processing and labeling the images to obtain image data and real labels; inputting the image data into a feature extraction network based on a PVT and a spatial channel self-attention module to obtain multi-scale features; inputting the multi-scale features into a multi-scale cross-fusion module to obtain four fused feature maps; inputting the fused feature maps into a decoder to obtain two prediction maps; inputting the real labels into the decoder to obtain two real labels, and performing loss calculation and iterative optimization on the two real labels and the prediction maps with the corresponding sizes respectively; and adding the two prediction maps after being restored to the size of the input images to obtain an endoscope polyp segmentation model.The application relates to the technical field of medical image segmentation, and solves the technical problem of low detection accuracy of small polyp regions and highly blurred edge polyps in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image segmentation, relates to deep learning technology, and particularly relates to a self-attention endoscopic polyp segmentation method based on PVT improvement. BACKGROUND

[0002] Colorectal cancer (CRC) has long been in the top three of cancer cases, seriously threatening human life and health. This cancer is usually triggered by colon polyps. Polyps are initially benign, but if not treated in time, they can turn malignant over time. Therefore, colonoscopy, as an effective CRC screening and prevention method, can provide information on the location and appearance of colorectal polyps, enabling doctors to discover and remove them before they develop into CRC. Some studies show that early colonoscopy can reduce the incidence of CRC by about 30%.

[0003] However, existing polyp detection mainly relies on endoscopy doctors to complete manually, which is not only a time-consuming and labor-intensive task, but also highly dependent on the experience and skills of doctors. Therefore, more and more computer-aided diagnosis systems are being developed to assist endoscopy doctors in making more accurate judgments.

[0004] In recent years, convolutional neural networks (CNN) have made significant progress in the field of medical image processing. Models improved based on FCN, U-Net, SegNet, and DeepLabV3+ have good segmentation capabilities in dealing with fuzzy edges and dramatic changes within polyps, but the current models may miss or incorrectly segment extremely small polyp regions and highly blurred edges, and do not fully utilize global information and long-distance context information. SUMMARY

[0005] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes a self-attention endoscopic polyp segmentation method based on PVT improvement, which is used to solve the technical problem of low detection accuracy of extremely small polyp regions and highly blurred edges in the existing method.

[0006] To achieve the above-mentioned purpose, the present application provides a self-attention endoscopic polyp segmentation method based on PVT improvement, comprising:

[0007] S1, a plurality of gastrointestinal tract images are collected, the plurality of gastrointestinal tract images are preprocessed and labeled to obtain a data set; wherein the data set includes image data and true labels;

[0008] S2, input the image data in the data set into a feature extraction network based on PVT and spatial channel self-attention module after data augmentation to obtain multi-scale features;

[0009] S3, constructing a multi-scale cross-fusion module based on up-sampling and Hadamard product, inputting the multi-scale features into the multi-scale cross-fusion module to obtain four fused feature maps;

[0010] S4, inputting the four fused feature maps into a decoder to obtain a first prediction map obtained by twice up-sampling and adding and a second prediction map obtained by thrice up-sampling and adding;

[0011] S5, inputting the real labels in the data set into the decoder to obtain a first real label and a second real label which have the same size as the first prediction map and the second prediction map respectively, performing loss calculation on the first real label and the second real label and the first prediction map and the second prediction map respectively by using a loss function, and performing iterative optimization according to the loss calculation result;

[0012] S6, restoring the first prediction map and the second prediction map to the size of the input image by bilinear interpolation and adding to obtain an endoscopic polyp segmentation model which inputs a patient's gastrointestinal tract image and outputs a polyp prediction image.

[0013] The input image is subjected to feature extraction by the feature extraction network based on PVT and spatial channel self-attention module to obtain multi-scale features with rich context information and global information; the PVT network can capture the global dependence in the image while maintaining sensitivity to local details, and in combination with the spatial channel self-attention module, the feature extraction network can more accurately identify the boundary and shape of the polyp and enhance the processing capacity for fuzzy edges and tiny polyps; the multi-scale cross-fusion module based on up-sampling and Hadamard product is constructed to enhance the complementarity between features and further improve the recognition ability of the endoscopic polyp segmentation model for polyps; the first prediction map and the second prediction map are obtained by using the decoder, and this deep supervision strategy can effectively improve the training efficiency and segmentation accuracy of the endoscopic polyp segmentation model; finally, by comparing the differences between the two prediction maps and the real labels, the model can continuously learn and optimize its own parameters to finally generate more accurate segmentation results.

[0014] Further, the gastrointestinal tract images are preprocessed and labeled, including:

[0015] S1-1, obtaining original videos of real endoscopic examinations in a hospital and splitting the original videos to obtain a plurality of frames of images;

[0016] S1-2, image pre-processing is performed on the plurality of frames of images, and a polyp region in the plurality of frames of pre-processed images is image-labeled by using a labeling tool, to obtain a plurality of label images;

[0017] S1-3, a plurality of label images and a plurality of frames of pre-processed images are one-to-one corresponding, to obtain a data set containing a plurality of image data and a plurality of real labels.

[0018] Further, the feature extraction network comprises an encoder and a spatial channel self-attention module;

[0019] The encoder is a backbone network of PVT.

[0020] The spatial channel self-attention module is composed of a spatial self-attention module and a channel self-attention module, and the fused self-attention output is obtained by the formula X'=conv3[conv 2(SA(conv1(X)))+conv 2(CA(conv1(X)))]; Wherein, X represents the input feature of the self-attention module, X' represents the output feature of the self-attention module, SA() represents the spatial self-attention module, CA() represents the channel self-attention module, conv1() represents the conv1 convolution block containing 3x3 convolution, conv2() represents the conv2 convolution block containing 1x1 convolution, Dropout layer and ReLU function, and conv3() represents the conv3 convolution block containing 1x1 convolution.

[0021] Further, the calculation formula of the spatial self-attention module is: Wherein, i, j represent different positions in the feature image, and i≠j, i∈S, j∈S, S represents a set of spatial positions of the feature image, Aij represents a feature vector at the jth position in the feature map A, c represents the number of channels, h represents the height, w represents the width, B i Bij represents a feature vector at the ith position in the feature map B obtained after convolution layer processing, C j Cij represents a feature vector at the jth position in the feature map C obtained after conv1 convolution block processing, D i Dij represents a feature vector at the ith position in the feature map D obtained after conv1 convolution block processing, and N=h×w represents the number of pixels, S ji Sij represents the spatial self-attention mapping, which is used to represent the influence of the ith position in the feature on the jth position, and a represents a scale parameter, which is a learnable parameter with an initial value of 0, N represents the number of pixels, Sij represents the spatial self-attention output of

[0022] ​Furthermore, the calculation formula of the channel self-attention module is: Among them, A p Represents the feature vector of the pth channel in the feature map A, A q represents the feature vector of the qth channel in the feature map A, and p≠q, p=1,2,…c, q=1,2,…c, X qp Represents the channel self-attention map, used to represent feature A q The influence of the p-th channel on the q-th channel, β represents the scale parameter, which is a learnable parameter with an initial value of 0. Indicates A q The channel self-attention output of .

[0023] As a pure Transformer backbone network, the PVT network's pyramidal structure demonstrates superiority over traditional CNN backbone networks in generating multi-scale feature maps. It can capture global dependencies in images while maintaining sensitivity to local details. Furthermore, the combination of spatial and channel self-attention modules—the spatial and channel self-attention modules—not only fully leverages PVT's strengths in capturing global information, but also supplements this with deep mining of local details and inter-channel relationships through the spatial and channel self-attention modules. This results in more comprehensive and informative multi-scale features, laying a solid foundation for subsequent multi-scale feature fusion and polyp segmentation tasks.

[0024] Furthermore, the calculation formula of the multi-scale cross fusion module is: Among them, H represents the Hadamard product of all elements in the brackets, f n represents the multi-scale features, f m ' represents the four fused feature maps, and f1∈R c×h×w , n={1,2,3,4},m={1,2,3,4},up() indicates that the upsampling layer DySample is used to perform the upsampling on f n Perform upsampling operation, conv() represents f n Apply 3x3 convolution, donwn() represents f n Perform downsampling operation.

[0025] The multi-scale cross-fusion module adjusts the feature image resolution through up- and down-sampling techniques, and uses the Hadamard product operation to fuse feature information from different levels, making full use of the advantages of features at different scales to generate a fused feature map that contains both global semantic information and fine details, enabling the model to more comprehensively capture and analyze image content when processing complex images.

[0026] Further, the decoder comprises an up-sampling operation and multi-level feature fusion, and the output of the decoder is obtained by formula ; wherein, P1 represents the first prediction map, P2 represents the second prediction map, f m ' represents the fused four feature maps, up() represents the up-sampling operation, that is, the input feature is up-sampled to twice the size by using a learnable up-sampling layer DySample, and the calculation process of P1 and P2 represents the multi-level feature fusion, that is, the features are added and fused by using a progressive up-sampling layer.

[0027] Further, the calculation formula of the loss function is: ; wherein, represents a weighted intersection joint loss function, represents a weighted binary cross-entropy loss function, G1 represents the first real label, and G2 represents the second real label.

[0028] The loss between the first prediction map, the second segmentation prediction and the first real label, the second real label is calculated by using the weighted intersection joint loss function and the weighted binary cross-entropy loss function, and actually, the training speed is accelerated and the generalization ability of the model is improved by using the deep supervision strategy. At the same time, the intersection joint loss function focuses on evaluating the overlapping degree of the predicted segmentation region and the real polyp region, helping the model to more accurately capture the outline and shape of the polyp; and the weighted binary cross-entropy loss function compares the predicted probability of each pixel point with the real label, prompting the model to make more accurate predictions at the pixel level; the combination of the two loss functions can more comprehensively evaluate the segmentation performance of the model and guide the model to continuously optimize in the training process.

[0029] Compared with the prior art, the beneficial effects of the present application are:

[0030] By introducing the feature extraction pyramid structure based on PVT and the designed double self-attention mechanism, the capturing ability of the model for global information and long-distance context information is improved, so that the model is more accurate in identifying the boundary and shape of the polyp; by using the multi-scale cross-fusion module based on up-sampling and Hadamard product, the complementarity and fusion of multi-scale features are realized, so that the model can capture global semantic information and local fine details at the same time when processing complex images, thereby improving the accuracy of polyp segmentation; finally, combined with the deep supervision strategy, two prediction maps are output in the decoding process, and the loss is calculated by using the weighted intersection joint loss function and the weighted binary cross-entropy loss function, respectively, and the training efficiency and segmentation accuracy of the model are effectively improved by using the supervision mode of the double prediction maps. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0032] Figure 1 A technical flow diagram of a self-attention endoscopic polyp segmentation method based on PVT improvement provided by the present application is shown in the figure.

[0033] Figure 2 A structural diagram of a spatial channel self-attention module provided by the present application is shown in the figure.

[0034] Figure 3 A structural diagram of a multi-scale cross-fusion module provided by the present application is shown in the figure.

[0035] Figure 4 A structural diagram of an endoscopic polyp segmentation model provided by the present application is shown in the figure.

[0036] Figure 5 An input image and a prediction result diagram of an endoscopic polyp segmentation model provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0037] The technical solutions of the present application will be described in detail below with reference to the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0038] Please refer to Figure 1 - Figure 5 The embodiments of the present application provide a self-attention endoscopic polyp segmentation method based on PVT improvement, which comprises:

[0039] S1, a plurality of gastrointestinal tract images are collected, and the plurality of gastrointestinal tract images are preprocessed and labeled to obtain a data set; wherein the data set comprises image data and real labels;

[0040] S2, after data augmentation of the image data in the data set, the image data is input into a feature extraction network based on PVT and a spatial channel self-attention module for feature extraction to obtain multi-scale features;

[0041] S3, a multi-scale cross-fusion module is constructed based on up-sampling and Hadamard product, and the multi-scale features are input into the multi-scale cross-fusion module to obtain four fused feature maps;

[0042] S4, inputting the fused four feature maps into a decoder to obtain a first prediction map obtained by twice upsampling and adding and a second prediction map obtained by thrice upsampling and adding;

[0043] S5, inputting the real labels in the data set into the decoder to obtain first real labels and second real labels which are respectively same in size with the first prediction map and the second prediction map, performing loss calculation on the first real labels and the second real labels and the first prediction map and the second prediction map respectively by using a loss function, and performing iterative optimization according to a loss calculation result;

[0044] S6, restoring the first prediction map and the second prediction map to the input image size by bilinear interpolation and adding to obtain an endoscopic polyp segmentation model with the input being a patient gastrointestinal tract image and the output being a polyp prediction image.

[0045] In the embodiment, first, real gastrointestinal endoscopy videos recorded by doctors during endoscopy are obtained from a hospital, and then the original videos are split into a plurality of frame images according to a preset frame interval by using a programming script, for subsequent image processing and labeling. Before image labeling, the frame images are first subjected to a preprocessing operation to improve image quality and labeling accuracy; the preprocessing operation specifically includes scale normalization, random flipping and cropping operation; then, an image labeling tool, such as labelimg, RectLabel or OpenCV, is used to create a bounding box around the polyp area by continuously adjusting the polygon size according to the guidance of professional doctors, to obtain a label data set with the polyp being white and the background being black; then, the original data is divided according to a preset proportion, for example, 8:2, to obtain a training set and a verification set containing image data and real labels.

[0046] Next, the training set is input into the built endoscopic polyp segmentation model for model training. Specifically, first, the images in the training set are input into the feature extraction network composed of the backbone network of PVT and the spatial channel self-attention module for feature extraction to obtain multi-scale features containing rich context information and global information. The backbone network of PVT is a network combining the Transformer architecture and the pyramid structure, which gradually reduces the size of the feature map through multiple stages to form feature pyramids of different scales. Specifically, at the beginning of each stage, the input image or the feature map obtained from the previous stage is divided into a series of fixed-size image blocks, which are then flattened and mapped to a high-dimensional space through linear transformation to generate feature sequences, which are then input into multiple Transformer encoders for further processing. Finally, through the processing of multiple stages, a feature pyramid containing multiple scale feature maps is generated. Next, in order to further improve the accuracy and effectiveness of feature extraction, a spatial channel self-attention module is added after each scale feature of the feature pyramid to extract more comprehensive and informative features, obtaining multi-scale features f1∈R C×H×W ,

[0047] The spatial channel self-attention module in this embodiment is composed of a spatial self-attention module and a channel self-attention module, and the outputs of the two modules are fused through the formula X' = conv3[conv2(SA(conv1(X)))+conv2(CA(conv1(X)))] to obtain; wherein X represents the input feature of the self-attention module, X' represents the output feature of the self-attention module, SA() represents the spatial self-attention module, CA() represents the channel self-attention module, conv1() represents 3x3 convolution, conv2() represents a combined convolution block containing 1x1 convolution, Dropout layer and ReLU function, and conv3() represents 1x1 convolution.

[0048] The spatial self-attention module is used to capture the context relationship between different spatial positions in the image and enhance the spatial dependency of the feature map, which is processed through the formula where i, j represent different positions in the feature image, and i≠j, i∈S, j∈S, S represents the spatial position set of the feature map, represents the feature vector at the jth position in the feature map A, c represents the number of channels, h represents the height, w represents the width, and B i represents the feature vector at the ith position in the feature map B obtained after convolution layer processing, C j represents the feature vector at the jth position in the feature map C obtained after conv1 convolution block processing, D irepresents the feature vector at the i-th position in the feature map D obtained after processing by the conv1 convolution block, and N = h × w, represents the number of pixels, S ji Represents the spatial self-attention map, used to represent features The influence of the i-th position on the j-th position, α represents the scale parameter, which is a learnable parameter with an initial value of 0 and represents the number of pixels. express The spatial self-attention output of

[0049] Specifically, the input features Will Input three conv1 convolution blocks respectively for convolution operation, and obtain three sizes of Then adjust the shape of these three feature maps to get the size of Feature maps: B, C and D, then C j The transpose and B i Multiply and use the softmax layer to calculate the spatial attention map S ji ∈R N×N ; and then Adjust the shape to S ji The transpose and D i Multiply, reshape the result into And multiply it by a learnable scale parameter α, and then add it to the feature Perform element-by-element summation to obtain the final output

[0050] The channel self-attention module focuses on the correlation between different channels in the feature map, improving the richness and accuracy of feature representation. It uses the formula Get the output features; where A p Represents the feature vector of the pth channel in the feature map A, A q represents the feature vector of the qth channel in the feature map A, and p≠q, p=1,2,…c, q=1,2,…c, X qp Represents the channel self-attention map, used to represent feature A q The influence of the p-th channel on the q-th channel, β represents the scale parameter, which is a learnable parameter with an initial value of 0. Indicates A q The channel self-attention output of .

[0051] Specifically, the original features Reshape into A p Multiply by A q Transpose The softmax layer is used to calculate the channel attention map Then X qp is transposed and multiplied by A p , and their results are reshaped to Finally, it is multiplied by a learnable scale parameter β, and an element-wise sum operation is performed with the feature A q to obtain the final output

[0052] During the entire feature extraction process, the backbone network of PVT achieves a comprehensive understanding of the image from local to global through its unique pyramid structure and the combination of Transformer encoder layers; the introduction of the spatial channel self-attention module further enhances the network's ability to capture complex scenes and subtle features; this design enables the endoscopic polyp segmentation model of the present application to exhibit higher robustness and accuracy when processing challenging medical images.

[0053] Next, in order to effectively integrate multi-scale features, the multi-scale features obtained through the feature extraction network are input into a multi-scale cross-fusion module based on upsampling and Hadamard product, and further fusion processing is performed through the formula ; where H represents the Hadamard product of all elements in the parentheses, f n represents the above multi-scale features, f m ' represents the four fused feature maps, and n = {1, 2, 3, 4}, m = {1, 2, 3, 4}, up() represents the up-sampling operation of f n using a learnable up-sampling layer DySample, conv() represents the application of a 3x3 convolution to f n , and donwn() represents the down-sampling operation of f n . Specifically, first, the features of different scales are unified to the target size using up-sampling and down-sampling, where the target size includes the size of each scale of feature map, i.e. there are 4 target sizes, then the Hadamard product is used to multiply the feature maps after up-sampling and down-sampling element by element, to obtain the four fused feature maps.

[0054] Next, a double-decoder structure is used to decode the four fused feature maps, and the output of the polyp segmentation model, i.e. the first prediction map P1 and the second prediction map P2, is obtained through the formula ; specifically, first, f m '(m = {1, 2, 3, 4}) is up-sampled and summed to the size of f1' through each level, to obtain P1, then the feature map of size f2' obtained by step-by-step up-sampling is summed and output to obtain P2;

[0055] Finally, the training of the polyp segmentation model also includes loss calculation and iterative optimization of the model; specifically, the corresponding true label is first passed to the decoder to obtain a first true label G1 and a second true label G2 which are the same size as P1 and P2 respectively, and then the final loss value is calculated by the formula , and then the loss value is passed back to the network through back propagation to update the weight and bias parameters of the model, and the loss value is continuously reduced in an iterative optimization manner, thereby gradually improving the segmentation accuracy and generalization ability of the model; wherein, represents a weighted intersection joint loss function, represents a weighted binary cross-entropy loss function.

[0056] At the same time, during the training process, the accuracy value of the current model training result is also evaluated by calculating the evaluation of the predicted graph using the true label. Specifically, the first predicted graph and the second predicted graph are first added by bilinear interpolation to obtain the final predicted graph Y, and then the true label is calculated by the evaluation index calculation formula to obtain the model evaluation result; wherein DSC represents the Dice coefficient, which calculates the overlap between the predicted segmentation and the true label, and the higher the value, the more overlap between the predicted segmentation and the true label, i.e. the better the segmentation effect; IoU represents the intersection over union, which measures the overlap ratio between the predicted segmentation and the true label, and the closer the value to 1, the higher the overlap between the predicted segmentation and the true label, i.e. the better the segmentation effect. Finally, the model weight with the highest evaluation accuracy during the training process is saved, which is used to realize input of patient gastrointestinal tract images and can output high-precision polyp segmentation images.

[0057] Some data in the above formula is calculated by removing the dimension and taking its numerical value, and the formula is obtained by software simulation of a large amount of data to obtain a formula closest to the real situation; the preset parameters and the preset threshold in the formula are set by a person skilled in the art according to the actual situation or obtained by a large amount of data simulation.

[0058] Working principle of the present application:

[0059] S1, data set preparation and preprocessing, by acquiring real endoscopic examination original video in the hospital, the video is split into frame images, and the frame images after preprocessing are labeled by using a labeling tool to obtain a data set with polyp images and true labels;

[0060] S2, feature extraction and self-attention mechanism, the pyramid features of the input image are extracted using the backbone network of PVT, and then the multi-scale features containing rich context information and global information are obtained through the spatial channel self-attention module;

[0061] S3, multi-scale feature fusion, a multi-scale cross-fusion module based on up-sampling and Hadamard product is used to cross-fuse multi-scale features to generate four fused feature maps;

[0062] S4, decoding and prediction, the four fused feature maps are input into a decoder to further fuse the features through step-by-step up-sampling and element-by-element addition to generate two polyp prediction maps with different sizes;

[0063] S5, loss calculation and optimization, real labels are input into the decoder to generate two real labels with the same size as the two polyp prediction maps, and the total loss is calculated by using a weighted intersection joint loss function and a weighted binary cross-entropy loss function on the two prediction maps and the corresponding real labels, and finally the model parameters are adjusted according to the loss result.

[0064] The above examples are only used to illustrate the technical method of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A self-attention endoscopic polyp segmentation method based on PVT improvement, characterized by: include S1, collecting a number of gastrointestinal images, preprocessing and labeling the gastrointestinal images to obtain a data set; wherein the data set includes image data and true labels; S2, after data enhancement, the image data in the dataset is input into a feature extraction network based on PVT and spatial channel self-attention module for feature extraction to obtain multi-scale features; S3, constructing a multi-scale cross fusion module based on up-sampling and down-sampling and Hadamard product, inputting the multi-scale features into the multi-scale cross fusion module, and obtaining four fused feature maps; S4, inputting the fused four feature maps into a decoder to obtain a first prediction map that is upsampled twice and added together and a second prediction map that is upsampled three times and added together; S5, inputting the true labels in the data set into the decoder to obtain first and second true labels of the same size as the first predicted image and the second predicted image, respectively, performing loss calculation on the first and second true labels and the first and second predicted images using a loss function, and performing iterative optimization based on the loss calculation results; S6. Restoring the first prediction image and the second prediction image to the input image size through bilinear interpolation and adding them together to obtain an endoscopic polyp segmentation model whose input is the patient's gastrointestinal tract image and output is the polyp prediction image.

2. The method for self-attention endoscopic polyp segmentation based on PVT improvement according to claim 1, characterized in that: Preprocessing and labeling the gastrointestinal images include: S1-1, obtain the original video of the real endoscopy examination in the hospital and split it into several frames; S1-2, performing image preprocessing on the plurality of frames of images, and using a labeling tool to label polyp regions in the plurality of preprocessed frames of images to obtain a plurality of labeled images; S1-3, a plurality of label images are matched one-to-one with a plurality of preprocessed frame images to obtain a data set containing a plurality of image data and a plurality of real labels.

3. The method for polyp segmentation in an endoscopy based on self-attention improved PVT according to claim 1, characterized in that: The feature extraction network includes an encoder and a spatial channel self-attention module; Wherein, the encoder is the backbone network of PVT; The spatial channel self-attention module consists of a spatial self-attention module and a channel self-attention module, and is constructed by the formula Get the fused self-attention output; where X represents the input feature of the spatial channel self-attention module, Represents the output features of the spatial channel self-attention module, SA() represents the spatial self-attention module, CA() represents the channel self-attention module, conv1() represents the conv1 convolution block containing 3x3 convolution, conv2() represents the conv2 convolution block containing 1x1 convolution, Dropout layer and ReLU function, and conv3() represents the conv3 convolution block containing 1x1 convolution.

4. The method for polyp segmentation in an endoscopy based on self-attention improved PVT according to claim 3, characterized in that: The calculation formula of the spatial self-attention module is: ; where i, j represent different positions in the feature image, and i≠j, i∈S, j∈S, S represents the spatial position set of the feature map, Represents the feature vector at the jth position in the feature map A, c represents the number of channels, h represents height, and w represents width. Represents the feature vector at the i-th position in the feature map B obtained after processing by the convolution layer, Represents the feature vector at the jth position in the feature map C obtained after processing by the conv1 convolution block, represents the feature vector at the i-th position in the feature map D obtained after processing by the conv1 convolution block, and 、 、 , N=h×w, represents the number of pixels, Represents the spatial self-attention map, used to represent features The influence of the i-th position on the j-th position, α represents the scale parameter, which is a learnable parameter with an initial value of 0 and represents the number of pixels. express The spatial self-attention output of 5. The method for self-attention endoscopic polyp segmentation based on PVT improvement according to claim 4 is characterized in that: The calculation formula of the channel self-attention module is: ;in, Represents the feature vector of the p-th channel in the feature map A, Represents the feature vector of the qth channel in the feature map A, and p≠q, p=1,2,…c, q=1,2,…c, Represents channel self-attention map, used to represent features The influence of the p-th channel on the q-th channel, β represents the scale parameter, which is a learnable parameter with an initial value of 0. express The channel self-attention output of .

6. The method for polyp segmentation in an endoscopy based on self-attention improved PVT according to claim 5, characterized in that: The calculation formula of the multi-scale cross fusion module is: ; Where H represents the Hadamard product of all elements in the brackets. represents the multi-scale features, represents the four fused feature maps, and , , , , n={1,2,3,4}, m={1,2,3,4}, up() represents the use of the learnable upsampling layer DySample to Perform upsampling operation, conv() represents Apply 3x3 convolution, donwn() means Perform downsampling operation.

7. The method for self-attention endoscopic polyp segmentation based on PVT improvement according to claim 6, characterized in that: The decoder includes upsampling operations and multi-level feature fusion, through the formula Get the output of the decoder; where, represents the first prediction graph, represents the second prediction graph, represents the four fused feature maps, up() represents the upsampling operation, that is, the input features are upsampled to twice the size using the learnable upsampling layer DySample, 、 The calculation process represents the multi-level feature fusion, that is, the features are added and fused through the upsampling layers step by step.

8. The method for self-attention endoscopic polyp segmentation based on PVT improvement according to claim 7, characterized in that: The calculation formula of the loss function is: ,in, represents the weighted intersection loss function, represents the weighted binary cross entropy loss function, represents the first true label, represents the second true label.

Images