PVT improvement-based self-attention endoscopic polyp segmentation method

Abstract

The invention discloses an improved self-attention endoscopic polyp segmentation method based on PVT, and the method comprises the steps: collecting a plurality of gastrointestinal tract images, and carrying out the preprocessing and marking of the images, and obtaining image data and real labels; inputting the image data into a feature extraction network based on PVT and a space channel self-attention module to obtain multi-scale features; inputting the multi-scale features into a multi-scale cross fusion module to obtain four fused feature maps; inputting the fused feature map into a decoder to obtain two prediction maps; inputting the real labels into a decoder to obtain two real labels, and respectively carrying out loss calculation and iterative optimization on the two real labels and the prediction maps with the corresponding sizes; and recovering the two prediction images to the size of the input image, and adding the two prediction images to obtain an endoscopic polyp segmentation model. The invention relates to the technical field of medical image segmentation, and solves the technical problem that the detection precision of a small polyp region and a polyp with a highly fuzzy edge is not high in the existing method.

Description

Technical Field

[0001] The present invention belongs to the field of medical image segmentation and relates to deep learning technology, and specifically is a self-attention endoscopic polyp segmentation method based on PVT improvement. Background Art

[0002] Colorectal cancer (CRC) has long been among the top three cancer cases, posing a serious threat to human life and health. This cancer is often caused by colon polyps. Polyps are initially benign, but if not treated in time, they may turn malignant over time. Therefore, colonoscopy, as an effective means of CRC screening and prevention, can provide information on the location and appearance of colorectal polyps, enabling doctors to detect and remove them in time before polyps develop into CRC. Some studies have shown that early colonoscopy can reduce the incidence of CRC by about 30%.

[0003] However, existing polyp detection mainly relies on manual work by endoscopists, which is not only a labor-intensive task, but also highly dependent on the doctor's experience and skills. Therefore, more and more computer-aided diagnosis systems are being developed to assist endoscopists in making more accurate judgments.

[0004] In recent years, convolutional neural networks (CNN) have made significant progress in the field of medical image processing. Models improved based on FCN, U-Net, SegNet and DeepLabV3+ have good segmentation capabilities in dealing with blurred edges and drastic changes inside polyps. However, the current models may miss or incorrectly segment extremely small polyp areas and highly blurred edges, and the use of global information and long-distance contextual information is not sufficient. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a self-attention endoscopic polyp segmentation method based on PVT improvement, which is used to solve the technical problem of low detection accuracy of polyps in extremely small polyp areas and highly blurred edges in the prior art.

[0006] To achieve the above object, the present invention provides a self-attention endoscopic polyp segmentation method based on PVT improvement, comprising:

[0007] S1, collecting a number of gastrointestinal images, preprocessing and labeling the gastrointestinal images to obtain a data set; wherein the data set includes image data and true labels;

[0008] S2, after data enhancement, the image data in the data set is input into a feature extraction network based on PVT and spatial channel self-attention module for feature extraction to obtain multi-scale features;

[0009] S3, constructing a multi-scale cross fusion module based on up and down sampling and Hadamard product, inputting the multi-scale features into the multi-scale cross fusion module, and obtaining four fused feature maps;

[0010] S4, inputting the fused four feature maps into a decoder to obtain a first prediction map that is upsampled twice and added and a second prediction map that is upsampled three times and added;

[0011] S5, inputting the true labels in the data set into the decoder to obtain the first true label and the second true label which are the same size as the first prediction map and the second prediction map respectively, performing loss calculation on the first true label and the second true label and the first prediction map and the second prediction map respectively using a loss function, and performing iterative optimization according to the loss calculation result;

[0012] S6, restoring the first prediction image and the second prediction image to the input image size through bilinear interpolation and adding them together to obtain an endoscopic polyp segmentation model whose input is the patient's gastrointestinal tract image and output is the polyp prediction image.

[0013] The feature extraction network based on PVT and spatial channel self-attention module is used to extract features from the input image, and multi-scale features with rich contextual information and global information are obtained. The PVT network can capture the global dependencies in the image while maintaining sensitivity to local details. Combined with the spatial channel self-attention module, the feature extraction network can more accurately identify the boundaries and shapes of polyps, and enhance the processing capabilities of fuzzy edges and tiny polyps. By constructing a multi-scale cross-fusion module based on upsampling and Hadamard product, the complementarity between features is enhanced, and the polyp recognition ability of the endoscopic polyp segmentation model is further improved. The decoder is used to obtain the first prediction map and the second prediction map. This deep supervision strategy can effectively improve the training efficiency and segmentation accuracy of the endoscopic polyp segmentation model. Finally, by comparing the differences between the two prediction maps and the true labels, the model can continuously learn and optimize its own parameters, and ultimately generate more accurate segmentation results.

[0014] Furthermore, the plurality of gastrointestinal tract images are preprocessed and annotated, including:

[0015] S1-1, obtaining the original video of the real endoscopy examination in the hospital and splitting it to obtain several frames of images;

[0016] S1-2, performing image preprocessing on the plurality of frames of images, and using a labeling tool to label the polyp regions in the plurality of frames of images after preprocessing, to obtain a plurality of label images;

[0017] S1-3, a plurality of label images are matched one by one with a plurality of preprocessed frame images to obtain a data set including a plurality of image data and a plurality of real labels.

[0018] Furthermore, the feature extraction network includes an encoder and a spatial channel self-attention module;

[0019] Wherein, the encoder is a backbone network of PVT;

[0020] The spatial channel self-attention module is composed of a spatial self-attention module and a channel self-attention module, and the fused self-attention output is obtained by the formula X'=conv3[conv 2(SA(conv1(X)))+conv 2(CA(conv1(X)))]; wherein X represents the input feature of the self-attention module, X' represents the output feature of the self-attention module, SA() represents the spatial self-attention module, CA() represents the channel self-attention module, conv1() represents the conv1 convolution block including 3x3 convolution, conv2() represents the conv2 convolution block including 1x1 convolution, Dropout layer and ReLU function, and conv3() represents the conv3 convolution block including 1x1 convolution.

[0021] Furthermore, the calculation formula of the spatial self-attention module is: Where i, j represent different positions in the feature image, and i≠j, i∈S, j∈S, S represents the spatial position set of the feature map, represents the feature vector at the jth position in the feature map A, c represents the number of channels, h represents the height, w represents the width, B i represents the feature vector at the i-th position in the feature map B obtained after the convolution layer processing, C j represents the feature vector at the jth position in the feature map C obtained after processing by the conv1 convolution block, D i represents the feature vector at the i-th position in the feature map D obtained after processing by the conv1 convolution block, and N = h × w, indicating the number of pixels, S ji Represents the spatial self-attention map, used to represent features The influence of the i-th position on the j-th position in , α represents the scale parameter, which is a learnable parameter with an initial value of 0 and represents the number of pixels, express The spatial self-attention output of .

[0022] Furthermore, the calculation formula of the channel self-attention module is: Among them, A p Represents the feature vector of the pth channel in the feature map A, A q represents the feature vector of the qth channel in the feature map A, and p≠q, p=1,2,…c, q=1,2,…c, X qp Represents the channel self-attention map, which is used to represent feature A q The influence of the p-th channel on the q-th channel in β represents the scale parameter, which is a learnable parameter with an initial value of 0. Indicates A q The channel self-attention output of .

[0023] As a pure Transformer backbone network, the pyramid structure of the PVT network shows a stronger advantage than the traditional CNN backbone network in generating multi-scale feature maps. It can capture the global dependencies in the image while maintaining sensitivity to local details. On this basis, combined with the spatial channel self-attention module, namely the spatial self-attention module and the channel self-attention module, not only does it give full play to the advantages of PVT in capturing global information, but it also supplements the deep mining of local details and inter-channel relationships through the spatial channel self-attention module, thereby obtaining more comprehensive and information-rich multi-scale features, laying a solid foundation for the subsequent multi-scale feature fusion and polyp segmentation tasks.

[0024] Furthermore, the calculation formula of the multi-scale cross fusion module is: Among them, H represents the Hadamard product of all elements in the brackets, f n represents the multi-scale features, f m ' represents the four fused feature maps, and f1∈R c×h×w , n={1,2,3,4},m={1,2,3,4},up() indicates using the learnable upsampling layer DySample to upsample f n Perform upsampling operation, conv() represents f n Apply 3x3 convolution, donwn() represents f n Perform downsampling operation.

[0025] The multi-scale cross-fusion module adjusts the feature image resolution through up and down sampling techniques, uses the Hadamard product operation to fuse feature information from different levels, and fully utilizes the advantages of features at different scales to generate a fused feature map that contains both global semantic information and fine details, enabling the model to more comprehensively capture and analyze image content when processing complex images.

[0026] Furthermore, the decoder includes upsampling operations and multi-level feature fusion, through the formula Get the output of the decoder; where P1 represents the first prediction image, P2 represents the second prediction image, and f m ' represents the four fused feature maps, up() represents the upsampling operation, that is, the input features are upsampled to twice the size using the learnable upsampling layer DySample, and the calculation process of P1 and P2 represents the multi-level feature fusion, that is, the features are added and fused through the upsampling layers step by step.

[0027] Furthermore, the calculation formula of the loss function is: in, represents the weighted intersection loss function, represents the weighted binary cross entropy loss function, G1 represents the first true label, and G2 represents the second true label.

[0028] The weighted intersection joint loss function and the weighted binary cross entropy loss function are used to calculate the losses between the first prediction map, the second segmentation prediction, and the first true label and the second true label, respectively. In fact, the deep supervision strategy is used to speed up the training and improve the generalization ability of the model. At the same time, the intersection joint loss function focuses on evaluating the overlap between the predicted segmentation area and the real polyp area, helping the model to more accurately capture the outline and shape of the polyp; while the weighted binary cross entropy loss function compares the predicted probability of each pixel with the real label, prompting the model to make more accurate predictions at the pixel level; combining these two loss functions can more comprehensively evaluate the segmentation performance of the model and guide the model to continuously optimize during the training process.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] By introducing a PVT-based feature extraction pyramid structure and designing a dual self-attention mechanism, the model's ability to capture global information and long-distance contextual information is improved, making the model more accurate in identifying the boundaries and shapes of polyps; through a multi-scale cross-fusion module based on up- and down-sampling and Hadamard product, the complementarity and fusion of multi-scale features are achieved, so that the model can capture global semantic information and local fine details at the same time when processing complex images, thereby improving the accuracy of polyp segmentation; finally, combined with the deep supervision strategy, two prediction maps are output during the decoding process, and the losses are calculated respectively by the weighted intersection joint loss function and the weighted binary cross entropy loss function. The supervision method of dual prediction maps is used to effectively improve the training efficiency and segmentation accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A technical flow chart of a self-attention endoscopic polyp segmentation method based on PVT improvement provided by the present invention;

[0033] Figure 2 A schematic diagram of the structure of the spatial channel self-attention module provided by the present invention;

[0034] Figure 3 A schematic diagram of the structure of a multi-scale cross-fusion module provided by the present invention;

[0035] Figure 4 A schematic diagram of the structure of the endoscopic polyp segmentation model provided by the present invention;

[0036] Figure 5 Schematic diagram of the input image and prediction results of the endoscopic polyp segmentation model provided by the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1 - Figure 5 The embodiment of the present invention provides a self-attention endoscopic polyp segmentation method based on PVT improvement, including:

[0039] S1, collecting a number of gastrointestinal images, preprocessing and labeling the gastrointestinal images to obtain a data set; wherein the data set includes image data and true labels;

[0040] S2, after data enhancement, the image data in the data set is input into a feature extraction network based on PVT and spatial channel self-attention module for feature extraction to obtain multi-scale features;

[0041] S3, constructing a multi-scale cross fusion module based on up and down sampling and Hadamard product, inputting the multi-scale features into the multi-scale cross fusion module, and obtaining four fused feature maps;

[0042] S4, inputting the fused four feature maps into a decoder to obtain a first prediction map that is upsampled twice and added and a second prediction map that is upsampled three times and added;

[0043] S5, inputting the true labels in the data set into the decoder to obtain the first true label and the second true label which are the same size as the first prediction map and the second prediction map respectively, performing loss calculation on the first true label and the second true label and the first prediction map and the second prediction map respectively using a loss function, and performing iterative optimization according to the loss calculation result;

[0044] S6, restoring the first prediction image and the second prediction image to the input image size through bilinear interpolation and adding them together to obtain an endoscopic polyp segmentation model whose input is the patient's gastrointestinal tract image and output is the polyp prediction image.

[0045] In this embodiment, first, a real gastrointestinal endoscopy video recorded by a doctor during an endoscopy is obtained from the hospital, and then the original video is split into several frame images according to a preset frame interval through a programming script for subsequent image processing and annotation. Before image annotation, the frame image must first be preprocessed to improve image quality and annotation accuracy; the preprocessing operation specifically includes scale normalization, random flipping and cropping operations; then, an image annotation tool such as labelimg, RectLabel or OpenCV is used, and according to the guidance of a professional doctor, a bounding box is created around the polyp area by continuously adjusting the polygon size to obtain a labeled data set with white polyps and black backgrounds; then, the original data is divided according to a preset ratio, such as 8:2, to obtain a training set and a validation set containing image data and true labels.

[0046] Next, the training set is input into the constructed endoscopic polyp segmentation model for model training. Specifically, the images in the training set are first input into the feature extraction network composed of the PVT backbone network and the spatial channel self-attention module for feature extraction, and multi-scale features containing rich contextual information and global information are obtained; among them, the PVT backbone network is a network that combines the Transformer architecture and the pyramid structure. It gradually reduces the size of the feature map through multiple stages, thereby forming feature maps of different scales; specifically, at the beginning of each stage, the input image or the feature map obtained in the previous stage will be divided into a series of fixed-size image blocks, and then these image blocks will be flattened and mapped to a high-dimensional space through a linear transformation to generate a feature sequence, and then these feature sequences will be input into multiple Transformer encoders for further processing; finally, through multiple stages of processing, a feature pyramid containing multiple scale feature maps is generated. Next, in order to further improve the accuracy and effectiveness of feature extraction, a spatial channel self-attention module is added after each scale feature of the feature pyramid to extract more comprehensive and information-rich features, and obtain multi-scale features f1∈R C×H×W ,

[0047] The spatial channel self-attention module in this embodiment is composed of a spatial self-attention module and a channel self-attention module, and the outputs of the two modules are fused by the formula X'=conv3[conv2(SA(conv1(X)))+conv2(CA(conv1(X)))] to obtain; wherein X represents the input feature of the self-attention module, X' represents the output feature of the self-attention module, SA() represents the spatial self-attention module, CA() represents the channel self-attention module, conv1() represents a 3x3 convolution, conv2() represents a combined convolution block including a 1x1 convolution, a Dropout layer and a ReLU function, and conv3() represents a 1x1 convolution;

[0048] Among them, the spatial self-attention module is used to capture the contextual relationship between different spatial positions in the image and enhance the spatial dependency of the feature map. It is calculated by formula Where i, j represent different positions in the feature image, and i≠j, i∈S, j∈S, S represents the spatial position set of the feature map, represents the feature vector at the jth position in the feature map A, c represents the number of channels, h represents the height, w represents the width, B i represents the feature vector at the i-th position in the feature map B obtained after the convolution layer processing, C j represents the feature vector at the jth position in the feature map C obtained after processing by the conv1 convolution block, D irepresents the feature vector at the i-th position in the feature map D obtained after processing by the conv1 convolution block, and N = h × w, indicating the number of pixels, S ji Represents the spatial self-attention map, used to represent features The influence of the i-th position on the j-th position in , α represents the scale parameter, which is a learnable parameter with an initial value of 0 and represents the number of pixels, express The spatial self-attention output of .

[0049] Specifically, the input features Will Input three conv1 convolution blocks respectively for convolution operation, and get three sizes of Then adjust the shape of these three feature maps to get the size of Feature maps: B, C and D, then C j The transpose and B i Multiply and use a softmax layer to compute the spatial attention map S ji ∈R N×N ; and then Adjust the shape to S ji The transpose and D i Multiply, reshape the result into And multiply it by a learnable scale parameter α, and then add it to the feature Perform element-by-element summation to get the final output

[0050] The channel self-attention module focuses on the correlation between different channels in the feature map, improving the richness and accuracy of feature representation. It uses the formula Get the output features; where A p Represents the feature vector of the pth channel in the feature map A, A q represents the feature vector of the qth channel in the feature map A, and p≠q, p=1,2,…c, q=1,2,…c, X qp Represents the channel self-attention map, which is used to represent feature A q The influence of the p-th channel on the q-th channel in β represents the scale parameter, which is a learnable parameter with an initial value of 0. Indicates A q The channel self-attention output of .

[0051] Specifically, the original features Reshape into A p Multiply by A q Transpose Then use the softmax layer to calculate the channel attention map Then put X qp Transpose and then add A p Perform matrix multiplication and reshape their results into Finally, it is multiplied by a learnable scale parameter β and combined with feature A q Perform element-by-element summation to get the final output

[0052] During the entire feature extraction process, the PVT backbone network achieves a comprehensive understanding of the image from local to global through its unique pyramid structure and the combination of Transformer encoder layers; the introduction of the spatial channel self-attention module further enhances the network's ability to capture complex scenes and subtle features; this design enables the endoscopic polyp segmentation model of the present invention to show higher robustness and accuracy when processing challenging medical images.

[0053] Next, in order to effectively integrate multi-scale features, the multi-scale features obtained by the feature extraction network are input into the multi-scale cross-fusion module based on upsampling and Hadamard product. Further fusion processing is performed; where H represents the Hadamard product of all elements in the brackets, f n represents the above multi-scale features, f m ' represents the four fused feature maps, and n = {1, 2, 3, 4}, m = {1, 2, 3, 4}, up() represents the use of the learnable upsampling layer DySample to f n Perform upsampling operation, conv() represents f n Apply 3x3 convolution, donwn() represents f n Specifically, up and down sampling are first used to unify the feature maps of different scales into a target size, where the target size includes the size of the feature map of each scale, that is, there are 4 target sizes, and then the feature maps after up and down sampling are multiplied element by element using the Hadamard product to obtain the four fused feature maps.

[0054] Then, a dual decoder structure is used to decode the four fused feature maps. The output of the polyp segmentation model is obtained, namely the first prediction map P1 and the second prediction map P2; specifically, firstly, f m '(m={1,2,3,4}) obtains P1 by upsampling and summing to the size of f1', and then sums and outputs the feature map of f2' obtained by upsampling step by step to obtain P2;

[0055] Finally, the training of the polyp segmentation model also includes the loss calculation and iterative optimization of the model. Specifically, the corresponding true label is first passed to the decoder to obtain the first true label G1 and the second true label G2 with the same size as P1 and P2, respectively, and then the formula The final loss value is calculated, and then the loss value is passed back to the network through back propagation, the weight and bias parameters of the model are updated, and the loss value is continuously reduced in an iterative optimization manner, thereby gradually improving the segmentation accuracy and generalization ability of the model; represents the weighted intersection loss function, represents the weighted binary cross entropy loss function;

[0056] At the same time, the training process also includes using the real label to evaluate the prediction map to evaluate the accuracy of the current model training result. Specifically, the first prediction map and the second prediction map are firstly added through bilinear interpolation to obtain the final prediction map Y, and then compared with the real label The evaluation index calculation formula The model evaluation results are obtained; DSC represents the Diess coefficient, which calculates the overlap between the predicted segmentation and the true label. The higher the value, the more overlap between the predicted segmentation and the true label, that is, the better the segmentation effect; IoU represents the intersection over union ratio, which measures the overlap ratio between the predicted segmentation and the true label. The closer the value is to 1, the higher the overlap between the predicted segmentation and the true label, that is, the better the segmentation effect. Finally, save the model weight with the highest evaluation accuracy during the training process, which can be used to input the patient's gastrointestinal images and output high-precision polyp segmentation images.

[0057] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0058] Working principle of the present invention:

[0059] S1, data set preparation and preprocessing, by obtaining the original video of real endoscopy examination in the hospital, splitting the video into frame images, and using annotation tools to annotate the preprocessed frame images to obtain a data set with polyp images and real labels;

[0060] S2, feature extraction and self-attention mechanism, uses the PVT backbone network to extract the pyramid features of the input image, and then obtains multi-scale features containing rich contextual information and global information through the spatial channel self-attention module;

[0061] S3, multi-scale feature fusion, uses a multi-scale cross-fusion module based on up- and down-sampling and Hadamard product to cross-fuse multi-scale features and generate four fused feature maps;

[0062] S4, decoding and prediction, the four fused feature maps are input into the decoder, and the features are further fused through level-by-level upsampling and element-by-element addition to generate two polyp prediction maps of different sizes;

[0063] S5, loss calculation and optimization, input the true label into the decoder, generate two true labels with the same size as the two polyp prediction images, and calculate the total loss with the two prediction images of corresponding sizes using the weighted intersection joint loss function and the weighted binary cross entropy loss function. Finally, iterative optimization is performed according to the loss results to adjust the model parameters.

[0064] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A self-attention endoscopic polyp segmentation method based on PVT improvement, characterized in that: include S1, collecting a number of gastrointestinal images, preprocessing and labeling the gastrointestinal images to obtain a data set; wherein the data set includes image data and true labels; S2, after data enhancement, the image data in the data set is input into a feature extraction network based on PVT and spatial channel self-attention module for feature extraction to obtain multi-scale features; S3, constructing a multi-scale cross fusion module based on up and down sampling and Hadamard product, inputting the multi-scale features into the multi-scale cross fusion module, and obtaining four fused feature maps; S4, inputting the fused four feature maps into a decoder to obtain a first prediction map that is upsampled twice and added and a second prediction map that is upsampled three times and added; S5, inputting the true labels in the data set into the decoder to obtain the first true label and the second true label which are the same size as the first prediction map and the second prediction map respectively, performing loss calculation on the first true label and the second true label and the first prediction map and the second prediction map respectively using a loss function, and performing iterative optimization according to the loss calculation result; S6, restoring the first prediction image and the second prediction image to the input image size through bilinear interpolation and adding them together to obtain an endoscopic polyp segmentation model whose input is the patient's gastrointestinal tract image and output is the polyp prediction image.

2. According to the PVT-improved self-attention endoscopic polyp segmentation method of claim 1, it is characterized in that: Preprocessing and labeling the gastrointestinal images include: S1-1, obtaining the original video of the real endoscopy examination in the hospital and splitting it to obtain several frames of images; S1-2, performing image preprocessing on the plurality of frames of images, and using a labeling tool to label the polyp regions in the plurality of frames of images after preprocessing, to obtain a plurality of label images; S1-3, a plurality of label images are matched one by one with a plurality of preprocessed frame images to obtain a data set including a plurality of image data and a plurality of real labels.

3. The method for polyp segmentation in self-attention endoscopy based on PVT improvement according to claim 1, characterized in that: The feature extraction network includes an encoder and a spatial channel self-attention module; Wherein, the encoder is a backbone network of PVT; The spatial channel self-attention module is composed of a spatial self-attention module and a channel self-attention module, and the fused self-attention output is obtained by the formula X'=conv3[conv 2(SA(conv1(X)))+conv 2(CA(conv1(X)))]; wherein X represents the input feature of the self-attention module, X' represents the output feature of the self-attention module, SA() represents the spatial self-attention module, CA() represents the channel self-attention module, conv1() represents the conv1 convolution block including 3x3 convolution, conv2() represents the conv2 convolution block including 1x1 convolution, Dropout layer and ReLU function, and conv3() represents the conv3 convolution block including 1x1 convolution.

4. The method for segmenting polyps in an endoscope based on self-attention improved PVT according to claim 1, characterized in that: The calculation formula of the spatial self-attention module is: Where i, j represent different positions in the feature image, and i≠j, i∈S, j∈S, S represents the spatial position set of the feature map, represents the feature vector at the jth position in the feature map A, c represents the number of channels, h represents the height, w represents the width, B i represents the feature vector at the i-th position in the feature map B obtained after the convolution layer processing, C j represents the feature vector at the jth position in the feature map C obtained after processing by the conv1 convolution block, D i represents the feature vector at the i-th position in the feature map D obtained after processing by the conv1 convolution block, and N = h × w, indicating the number of pixels, S ji Represents the spatial self-attention map, used to represent features The influence of the i-th position on the j-th position in , α represents the scale parameter, which is a learnable parameter with an initial value of 0 and represents the number of pixels, express The spatial self-attention output of .

5. The method for segmenting polyps in an endoscope based on self-attention improved PVT according to claim 1, characterized in that: The calculation formula of the channel self-attention module is: Among them, A p Represents the feature vector of the pth channel in the feature map A, A q represents the feature vector of the qth channel in the feature map A, and p≠q, p=1,2,…c, q=1,2,…c, X qp Represents the channel self-attention map, which is used to represent feature A q The influence of the p-th channel on the q-th channel in β represents the scale parameter, which is a learnable parameter with an initial value of 0. Indicates A q The channel self-attention output of .

6. The method for self-attention endoscopic polyp segmentation based on PVT improvement according to claim 1, characterized in that: The calculation formula of the multi-scale cross fusion module is: Among them, H represents the Hadamard product of all elements in the brackets, f n represents the multi-scale features, f m ' represents the four fused feature maps, and f1∈R c×h×w , n={1,2,3,4},m={1,2,3,4},up() indicates using the learnable upsampling layer DySample to upsample f n Perform upsampling operation, conv() represents f n Apply 3x3 convolution, donwn() represents f n Perform downsampling operation.

7. The method for self-attention endoscopic polyp segmentation based on PVT improvement according to claim 1, characterized in that: The decoder includes upsampling operations and multi-level feature fusion, through the formula Get the output of the decoder; Among them, P1 represents the first prediction graph, P2 represents the second prediction graph, and f m ' represents the four fused feature maps, up() represents the upsampling operation, that is, the input features are upsampled to twice the size using the learnable upsampling layer DySample, and the calculation process of P1 and P2 represents the multi-level feature fusion, that is, the features are added and fused through the upsampling layers step by step.

8. The method for self-attention endoscopic polyp segmentation based on PVT improvement according to claim 1, characterized in that: The calculation formula of the loss function is: in, represents the weighted intersection loss function, represents the weighted binary cross entropy loss function, G1 represents the first true label, and G2 represents the second true label.

Images