Pyramid vision Transform-based polyp image segmentation method

Abstract

The invention relates to a polyp image segmentation method based on pyramid vision Transform, and belongs to the technical field of digital medical treatment. The method comprises the following steps: capturing global context information through a pyramid vision Transform module, and obtaining feature maps with different resolutions from a plurality of extraction stages; detail information is extracted from the shallow feature map of the first stage through a cyclic cross attention module, and shallow features are obtained; a multi-scale feature fusion sensing module is used for carrying out fusion processing on the feature maps extracted in multiple stages, deep features are extracted, and the deep features and shallow features are fused and then used for image segmentation; and designing weighted binary cross entropy loss and weighted IoU loss to train the above process, and segmenting the polyp image after training is completed. According to the invention, through the pyramid vision Transform module, the cyclic cross attention module and the multi-scale feature fusion perception module, the global information learning ability, the detail information capturing ability and the segmentation performance are significantly improved.

Description

Technical Field

[0001] The invention belongs to the technical field of digital medicine and relates to a polyp image segmentation method based on pyramid vision Transformer. Background Art

[0002] Colorectal cancer (CRC) is the third most common cancer in the world. In recent years, it has seriously endangered human life and health. According to the 2020 World Cancer Report released by the International Agency for Research on Cancer (IARC) of the World Health Organization, cancer has gradually become the second leading cause of death in the world after heart disease. Statistics show that colorectal cancer accounts for 10% of the diagnosed cancers, about 1.93 million people, and the mortality rate is 9.4%, ranking second, second only to lung cancer mortality. Colorectal polyps are abnormal tissues that grow in the colorectal mucosa and are precursors to colorectal cancer. Although polyps are initially benign, if they are not treated over time, they may develop into colorectal cancer. Polyps can be divided into: serrated adenomas, adenomatous polyps and hyperplastic polyps. Among them, adenomatous polyps are the direct cause of colorectal cancer. Clinically, it is shown that for every 1% increase in the detection rate of adenomas, the risk of CRC will decrease by 3%.

[0003] At present, colonoscopy is the most effective method for diagnosing colorectal diseases. A large number of studies have shown that early examination and effective and timely treatment of colorectal cancer patients can effectively reduce the mortality rate of patients. However, colonoscopy is highly dependent on the judgment of endoscopists. Nearly 25% of polyps may be ignored during colonoscopy due to the skills and subjectivity of clinicians. Accurate positioning of polyp contours is crucial for colonoscopy. Computer-assisted polyp segmentation systems can segment and mark the location of polyps in colonoscopy videos in real time, which can effectively reduce the probability of missed diagnosis and misdiagnosis. Therefore, using deep learning-based image processing methods to improve existing colonoscopy technology is undoubtedly one of the most effective methods.

[0004] For the segmentation of colon polyp images, researchers have proposed a large number of different image segmentation methods, which can be mainly divided into two types: colorectal polyp image segmentation algorithms based on manual features, such as texture, geometric features, color or gradient histograms. Ameling et al. proposed color wavelet covariance (CWC) for texture analysis; Bernal et al. used the watershed algorithm as a texture-based technique to segment potential polyp areas; Tajbakhsh et al. used the Canny edge detection algorithm to extract edge features to distinguish polyp areas; Mamonov et al. used a shape-based method to detect polyp areas. Since polyps have similar colors and textures to surrounding tissues and fuzzy boundaries increase the difficulty of segmentation, and models designed based on low-level features have poor generalization capabilities, these early solutions have a high false positive rate and risk of missed detection in polyp extraction, making them difficult to apply on a large scale.

[0005] Based on deep learning algorithms, due to the great success of convolutional neural network (CNN) in the field of image analysis, CNN is also used in medical image analysis. In the medical task of polyp segmentation, Akbari first proposed a polyp segmentation model using a fully convolutional network (FCN). FCN uses full convolution to replace the fully connected layer in CNN to achieve an end-to-end segmentation network. Although the segmentation accuracy has been greatly improved compared with traditional methods, it still cannot meet clinical applications. Olaf et al. proposed that the U-Net structure showed significant superiority in the field of medical image segmentation, so a large number of variant networks based on U-Net were generated. The UNet++ network proposed by Zhou et al. uses a densely connected skip connection method to integrate multi-scale functions, fuse features from different layers, and reduce the semantic differences between encoder and decoder feature maps; Debesh Jha et al. proposed the ResUet++ model based on ResUet, integrating ASPP, attention and residual structure systems into each layer of the network to improve the accuracy of network segmentation; ACSNet uses local context attention and global context modules to extract more context features of the target area, and learns to aggregate context information through adaptive selection modules. The performance of these networks based on U-Net has been greatly improved. The encoder-decoder structure of the U-shaped network has certain advantages in extracting multi-scale features, but after a series of convolutions and downsampling, it often leads to the loss of feature detail information, which is extremely unfavorable for the segmentation of small and irregular polyps. It comes from the role of attention in human perception, which can filter out unimportant information and retain key information. With the successful application of attention mechanism in natural language processing and natural image processing, attention is also introduced into medical image processing. Fan et al. designed a parallel reverse attention network (PraNet), a parallel decoder (PPD) aggregates high-level features to generate a guidance map to predict the rough area of ​​polyps, and uses a reverse attention module to mine the detailed information of polyp boundaries, but lacks bottom-up feature sharing. UACANet uses parallel axial attention in the encoder network for bottom-up feature sharing, which makes up for the shortcomings of PraNet.

[0006] All of the above methods are based on CNN. Due to the inherent limitations of convolution operations, CNN cannot model long-range dependencies. Compared with CNN, Transformer has long-range dependencies and uses self-attention mechanism to capture global context information, but the lack of spatial perception bias when modeling local information leads to limited positioning ability. In order to fully utilize the advantages of CNN and Transformer, a large number of segmentation methods innovatively combine Transformer and CNN.

[0007] Although many colonoscopic polyp segmentation methods have been developed, accurate segmentation of polyps is still a challenging task. There are two main reasons. On the one hand, most polyps have similar color and texture to normal tissues. At the same time, polyps have fuzzy boundaries, different sizes, irregular shapes and other characteristics, which increase the difficulty of segmentation. On the other hand, the network has a complex structure, which increases the complexity of the network. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a polyp image segmentation method based on pyramid vision transformer, which effectively improves the accuracy of network polyp segmentation while maintaining a simple structure. It is mainly composed of pyramid vision transformer (Pyramid Vision Transformer, PVT), recurrent cross attention module (RecurrentCriss-Cross Attention module, RCCA), and multi-scale feature fusion perception module (Multi-scale FeatureFusion Perception, MSFFP).

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A polyp image segmentation method based on pyramid vision Transformer, comprising the following steps:

[0011] S1, captures global context information through the pyramid vision Transformer module and obtains feature maps of different resolutions from multiple extraction stages;

[0012] S2, extracting detail information from the shallow feature map of the first stage through the cyclic cross attention module to obtain shallow features;

[0013] S3, fusing the feature maps extracted in multiple stages through a multi-scale feature fusion perception module to extract deep features, and fusing the deep features with shallow features for image segmentation;

[0014] S4. Design weighted binary cross entropy loss and weighted IoU loss to train the above process, and use them to segment polyp images after training.

[0015] Further, in step S1, the pyramid vision Transformer module includes multiple stages of feature encoders, each of which includes a Patch Embedding module and a Transformer Encoder module, and the feature pyramid is output as {F1, F2, F3, F4} by adjusting the stride;

[0016] Given an input image size Several feature maps are obtained through the feature backbone network Among them C i ∈{64,128,320,512}, i∈{1,2,3,4}.

[0017] Further, in step S2, the first-stage output feature F1 of the pyramid vision Transformer module is fed to the cyclic cross attention module to extract the shallow feature F containing detail information. A , the process is:

[0018] Given features After linear transformation and 1×1 convolution, we get the Q, K, and V matrices.

[0019] At each position u in the spatial dimension of Q, Q u , K u The inner product operation is used to obtain the relationship matrix of different elements; Softmax normalization is performed on it to obtain the attention weight map of each element

[0020] The weights are weighted and summed with the corresponding values ​​and added to the original input F u After adding, the output result is F u ′ , which is expressed as follows:

[0021]

[0022] where · represents the inner product, represents the dimension of each u position K, i∈{1,…,H+W-1}, u represents the position;

[0023] A cross-attention module is cascaded after the first-level cross-attention module. On the basis of extracting the feature F′ from the original module, the context information of elements that are not in the same row and column is further aggregated, and the complete context information is finally learned. The expression is as follows:

[0024] F″=C Att (C Att (F1)

[0025] F A =conv 1×1 (F″)

[0026] Among them C Att represents cyclic cross attention and conv represents convolution.

[0027] Further, in step S3, F4 is input into the multi-scale feature fusion perception module for processing. The multi-scale feature fusion perception module consists of four convolutional layers. The convolution kernels of different branches have different sizes, which are respectively composed of a standard convolution with a convolution kernel size of K∈{1,3,5,7} and a dilated convolution with a convolution kernel size of 3×3, and the dilation rate is R∈{1,3,5,7};

[0028] The receptive field of the feature map in each branch is expanded by dilated convolution operations, and K×K convolution is implemented in K×1 and 1×K modes. The outputs of the four branches are concatenated and sent to point convolution to adjust the number of channels of the model, and then added to the initial input and the final result is output through the Relu activation function.

[0029] Furthermore, the final result of the multi-scale feature fusion perception module is first adjusted through a 1×1 convolutional layer, F H Then, the feature F is obtained by upsampling eight times through bilinear interpolation. M , and finally the feature map F A With F M After concatenation, the images are passed through two 3×3 standard convolutions and then bilinear interpolation upsamples them four times to the original resolution to output the segmentation mask.

[0030] F M =Up ×8 (F H )

[0031] F O =concat(F M, F A )

[0032] out=Up ×4 (Conv(Conv(F O )))

[0033] Among them, Up stands for upsampling, concat stands for concatenation, and Conv stands for convolution;

[0034] Further, in step S5, the weighted binary cross entropy loss is used as part of the total loss, and the total loss function is expressed as:

[0035]

[0036] in, and They represent weighted binary cross entropy loss and weighted IoU loss respectively, and their expressions are:

[0037]

[0038] Among them, y i represents the true label, represents the predicted probability and N represents the total number of pixels in the image.

[0039] The beneficial effects of the present invention are:

[0040] The present invention captures global context information more effectively through the pyramid vision transformer module (PVT), and the global information learning ability is significantly enhanced compared with the traditional CNN. At the same time, the introduction of the multi-scale feature fusion perception module (MSFF) enables the model to fuse multi-scale features and make full use of high-level semantic information and low-level detail information, thereby improving segmentation accuracy.

[0041] In order to further enhance the ability to capture detail information, after the first stage of the feature encoder, the present invention uses a recurrent cross-attention module (RCCA) to accurately obtain detail information such as polyp boundaries, thereby achieving accurate segmentation of the subtle boundaries of polyps. In addition, the RCCA module also collects context information in the horizontal and vertical directions, models the full-image dependency of local features, and further optimizes the context information.

[0042] In terms of loss function design, the present invention adopts weighted binary cross entropy loss and weighted IoU loss to balance the learning of target and background areas and improve the model's sensitivity to polyp boundaries and segmentation accuracy.

[0043] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 It is a schematic diagram of the overall process of the polyp image segmentation method of the present invention;

[0046] Figure 2A schematic diagram of the flow of cross attention in the cyclic cross attention module of the present invention;

[0047] Figure 3 A schematic diagram of a flow chart of a cyclic cross attention in a cyclic cross attention module of the present invention;

[0048] Figure 4 It is a schematic diagram of the process of the multi-scale feature fusion perception module of the present invention;

[0049] Figure 5 It is a schematic diagram of the data set segmentation process under the embodiment;

[0050] Figure 6 This is an example of polyp image segmentation on different data sets according to the present invention. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0053] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0054] See also Figure 1 to Figure 6 , which is a polyp image segmentation method based on pyramid vision Transformer.

[0055] Example

[0056] This embodiment provides a specific implementation of a polyp image segmentation method based on a pyramid vision Transformer, which at least includes the following steps:

[0057] S1, captures global context information through the pyramid vision Transformer module and obtains feature maps of different resolutions from multiple extraction stages;

[0058] S2, extracting detail information from the shallow feature map of the first stage through the cyclic cross attention module to obtain shallow features;

[0059] S3, fusing the feature maps extracted in multiple stages through a multi-scale feature fusion perception module to extract deep features, and fusing the deep features with shallow features for image segmentation;

[0060] S4. Design weighted binary cross entropy loss and weighted IoU loss to train the above process, and use them to segment polyp images after training.

[0061] Compared with CNN, this feature encoder has stronger global information learning ability. Following the design concept of DeepLabv3+, the MSFF module is introduced in the last layer of PVT to strengthen the deep features learned from the PVT backbone network. Generally speaking, different levels of features in the encoder have different contextual information, and high-level features have more semantic information, which helps to locate objects; the multi-scale feature fusion perception module is used to strengthen the learning of semantic information in high-level features while fusing multi-scale features. There is rich detail information in low-level features, such as the size, shape, and boundary of polyps, which helps to capture the subtle boundaries of polyps. Therefore, the feature map output in the first stage of the feature encoder obtains detailed information such as the boundary of polyps through the recurrent cross attention module (RCCA).

[0062] In step S1 of this embodiment, the pyramid vision Transformer module includes multiple stages of feature encoding. By adjusting the stride to 4, 8, 16 and 32, the output feature pyramid is {F1, F2, F3, F4}. Given an input image size Four feature maps are obtained through the feature backbone network Among them C i ∈{64,128,320,512}, i∈{1,2,3,4}. That is, PVTv2 is used to capture global context information and obtain stronger multi-scale feature representation, and four feature maps of different resolutions are obtained, denoted as F1~F4. Specifically, it includes the following process:

[0063] First stage (i=1):

[0064] First, the given image is divided into multiple patches through the Patch Embedding module, and these patches are mapped to a high-dimensional space. This step not only reduces the spatial resolution, but also increases the feature dimension for subsequent processing.

[0065] Next, the high-dimensional features are transformed through the first Transformer Encoder module to capture the global dependencies in the image and obtain the first feature map F1.

[0066] Second stage (i=2):

[0067] The feature map F output by the first stage i Enter the second Patch Embedding module for further segmentation and mapping to a higher-dimensional feature space.

[0068] The second Transformer Encoder module continues to transform the features and generates a feature map F2 of the second scale.

[0069] The third stage (i=3):

[0070] Similarly, the feature map F2 output by the second stage is processed by the third Patch Embedding module and the Transformer Encoder module to generate a feature map F3 of the third scale.

[0071] Fourth stage (i=4):

[0072] Finally, the feature map F3 output by the third stage passes through the fourth Patch Embedding module and the Transformer Encoder module to generate the highest-scale feature map F4.

[0073] In each stage, the Patch Embedding module is responsible for dividing the image into patches of fixed size and increasing the channel dimension through linear projection. The Transformer Encoder module uses the self-attention mechanism to transform the features and capture the global context information in the image.

[0074] In step S2 of this embodiment, RCCA removes background noise from the shallow encoder feature map F1 to enhance the representation of detail information, which mainly includes texture, fine edges and tiny structures. It is fed into the cyclic cross attention module, and after a convolution layer with a convolution kernel size of 1, batch normalization and Relu activation function, the feature F with boundary information is obtained. A The cross-attention gathers contextual information from both horizontal and vertical directions, thus enabling modeling of the full-image dependency of local feature representations. Figure 2 As shown, given the feature After linear transformation and 1×1 convolution, we get the Q, K, and V matrices. At each position u in the spatial dimension of Q, Q u , K u The inner product operation is used to obtain the relationship matrix of different elements. In order to reduce the amount of calculation, only the context information of each element in the horizontal and vertical directions is aggregated. Finally, Softmax normalization is performed to obtain the attention weight map of each element. The weights are weighted and summed with the corresponding values ​​and added to the original input F u After adding, the output result is F′ u , which is expressed as follows:

[0075]

[0076] where · represents the inner product, Represents the dimension of each u position K, i∈{1,…,H+W-1}, and u represents the position.

[0077] Since polyps are similar in appearance to mucosa in color, in order to obtain key details such as the edge of the polyp, the feature tensor output by the first stage of PVTv2 is used to construct the semantic association between the polyp and the background area through a cyclic cross-attention module, where each pixel collects effective context information from semantically related pixels in the entire image. The cross-attention module only aggregates the context information in the vertical and horizontal directions of each position. For example, the blue block only aggregates the context information of elements in the same row and column. This feature is sparse and incomplete, which is extremely unfavorable for the segmentation of polyp boundary details. In order to obtain dense context information, such as Figure 3 As shown in , a recurrent cross attention module is further proposed, that is, a cross attention module is cascaded after the first-level cross attention module, and the context information of elements that are not in the same row and column are further aggregated on the basis of the feature F′ extracted by the original module, and finally the complete context information is learned. Figure 3 As shown, the feature F is outputted F″ after RCCA. The expression is as follows:

[0078] F″=CC Att (CC Att (F)

[0079]

[0080] CC Att represents cyclic cross attention and conv represents convolution.

[0081] In step S3 of this embodiment, F4 is input into the MSFFP module to capture richer semantic information, such as Figure 4 , the output of the deep feature F4 in the PVTv2 encoder is input into the multi-scale feature fusion perception module to extract multi-scale features and further enhance feature expression. This module consists of four convolutional layers. The convolution kernels of different branches have different sizes. They are composed of standard convolutions with convolution kernels of size K∈{1,3,5,7} and dilated convolutions with convolution kernels of size 3×3, and the dilation rate is R∈{1,3,5,7}. The dilated convolution operation can expand the receptive field of each branch, which is convenient for capturing the global context information of multiple receptive fields. The K×K convolution is implemented in K×1 and 1×K ways, which can greatly reduce the number of parameters. The outputs of the four branches are spliced ​​and sent to the point convolution to adjust the number of channels of the model, and then added to the initial input and the final result is output through the Relu activation function.

[0082] The final result of the MSFF module is first adjusted through a 1×1 convolution layer, and then upsampled eight times by bilinear interpolation to obtain the feature F M , and finally the feature map F A With F M After concatenation, the images are passed through two 3×3 standard convolutions and then bilinear interpolation upsamples them four times to the original resolution to output the segmentation mask.

[0083] In step S4 of this embodiment, since the ratio of pixels in the polyp and background regions is relatively large, the polyp image segmentation method of the present invention focuses on learning the background region. At the same time, the pixels near the boundary are very similar, which is easy to cause misclassification. In order to make the method treat the target and background regions equally, the weighted binary cross entropy loss is used as part of the total loss. Therefore, the total loss function can be written as:

[0084]

[0085] in, and Respectively represent weighted binary cross entropy loss and weighted IoU loss. Their expressions are as follows:

[0086]

[0087] Among them, y i represents the true label, represents the predicted probability and N represents the total number of pixels in the image.

[0088] This example implements the construction of the network model on the Pytorch framework and conducts experiments on the NVIDIA RTX 3090 24GB GPU. Before starting training, the resolution of all images is uniformly adjusted to 352×352. During training, we follow the multi-scale training strategy and scale the images to [0.75, 1, 1.25]. In addition, data enhancement methods such as image cropping and random flipping are used to reduce the probability of model overfitting. The AdamW optimizer is used for end-to-end optimization, and the initial learning rate is 1×10 -4 Weight decay is 1e -4 , batch_size is set to 8, and the model is trained for 100 epochs. During the test phase, the image resolution is also adjusted to 352×352.

[0089] This embodiment uses five public polyp datasets as shown in Table 1 for experiments. First, the datasets Kvasir-SEG and CVC-ClinicDB are randomly divided into training sets and test sets in a ratio of 9:1. Specifically, 900 images from Kvasir-SEG and 550 images from CVC-ClinicDB are used for model training, and the remaining datasets together constitute the test set. The dataset division diagram is shown in the figure below. Figure 5 .

[0090] Table 1

[0091]

[0092] In order to quantitatively analyze and evaluate the polyp image segmentation method of the present invention, the mean intersection over union (mIoU) and the mean Dice coefficient (mDice) are used as the main evaluation indicators of the segmentation model. In addition, since many methods regard polyp segmentation as a salient target detection task, four other indicators commonly used in salient target detection tasks are also used to evaluate the performance of the model, namely, weighted Dice coefficient Similarity measure (S α ), Enhanced alignment metrics Mean absolute error (MAE). Among the six evaluation indicators, except for the indicator MAE, the smaller the better, the larger the other five indicators are, the better. The calculation formulas for Dice and IoU are as follows:

[0093]

[0094]

[0095] Among them, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

[0096]

[0097] Where PR represents precision, RE represents recall, and β represents the weight coefficient.

[0098] Assuming the output size is h×w, then:

[0099]

[0100] Among them, Pred represents the prediction result, GT represents the true ground label, and N = h × w represents the number of pixels.

[0101] In this example, the polyp image segmentation method of the present invention is named PMNet, and compared with 6 advanced polyp segmentation models, namely UNet, ResNet++, PraNet, MSNet, HarDMSEG, and Polyp-PVT models. The final results of the Kvasir-SEG experiment are shown in Table 2:

[0102] Table 2

[0103]

[0104] As shown in Table 2, the mDice and mIOU of PMNet in the Kvasir-SEG dataset are 92% and 87.2%, respectively, which are 10.2% higher than the classic model UNet and 12.6% higher than the widely used network PraNet.

[0105] Table 3 shows the mDice and mIOU indicators of PMNet of the present invention and other existing solutions on different data sets:

[0106] Table 3

[0107]

[0108] By comparing the mDice and mIOU indicators of different data sets, it can be seen that the test results of the method of the present invention on the training set Kvasir-SEG are 0.920 and 0.872, and the test results on CVC-ClinicDB are 0.939 and 0.895, which have higher performance than the other six models. From the test results in Table 3, it can be seen that the PMNet of the present invention achieves the most advanced performance on all four data sets, indicating that PMNet has better learning ability and generalization ability.

[0109] The above results show that the present invention achieves accurate segmentation with a simple encoder-decoder architecture. The inference results of the model PPNet on five public datasets are shown in Figure 2. Figure 6The original data from the first row to the fifth row come from the datasets Kvasir-SEG, CVC-ColonDB, ETIS, CVC-ClinicDB, and CVC300-T, respectively. The output results of the PMNet model are shown in the ninth column. Compared with the output results of the other six methods, it can be intuitively seen that our model has better output boundaries and higher prediction accuracy.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A polyp image segmentation method based on pyramid vision Transformer, characterized by: It includes the following steps: S1, captures global context information through the pyramid vision Transformer module and obtains feature maps of different resolutions from multiple extraction stages; S2, extracting detail information from the shallow feature map of the first stage through the cyclic cross attention module to obtain shallow features; S3, processing the feature maps extracted in multiple stages through a multi-scale feature fusion perception module, extracting deep features, and fusing the deep features with shallow features for image segmentation; S4. Design weighted binary cross entropy loss and weighted IoU loss to train the above process, and use them to segment polyp images after training.

2. The polyp image segmentation method based on pyramid vision Transformer according to claim 1, characterized in that: In step S1, the pyramid vision Transformer module includes multiple stages of feature encoders, each of which includes a Patch Embedding module and a Transformer Encoder module, and the feature pyramid is output as {F1, 2, F3, F4} by adjusting the stride; Given an input image size Several feature maps are obtained through the feature backbone network Among them C i ∈{64,128,320,512}, i∈{1,2,3,4}.

3. The polyp image segmentation method based on pyramid vision Transformer according to claim 2, characterized in that: In step S2, the first-stage output feature F1 of the pyramid vision Transformer module is fed to the recurrent cross-attention module to extract the shallow feature F containing detail information. A , the process is: Given features After linear transformation and 1×1 convolution, we get the Q, K, and V matrices. At each position u in the spatial dimension of Q, Q u , K u The inner product operation is used to obtain the relationship matrix of different elements; Softmax normalization is performed on it to obtain the attention weight map of each element The weights are weighted and summed with the corresponding values ​​and added to the original input F u After adding, the output result is F ′ ′ , which is expressed as follows: where · represents the inner product, represents the dimension of each u position K, i∈{1,…,H+W-1}, u represents the position; A cross-attention module is cascaded after the first-level cross-attention module. On the basis of extracting the feature F′ from the original module, the context information of elements that are not in the same row and column is further aggregated, and the complete context information F is finally learned. ′′ , expressed as follows: F ′′ =C Att (C Att (F1)) F A =conv 1×1 (F″) Among them C Att represents cross attention and conv represents convolution.

4. The polyp image segmentation method based on pyramid vision Transformer according to claim 3 is characterized by: In step S3, F4 is input into the multi-scale feature fusion perception module for processing. The multi-scale feature fusion perception module consists of four convolutional layers. The convolution kernels of different branches have different sizes, which are respectively composed of a standard convolution with a convolution kernel size of K∈{1,3,5,7} and a dilated convolution with a convolution kernel size of 3×3, and the dilation rate is R∈{1,3,5,7}; The receptive field of each branch is expanded by dilated convolution operations, and K×K convolution is implemented in K×1 and 1×K modes. The outputs of the four branches are concatenated and sent to point convolution to adjust the number of channels of the model, and then added to the initial input and the final result is output through the Relu activation function.

5. The polyp image segmentation method based on pyramid vision Transformer according to claim 4, characterized in that: The final result of the multi-scale feature fusion perception module is first adjusted through a 1×1 convolution layer, and then upsampled eight times by bilinear interpolation to obtain the feature F M , and finally the feature map F A With F M After concatenation, the images are passed through two 3×3 standard convolutions and then bilinear interpolation upsamples them four times to the original resolution to output the segmentation mask.

6. The polyp image segmentation method based on pyramid vision Transformer according to claim 5, characterized in that: In step S5, the weighted binary cross entropy loss is used as part of the total loss, and the total loss function is expressed as: in, and They represent weighted binary cross entropy loss and weighted IoU loss respectively, and their expressions are: Among them, y i represents the true label, represents the predicted probability and N represents the total number of pixels in the image.

Images