Endoscopic polyp image segmentation method based on attention mechanism and style mixing technology

Abstract

The invention discloses an endoscopic polyp image segmentation method based on an attention mechanism and style mixing technology, which comprises the steps of constructing a basic architecture of a polyp image segmentation model based on an encoder-decoder architecture, using an MSCAN as an encoder, and carrying out image segmentation on the polyp image based on the MSCAN. And multi-scale features are fused and extracted, and a MixStyle technology is applied between stages of an encoder, so that the domain generalization performance of the model is improved. In a decoder, multi-scale features generated by a UAM aggregation encoder are used, and more important context information of the aggregated features is detected from channel representation and space representation in sequence; the OAM tracking target and complementary edge information are used to construct the relation between the characteristics of the encoder and the decoder, the distribution difference between the characteristics of the encoder and the decoder is reduced, and a more accurate segmentation image is obtained. In the training process, API loss is used for guiding training of the model in a self-adaptive mode, and the performance of the polyp image segmentation model is integrally improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an endoscopic polyp image segmentation method based on an attention mechanism and a style mixing technology. Background Art

[0002] Digestive tract cancer is showing a high incidence trend, and the number of new cases and deaths from colorectal cancer ranks among the highest in the world. Colorectal cancer is a common digestive tract cancer, and most colorectal cancers are caused by colorectal adenomatous polyps. Studies and experiments have shown that the survival rate of colorectal cancer patients is closely related to the stage of polyps. The later the stage of polyps is detected, the lower the survival rate. If colorectal cancer patients can be found, diagnosed, and treated earlier, the possibility of saving their lives is greater. Therefore, early screening, diagnosis, and treatment of colorectal cancer are of great significance.

[0003] At present, screening for colorectal polyps through intestinal endoscopy is an effective method and the "gold standard" for intestinal disease screening. Through endoscopy, doctors can visually observe the size and shape of polyps, as well as the pathological changes in intestinal tissue, and conduct clinical diagnosis and treatment of intestinal diseases. However, on the one hand, since traditional endoscopy completely relies on manual observation and judgment by doctors, it is easily interfered by subjective factors during the diagnosis process. At the same time, doctors will inevitably experience visual fatigue after long hours of work, leading to problems of missed detection and false detection. On the other hand, since the size and shape of polyps and lesions of each patient are different, there are often some tiny polyps with very small diameters, which are difficult for doctors to detect during clinical examinations. Therefore, it is particularly important to use computer-assisted technology to help doctors make diagnoses and reduce the problems of missed detection and false detection in endoscopy.

[0004] In recent years, deep learning has made great progress in the field of polyp image segmentation. However, deep learning models are often based on the assumption that training data and test data are identically distributed. When this assumption is violated, the effectiveness of the model will drop significantly. Research in this area is called domain generalization (DG). For polyp images, each patient has different intestinal tissues, different degrees of lesions, different types of polyps, and different morphologies, showing different styles. The training data cannot cover the statistical distribution of polyp images of all patients, and in actual application, the model often has to face polyp images of completely unfamiliar patients. Therefore, it is particularly important for clinical applications to invent a technology to optimize the polyp image segmentation algorithm from the perspective of domain generalization and improve the performance of polyp image segmentation. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, an endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is adopted to solve the problems raised in the above background technology.

[0006] An endoscopic polyp image segmentation method based on attention mechanism and style mixing technology includes the following steps:

[0007] Step S1, obtaining an endoscopic examination video, annotating the polyp part of the video at the pixel level, and obtaining a true segmentation label;

[0008] Step S2: constructing a basic architecture of a polyp image segmentation model based on an encoder-decoder architecture, wherein the encoder is used to extract multi-scale features of an input image, and the decoder is used to process the features obtained by the encoder to obtain a segmentation prediction map;

[0009] Step S3, constructing an encoder MSCAN using four stages with gradually decreasing resolutions, generating a total of four feature maps with different scales, wherein each stage contains multiple building blocks, and the basic component unit of the building block is the MSCA module;

[0010] Step S4: After the first three stages of the encoder, the style mixing technique MixStyle is added for implicit data enhancement to perturb the style information of the source domain training instances to regularize the model training;

[0011] Step S5: In the decoder, the UAM module is used to aggregate the multi-scale features generated by the encoder, and then the aggregated features are sequentially detected from the channel and spatial representations to obtain more context information;

[0012] Step S6: In the decoder, the OAM module is used to track the target and complementary edge information, to build the connection between the encoder and decoder features, to reduce the distribution difference between the features of the two, and to obtain an accurate segmentation map;

[0013] Step S7: In the OAM module, multiple segmentation prediction images of different sizes are generated and interpolated to the same size as the true label. Loss values ​​are calculated with them during training to introduce deep supervision.

[0014] Step S8: During the model training process, the three loss functions of BCE loss, IoU loss and L1 loss are combined, and the pixel intensity ω is defined to obtain API loss, and an adaptive method is used to guide the model training;

[0015] Step S9: Combine the networks of steps S2 to S7 and train them using the data of step S1. Calculate the training loss according to the method of step S8. After the training is completed, obtain the final model and export the model for deployment and reasoning.

[0016] As a further technical solution of the present invention: the specific steps in step S1 include:

[0017] The original video data without personal information is collected from real hospital endoscopic diagnosis scenes, and only contains the lesion name and time period, and is annotated under the guidance of specialist doctors;

[0018] Preprocessing is then performed, mainly including splitting the video frames, removing the endoscope manufacturer information, and unifying the size. Labelme is used to manually mark polyp lesions as positive samples, and automated tools generate negative samples without polyps.

[0019] Finally, the dataset is divided into training, validation, and test datasets in a ratio of 8:1:1.

[0020] As a further technical solution of the present invention: the specific steps of constructing the basic architecture of the polyp image segmentation model based on the encoder-decoder architecture in step S2 include:

[0021] The encoder is an encoder MSCAN, which is used to extract multi-scale features of an input image;

[0022] The encoder MSCAN outputs four layers of feature maps, denoted as E1, E2, E3 and E4, and the last three layers of feature maps will be input to the decoder to perform the segmentation task;

[0023] The decoder includes a UAM module and an OAM module. The UAM module is used to perform feature aggregation on the last three layers of feature maps output by the encoder, and the OAM module is used to interact the aggregated feature maps with the original feature maps to obtain the final segmentation prediction map.

[0024] As a further technical solution of the present invention: the specific steps in step S3 include:

[0025] The MSCA module includes three parts: a deep convolution for aggregating local information, a multi-branch deep banded convolution for capturing multi-scale context, and a 1×1 convolution for modeling the connection between different channels, and the output of the 1×1 convolution is used as an attention weight to weight the input;

[0026] The expression of the MSCA module is:

[0027]

[0028] Among them, E represents the input feature, represents a 1×1 convolution, α and X represent the attention map and output respectively, represents element-by-element matrix multiplication operation, DWConv represents depth convolution, Scale i,i∈{0,1,2,3} represents the i-th branch in MSCA, and Scale0 is a directly connected branch.

[0029] As a further technical solution of the present invention: the MSCA module uses two lightweight depth banded convolutions in each branch to approximate the standard depth convolution of the large kernel;

[0030] The encoder MSCAN has four different sizes, namely MSCAN-T, MSCAN-S, MSCAN-B and MSCAN-L. The difference between the four sizes is the number of blocks contained in each stage and the number of output channels.

[0031] As a further technical solution of the present invention: the specific steps in step S4 include:

[0032] Based on the style mixing technology MixStyle, a plug-and-play module is inserted between the bottom layers of CNN, and the feature statistics of two instances are mixed according to certain weights to simulate a new style.

[0033] As a further technical solution of the present invention: given an input batch x, a reference batch is first generated from x based on the style mixing technology MixStyle Because x is randomly sampled from all training data, where the data may come from multiple domains at random, Can be get;

[0034] generate After that, MixStyle calculates the mixed feature statistics according to the following formula:

[0035]

[0036] in, It is sampled from the Beta distribution, that is, λ~Beta(α,α), α∈(0,∞) is a hyperparameter;

[0037] Finally, the mixed feature statistics are applied to x to obtain:

[0038]

[0039] As a further technical solution of the present invention: the specific steps in steps S5 and S6 include:

[0040] Aggregate the outputs of the last three layers of the encoder, E2, E3, and E4, into 32, 64, and 128 channels, respectively, and then perform the following operations to obtain the aggregated representation:

[0041]

[0042] E″3=f(cat[E′3,f(Up(E4))])

[0043] E″2=f(Up(E″3))

[0044] X = f(cat[E′2,E″2])

[0045] Among them, Up(·) represents the upsampling operation, f(·) represents the convolution operation, and cat(·) represents the channel-by-channel feature connection;

[0046] Features after aggregation The size is the same as E2; after aggregation, the relatively important channel context information is first distinguished, and then the spatial information is enhanced according to the complementary confidence scores obtained from the channel context;

[0047] Using the softmax and sigmoid functions, we can obtain the relationship between channels through self-attention, aggregate the features of different channels, and obtain the attention score.

[0048]

[0049] Next, to refine the aggregate representation X, we use channel attention, apply confidence channel weights, and c The distribution of and confidence ratio γ preserves the confidence channel:

[0050]

[0051] Among them, F -1 (γ) represents α c Then, the input is refined by spatial attention calculation To distinguish salient objects and generate the first decoder representation The formula is:

[0052]

[0053] in, Use 1×1 convolution to project the input features into Finally, D0 is upsampled to DS0 to obtain deep supervision.

[0054] As a further technical solution of the present invention: the specific steps in step S7 include:

[0055] Reduce the distribution difference between the encoder and decoder feature representations while using fewer parameters to obtain a more accurate segmentation map, where D0 is the output of the UAM module;

[0056] OAM is represented from each decoder The target and complementary edge information are tracked in O The calculation method is:

[0057] α O =σ(D i )

[0058] Here, σ(·) represents the sigmoid function; however, α O It is not always possible to detect clear edge regions of the entire object, so a complementary edge weight α is generated E To cover undetected areas;

[0059] For each pixel x in D ij , invert the detected area, eliminate the background noise corresponding to the denoising ratio d, and perform missed area detection. The formula is:

[0060]

[0061] Combine the encoder output features E1 and E2 with the decoder features D1 and D2, and i+1 Perform upsampling operation to generate DS i+1 ;

[0062]

[0063] Combining the above parts, the final completed model is obtained.

[0064] As a further technical solution of the present invention: the specific steps in step S8 include:

[0065] The loss function used in the training process is API loss, which combines three loss functions: BCE loss, IoU loss, and L1loss, and uses an adaptive method to guide the training of the model;

[0066] API loss defines the pixel intensity ω and applies it to each pixel, giving pixels closer to the edge of the detected object a greater weight. The formula is:

[0067]

[0068] Among them, λ is the penalty term and K is the kernel size;

[0069] Using ω to improve BCE loss, we get adaptive BCE (aBCE) loss; where y c and are the true value and the predicted value respectively; by using ω, aBCE loss enables the network to pay more attention to the local structure of the edge;

[0070]

[0071] Using ω to improve IoU loss, we get adaptive IoU (aIoU) loss; aIoU loss can pay more attention to features related to dense areas and optimize the global structure;

[0072]

[0073] ω is used to improve L1 loss. L1 loss itself enables the network to learn robustly for noisy labels. The improved L1 loss can better distinguish relatively important pixels and exclude noisy pixels.

[0074]

[0075] By optimizing the local and global structures and combining the above improved losses, we can get the API loss, which is:

[0076]

[0077] Finally, the three deep supervisions (DS0, DS1, and DS2) output by the decoder and the set of three supervisions DS e And the true value (Ground Truth, GT) is sent to Calculate the loss value and sum it to get the final loss function of the model. The formula is:

[0078]

[0079] Compared with the prior art, the present invention has the following technical effects:

[0080] Adopting the above technical solution, through the endoscopic polyp image segmentation method based on the attention mechanism and style mixing technology, based on the encoder-decoder architecture, the basic architecture of the polyp image segmentation model is constructed, MSCAN is used as the encoder, multi-scale features are fused and extracted, and MixStyle technology is applied between the stages of the encoder to improve the domain generalization performance of the model. In the decoder, UAM is used to aggregate the multi-scale features generated by the encoder, and then the aggregated features are successively detected from the channel and spatial representations of the more important context information; OAM is used to track the target and complementary edge information, and the connection between the encoder and decoder features is constructed to reduce the distribution difference between the two features and obtain a more accurate segmentation map. During the training process, API loss is used to guide the training of the model in an adaptive manner, which improves the performance of the polyp image segmentation model as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0082] Figure 1 A schematic diagram of the steps of the endoscopic polyp image segmentation method according to the disclosed embodiment of the present application;

[0083] Figure 2 This is a visualization segmentation result diagram of the embodiment disclosed in this application. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, an endoscopic polyp image segmentation method based on an attention mechanism and a style mixing technique comprises the following steps:

[0086] Step S1, obtain the endoscopic examination video from the hospital, remove redundant information, perform pixel-level annotation on the polyp part, and obtain the real segmentation label. The original video data is collected from the real hospital endoscopic diagnosis scene, and does not contain any patient personal information for privacy and ethical issues. It only contains the name of the lesion and the time period of occurrence. The data is accompanied by a report with personal information removed. Specialists use this as a basis to guide the data annotation process. In the preprocessing and annotation stages, the video is split into frames, and the accompanying information of the endoscope manufacturer in the picture, such as timestamps, is removed, and the size is unified to 1160×1080. Use labelme to use the broken line to frame the edge of the polyp lesion, and generate a label with a white foreground (polyp) and a black background (intestine) as a positive sample. For frames without polyps, an automated tool is used to generate a completely black label as a negative sample. The positive and negative samples are named the same and divided into training, validation, and test data sets according to the ratio of 8:1:1.

[0087] Step S2, based on the encoder-decoder architecture, construct the basic architecture of the polyp image segmentation model, wherein the encoder is used to extract the multi-scale features of the input image, and the decoder is used to process the features obtained by the encoder to obtain a segmentation prediction map. The encoder is MSCAN, which is used to extract the multi-scale features of the input image. MSCAN outputs a total of four layers of feature maps (featuremap), represented as E1, E2, E3 and E4 respectively, and the last three layers of feature maps will be input to the decoder to perform the segmentation task. The decoder includes UAM and OAM, wherein UAM is used to perform feature aggregation on the last three layers of feature maps output by the encoder, and OAM interacts the aggregated feature maps with the original feature maps to obtain the final segmentation prediction map.

[0088] Step S3, use four stages with gradually decreasing resolution to build the encoder MSCAN, and generate four feature maps of different scales, where each stage contains multiple building blocks, and the basic unit of the building block is the MSCA module. The core idea of ​​MSCA is to use deep banded convolution to replace large kernel convolution and build spatial attention, while expanding the structure to multi-scale to better integrate multi-scale features. MSCA consists of three parts: a deep convolution (DWConv) for aggregating local information, a multi-branch deep banded convolution for capturing multi-scale context, and a 1×1 convolution (1×1convolution, 1×1Conv) for modeling the connection between different channels. The output of the 1×1 convolution is directly used as the attention weight to weight the input. The mathematical expression of MSCA is as follows:

[0089]

[0090] Where E represents the input feature, represents a 1×1 convolution, α and X represent the attention map and output respectively, represents element-by-element matrix multiplication operation, DWConv represents depth convolution, Scale i ,i∈{0,1,2,3} represents the i-th branch in MSCA, where Scale0 is the directly connected branch. In each branch of MSCA, two depthwise banded convolutions are used to approximate the standard depthwise convolution with a large kernel, and the kernel size of each branch is set to 7, 11, and 21, respectively. The reason for choosing depthwise banded convolution is that it is lightweight. For example, to simulate a standard two-dimensional convolution with a kernel size of 11×11, only a pair of 7×1 and 1×7 convolutions are required. Stacking a series of blocks together forms a convolutional encoder, namely MSCAN. MSCAN adopts a general hierarchical structure, which includes four stages with decreasing spatial resolution, with resolutions of and Here H and W represent the height and width of the input image, respectively. Each stage contains a downsampling module and a series of blocks, where the downsampling module contains a 3×3 convolution with a stride of 2 and a batch normalization (BN) layer. MSCAN has four different sizes, namely MSCAN-T, MSCAN-S, MSCAN-B and MSCAN-L. The difference lies in the number of blocks contained in each stage and the number of output channels. We use MSCAN-T as the backbone. The number of blocks contained in each stage is 3, 3, 5 and 2 respectively, and the number of output channels of each stage is 32, 64, 160 and 256 respectively.

[0091] Step S4: After the first three stages of the encoder, MixStyle is added for implicit data augmentation, aiming to perturb the style information of the source domain training instances to regularize the model training and improve the domain generalization performance of the model. MixStyle can be easily integrated into mini-batch training. Given an input batch x, MixStyle first generates a reference batch from x. Because x is randomly sampled from all training data, where the data may come from multiple domains at random, Can be Get. Generate After that, MixStyle calculates the mixed feature statistics as follows:

[0092]

[0093] in is sampled from the Beta distribution, that is, λ~Beta(α,α), α∈(0,∞) is a hyperparameter, which is set to 0.1. Finally, the mixed feature statistics are applied to x, and we get:

[0094]

[0095] During training, we use a 50% probability to decide whether to apply MixStyle in the forward pass. During testing, MixStyle is not applied.

[0096] Step S5: In the decoder, the multi-scale features generated by the encoder are aggregated using UAM, and the more important context information is detected from the channel and spatial representations of the aggregated features. UAM first aggregates the outputs of the last three layers of the encoder, E2, E3, and E4, into 32, 64, and 128 channels, respectively, and then performs the following operations to obtain the aggregated representation:

[0097]

[0098] E″3=f(cat[E′3,f(Up(E4))])#(8)

[0099] E″2=f(Up(E″3))#(9)

[0100] X=f(cat[E′2,E″2])#(10)

[0101] Among them, Up(·) represents the upsampling operation, f(·) represents the convolution operation, and cat(·) represents the channel-by-channel feature connection. The aggregated features The size is the same as E2. After aggregation, the relatively important channel context information is first distinguished, and then the spatial information is enhanced according to the complementary confidence scores obtained from the channel context. As shown in Formula 11, represents the features after channel pooling, Represents 1×1 convolution. Using softmax and sigmoid functions (denoted by σ(·)), we obtain the relationship between channels through self-attention and aggregate the features of different channels to obtain the attention score.

[0102]

[0103] Next, to refine the aggregate representation X, we use channel attention, apply confidence channel weights, and c The distribution of and confidence ratio γ preserves the confidence channel:

[0104]

[0105] Among them, F -1 (γ) represents α c Then, the refined input is calculated through spatial attention To distinguish salient objects and generate the first decoder representation as follows:

[0106]

[0107] in, Use 1×1 convolution to project the input features into Finally, D0 is upsampled to DS0 to obtain deep supervision.

[0108] Step S6: In the decoder, OAM is used to track the target and complementary edge information, build the connection between the encoder and decoder features, reduce the distribution difference between the two features, and obtain a more accurate segmentation map. In order to improve the efficiency of the decoder, D will be kept as a single channel. OAM is obtained from each decoder representation. The target and complementary edge information are tracked in O The calculation method is:

[0109] α O =σ(D i )#(15)

[0110] Here, σ(·) represents the sigmoid function. However, α O It is not always possible to detect clear edge regions of the entire object, so a complementary edge weight α is generated E to cover the undetected area. For each pixel x in D ij , invert the detected area, eliminate the background noise corresponding to the denoising ratio d, and perform missed area detection as follows:

[0111]

[0112] Furthermore, the encoder output features E1 and E2 are combined with the decoder features D1 and D2, and D i+1 Perform upsampling operation to generate DS i+1 .

[0113]

[0114] Step S7: In the OAM module, multiple segmentation prediction images of different sizes are generated and interpolated to the same size as the true label. Loss values ​​are calculated with them during training to introduce deep supervision.

[0115] Step S8: Train the constructed complete model with the prepared data, combine the three loss functions of BCE loss, IoU loss and L1 loss, and define the pixel intensity ω to form API loss, and use an adaptive method to guide the training of the model. API loss defines the pixel intensity ω and applies it to each pixel, so that the pixels close to the edge of the detected object get a larger weight, as follows:

[0116]

[0117] Where λ is the penalty term, K is the kernel size, and based on experience, λ is set to 0.5 and K is set to K∈{3,15,31}. Using ω to improve BCE loss, we get adaptive BCE (aBCE) loss, as shown in formula (3-16), where y c and are the true value and the predicted value respectively. By using ω, aBCE loss enables the network to pay more attention to the edge local structure.

[0118]

[0119] Using ω to improve IoU loss, we get adaptive IoU (aIoU) loss, as shown in formula (3-17). Compared with the original IoU loss, aIoU loss can pay more attention to the features related to dense areas and optimize the global structure.

[0120]

[0121] ω is used to improve L1 loss, as shown in formula (3-18). L1 loss itself enables the network to learn robustly for noisy labels, and the improved L1 loss can better distinguish relatively important pixels and exclude noisy pixels.

[0122]

[0123] In order to comprehensively consider the optimization of local and global structures, the three improved losses mentioned above are combined to obtain API loss:

[0124]

[0125] Finally, the three deep supervisions (DS0, DS1, and DS2) output by the decoder and the set of three supervisions DS e And the true value (Ground Truth, GT) is sent to Calculate the loss value and sum it to get the final loss function of the model, as follows:

[0126]

[0127] Step S9: After training, the test data can be used for reasoning, or it can be exported as an onnx model for deployment and reasoning in embedded devices. The specific process of deploying and reasoning is as follows: transfer the optimal model to the Jetson Xavier NX development board, configure the corresponding python environment in the development board, and use the python-based model conversion tool torch2trt to convert the optimal model into a tensorRT model with fp16 precision in the development board; use the QT framework based on C++ language to write the deployment and reasoning code of the tensorRT model in the development board to ensure that the input polyp image or video can obtain the polyp segmentation result normally after passing through the tensorRT model and display it in the graphical interface; obtain the actual scene data of the endoscope and input it into the deployed model to obtain the polyp segmentation result, determine the location of the polyp and assist the doctor in diagnosis.

[0128] In this embodiment, the public SUN-SEG dataset is used to test the easy / unseen and hard / unseen test sets using its official training set, and achieves excellent experimental results on multiple deep learning indicators for measuring image segmentation. The official project address of the dataset is as follows: https: / / github.com / GewelsJI / VPS. The test results of the present invention are shown in Tables 1 and 2:

[0129] Table 1

[0130]

[0131] Table 2

[0132]

[0133] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included in the scope of protection of the present invention.

Claims

1. An endoscopic polyp image segmentation method based on attention mechanism and style mixing technology, characterized in that: The following steps are involved: Step S1, obtaining an endoscopic examination video, annotating the polyp part of the video at the pixel level, and obtaining a true segmentation label; Step S2: constructing a basic architecture of a polyp image segmentation model based on an encoder-decoder architecture, wherein the encoder is used to extract multi-scale features of an input image, and the decoder is used to process the features obtained by the encoder to obtain a segmentation prediction map; Step S3, constructing an encoder MSCAN using four stages with gradually decreasing resolutions, generating a total of four feature maps with different scales, wherein each stage contains multiple building blocks, and the basic component unit of the building block is the MSCA module; Step S4: After the first three stages of the encoder, the style mixing technique MixStyle is added for implicit data enhancement to perturb the style information of the source domain training instances to regularize the model training; Step S5: In the decoder, the UAM module is used to aggregate the multi-scale features generated by the encoder, and then the aggregated features are sequentially detected from the channel and spatial representations to obtain more context information; Step S6: In the decoder, the OAM module is used to track the target and complementary edge information, to build the connection between the encoder and decoder features, to reduce the distribution difference between the features of the two, and to obtain an accurate segmentation map; Step S7: In the OAM module, multiple segmentation prediction images of different sizes are generated and interpolated to the same size as the true label. Loss values ​​are calculated with them during training to introduce deep supervision. Step S8: During the model training process, the three loss functions of BCE loss, IoU loss and L1loss are combined, and the pixel intensity ω is defined to obtain API loss, and an adaptive method is used to guide the model training; Step S9: Combine the networks of steps S2 to S7 and train them using the data of step S1. Calculate the training loss according to the method of step S8. After the training is completed, obtain the final model and export the model for deployment and reasoning.

2. According to claim 1, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: The specific steps in step S1 include: The original video data without personal information is collected from real hospital endoscopic diagnosis scenes, and only contains the lesion name and time period, and is annotated under the guidance of specialist doctors; Preprocessing is then performed, mainly including splitting the video frames, removing the endoscope manufacturer information, and unifying the size. Labelme is used to manually annotate polyp lesions as positive samples, and an automated tool generates negative samples without polyps. Finally, the dataset is divided into training, validation, and test datasets in a ratio of 8:1:

1.

3. According to claim 1, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: In step S2, the specific steps of constructing the basic architecture of the polyp image segmentation model based on the encoder-decoder architecture include: The encoder is an encoder MSCAN, which is used to extract multi-scale features of an input image; The encoder MSCAN outputs four layers of feature maps, denoted as E1, E2, E3 and E4, and the last three layers of feature maps will be input to the decoder to perform the segmentation task; The decoder includes a UAM module and an OAM module. The UAM module is used to perform feature aggregation on the last three layers of feature maps output by the encoder, and the OAM module is used to interact the aggregated feature maps with the original feature maps to obtain the final segmentation prediction map.

4. According to claim 1, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: The specific steps in step S3 include: The MSCA module includes three parts: a deep convolution for aggregating local information, a multi-branch deep banded convolution for capturing multi-scale contexts, and a 1×1 convolution for modeling the connection between different channels, and the output of the 1×1 convolution is used as an attention weight to weight the input; The expression of the MSCA module is: Among them, E represents the input feature, represents a 1×1 convolution, α and X represent the attention map and output respectively, represents element-by-element matrix multiplication operation, DWConv represents depth convolution, Scale i ,i∈{0,1,2,3} represents the i-th branch in MSCA, and Scale0 is a directly connected branch.

5. According to claim 4, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: The MSCA module uses two lightweight depth-banded convolutions in each branch to approximate the standard depth-banded convolution of a large kernel. The encoder MSCAN has four different sizes, namely MSCAN-T, MSCAN-S, MSCAN-B and MSCAN-L. The four sizes differ in the number of blocks contained in each stage and the number of output channels.

6. According to claim 1, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: The specific steps in step S4 include: Based on the style mixing technology MixStyle, a plug-and-play module is inserted between the bottom layers of CNN, and the feature statistics of two instances are mixed according to certain weights to simulate a new style.

7. According to claim 6, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: Given an input batch x, the style mixing technique MixStyle first generates a reference batch from x Because x is randomly sampled from all training data, where the data may come from multiple domains at random, Can be get; generate After that, MixStyle calculates the mixed feature statistics according to the following formula: in, It is sampled from the Beta distribution, that is, λ~Beta(α,α), α∈(0,∞) is a hyperparameter; Finally, the mixed feature statistics are applied to x to obtain: 。 8. According to claim 1, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: The specific steps in steps S5 and S6 include: Aggregate the outputs of the last three layers of the encoder, E2, E3, and E4, into 32, 64, and 128 channels, respectively, and then perform the following operations to obtain the aggregated representation: E″3=f(caT[E′3,f(Up(E4))] E″2=f(Up(E″3)) X = f(cat|E′2, E″2]) Among them, UP(·) represents upsampling operation, f(·) represents convolution operation, and cat(·) represents channel-by-channel feature connection; Features after aggregation The size is the same as E2; after aggregation, the relatively important channel context information is first distinguished, and then the spatial information is enhanced according to the complementary confidence scores obtained from the channel context; Using the softmax and sigmoid functions, we can obtain the relationship between channels through self-attention, aggregate the features of different channels, and obtain the attention score. Next, to refine the aggregate representation X, we use channel attention, apply confidence channel weights, and c The distribution of and confidence ratio γ preserves the confidence channel: Among them, F -1 (γ) represents α c Then, the input is refined by spatial attention calculation To distinguish salient objects and generate the first decoder representation The formula is: in, Use 1×1 convolution to project the input features into Finally, D0 is upsampled to DS0 to obtain deep supervision.

9. According to claim 1, the endoscopic polyp image segmentation method based on attention mechanism and style mixing technology is characterized in that: The specific steps in step S7 include: Reduce the distribution difference between the encoder and decoder feature representations while using fewer parameters to obtain a more accurate segmentation map, where D0 is the output of the UAM module; OAM is represented from each decoder The target and complementary edge information are tracked in O The calculation method is: a O =σ(D i ) Here, σ(·) represents the sigmoid function; however, α O It is not always possible to detect clear edge regions of the entire object, so a complementary edge weight α is generated E To cover undetected areas; For each pixel x in D ij , invert the detected area, eliminate the background noise corresponding to the denoising ratio d, and perform missed area detection. The formula is: Combine the encoder output features E1 and E2 with the decoder features D1 and D2, and i+1 Perform upsampling operation to generate DS i+1 ; Combining the above parts, the final completed model is obtained.

10. The endoscopic polyp image segmentation method based on attention mechanism and style mixing technology according to claim 1, characterized in that: The specific steps in step S8 include: The loss function used in the training process is API loss, which combines three loss functions: BCE loss, IoU loss, and L1loss, and uses an adaptive method to guide the training of the model; API loss defines the pixel intensity ω and applies it to each pixel, giving pixels closer to the edge of the detected object a greater weight. The formula is: Among them, λ is the penalty term and K is the kernel size; Using ω to improve BCE loss, we get adaptive BCE (aBCE) loss; where y c and are the true value and the predicted value respectively; by using ω, aBCE loss enables the network to pay more attention to the local structure of the edge; Using ω to improve IoU loss, we get adaptive IoU (aIoU) loss; aIoU loss can pay more attention to features related to dense areas and optimize the global structure; ω is used to improve L1loss. L1 loss itself enables the network to learn robustly for noisy labels. The improved L1 loss can better distinguish relatively important pixels and exclude noisy pixels. By optimizing the local and global structures and combining the above improved losses, we can get the API loss, which is: Finally, the three deep supervisions (DS0, DS1, and DS2) output by the decoder and the set of three supervisions DS e With the real value input Calculate the loss value and sum it to get the final loss function of the model. The formula is: 。

Images