A method and system for polyp image segmentation

Abstract

The present invention relates to the field of image recognition technology, and in particular to a polyp image segmentation method and system. The polyp image segmentation model is constructed including: a pre-trained network, a cascade fusion partial decoder, a plurality of multi-scale mobile convolution modules and an axial layered multi-scale cross-reverse attention module. The image to be processed is extracted with high-level features of different resolutions through the pre-trained network, and a global feature map is generated through the cascade fusion partial decoder; each high-level feature is input into the corresponding multi-scale mobile convolution module, and the features of different scales of each high-level feature are respectively extracted to obtain the target fusion feature corresponding to each high-level feature; the global feature map and the target fusion feature corresponding to each high-level feature are input into a plurality of axial layered multi-scale cross-reverse attention modules, and the output of the last axial layered multi-scale cross-reverse attention module is output through a Sigmoid activation function to output a polyp image segmentation prediction map. The present invention improves the segmentation accuracy of polyp images.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a polyp image segmentation method and system. Background Art

[0002] Among digestive tract malignancies, colorectal cancer poses a major threat to global health with its high morbidity and mortality. Fortunately, studies have shown that most colorectal cancers originate from benign colon polyps, and the process from polyps to cancer can take up to several years. Therefore, early identification begins with high-quality endoscopic images. Endoscopy physicians use high-resolution endoscopic cameras to thoroughly examine the inner wall of the colorectum. Image clarity and correct lighting are essential for accurate identification of polyps. Once the image is obtained, the physician will make a preliminary classification based on the visual characteristics of the polyp, which include size, shape (such as pedicle, flat or convex), surface texture and color. In order to improve the accuracy of identification, physicians will use staining or narrow-band imaging techniques, which can enhance the visual contrast of polyps, making early, small or subtle color changes easier to detect. In some advanced systems, endoscopists can use manual or semi-automatic tools to annotate abnormal areas in the image. These tools help physicians focus their attention and conduct further evaluation by highlighting suspicious areas. With the continuous advancement of medical technology, especially the development of computer-aided diagnosis (CAD) technology, medical image analysis plays an increasingly important role in the early detection and treatment of digestive tract diseases. Automatic segmentation of colon polyps is a key task in the early diagnosis of colorectal cancer, but this task is challenging due to the diversity and hidden nature of polyps. Different types of polyps may differ significantly in morphology and color, and some smaller or flat polyps are difficult to detect in endoscopic images. In addition, the boundary between polyps and the surrounding mucosa is blurred, which increases the difficulty of automatic segmentation. To overcome these challenges, researchers have been exploring more advanced image processing algorithms to improve the accuracy and reliability of polyp segmentation, thereby providing important information for clinical prevention and early treatment.

[0003] Traditional polyp segmentation methods based on manual features include several key steps, starting with preprocessing of the acquired endoscopic images, such as noise removal and contrast enhancement, to improve image clarity. Subsequently, key manual features such as color, texture, edge, and shape are extracted from the image, which help distinguish polyps from normal tissues. For example, polyps usually have obvious differences in color and texture from surrounding tissues. Image segmentation algorithms such as threshold segmentation, region growing, or snake model are used to accurately identify and isolate polyp regions. Finally, post-processing steps such as noise removal, hole filling, and edge smoothing are performed to optimize the segmentation results. Although this method can effectively identify polyps to a certain extent, it usually requires manual adjustment of parameters, has a high dependence on image quality, and is inefficient when processing large or complex data. In recent years, the rise of deep learning technology has brought revolutionary progress to the field of polyp segmentation, especially convolutional neural networks (CNNs), which can automatically learn complex feature representations from data, greatly simplify the process of feature engineering, and show great potential in automatic feature extraction and complex pattern recognition. The UNet network has made several key improvements on the basis of the convolutional neural network (CNN). It forms a U-shaped structure with a contraction path (encoder) and a symmetrical expansion path (decoder). The contraction path captures the image context information, and the expansion path performs accurate pixel-level positioning, which enhances the sensitivity to small targets. The UNet++ network consists of multiple sub-U-Nets with jump connections. Dense connections are introduced between the encoder and the decoder to capture features of different scales. These dense connections can better integrate information at different levels, thereby improving segmentation accuracy, making polyp segmentation significantly improved in accuracy and efficiency. The parallel reverse attention network (PraNet) significantly improves the recognition and segmentation capabilities of target areas in complex medical images by introducing the reverse attention mechanism. In the parallel reverse attention network (PraNet), the core idea is to use the reverse attention module of parallel processing to capture the key features of polyps. This method can effectively highlight the polyp area and suppress background interference, thereby improving the accuracy and efficiency of segmentation. Based on the parallel reverse attention network (PraNet), the contextual axial preserving attention network (CaraNet) performs multi-scale feature extraction on different dimensions through pyramid pooling, and introduces the contextual axial reverse attention mechanism to more effectively capture and parse the contextual information of the image. The contextual axial preserving attention network (CaraNet) adopts the strategy of borrowing from the UNet architecture. By combining axial attention and convolutional layers, it not only improves the feature extraction capability, but also effectively restores the original resolution of the image through the step-by-step upsampling process. By effectively integrating the attention mechanism in Transformer technology with the advantages of traditional convolutional networks, the contextual axial preserving attention network (CaraNet) provides a new and powerful tool for high-precision medical image segmentation tasks.

[0004] Although the context-preserving attention network (CaraNet) can effectively identify and segment small medical objects, it still has limitations in capturing polyps with very small or irregular boundaries. This limitation mainly stems from the model's inability to fully distinguish adjacent structures when dealing with the boundaries of objects in highly uneven or complex backgrounds. In addition, the context-preserving attention network (CaraNet) performs unstable on new images outside the training set, especially when encountering images with significantly different distributions from the training data. The polyp image segmentation accuracy of the context-preserving attention network (CaraNet) is greatly reduced. This generalization problem seriously affects the application effect of the model in actual clinical environments. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the defects of the prior art that, when processing the boundaries of objects in highly uneven or complex backgrounds, the prior art cannot completely distinguish adjacent structures, resulting in limitations in capturing polyps with very small or irregular boundaries, and the prior art is unstable on new images outside the training set, resulting in poor polyp image segmentation accuracy.

[0006] In order to solve the above technical problems, the present invention provides a polyp image segmentation method, comprising the following steps:

[0007] Constructing a polyp image segmentation model, the polyp image segmentation model comprising: a pre-trained network, a cascade fusion partial decoder, a plurality of multi-scale mobile convolution modules and an axial layered multi-scale cross inverse attention module;

[0008] Input the image to be processed into the pre-trained network to extract high-level features of different resolutions;

[0009] High-level features of different resolutions are aggregated through cascaded fusion partial decoders to obtain a global feature map;

[0010] Each high-level feature is input into the corresponding multi-scale mobile convolution module to extract the features of each high-level feature at different scales, and obtain the target fusion features corresponding to each high-level feature, including:

[0011] After the high-level features are processed by batch normalization and PReLU activation function, they are reduced in dimension through the first 1x1 convolutional layer to obtain the high-level features after dimensionality reduction.

[0012] The high-level features after dimensionality reduction are input into multiple parallel moving inverted bottleneck convolution branches to extract features of different scales, and the output features of each branch are obtained;

[0013] Concatenate the output features of all branches along the channel dimension to obtain a comprehensive feature map;

[0014] After the comprehensive feature map is processed by batch normalization and PReLU activation function in sequence, it is passed through the second 1x1 convolution layer for dimensionality increase to obtain the comprehensive feature map after dimensionality increase;

[0015] Perform residual connection between high-level features and their corresponding dimensional integrated feature maps to obtain target fusion features corresponding to high-level features;

[0016] The target fusion features corresponding to the global feature map and each high-level feature are input into multiple axial hierarchical multi-scale cross-inverse attention modules. The polyp area is identified by erasing the foreground. The output of the last axial hierarchical multi-scale cross-inverse attention module is passed through the Sigmoid activation function to obtain the polyp image segmentation prediction map.

[0017] Preferably, each moving inverted bottleneck convolution branch includes multiple moving inverted bottleneck convolution modules connected in series along the forward propagation direction, and the output of the last moving inverted bottleneck convolution module of each branch is spliced ​​with the outputs of other moving inverted bottleneck convolution modules in the branch along the channel dimension to obtain the output features of each branch.

[0018] Preferably, each moving inverted bottleneck convolution module includes: an extended convolution layer, a first two-dimensional batch normalization layer, a first ReLU6 activation function layer, a depth convolution layer, a second two-dimensional batch normalization layer, a second ReLU6 activation function layer, a squeeze and excitation block, a projection convolution layer, and a third two-dimensional batch normalization layer connected in sequence along the forward propagation direction.

[0019] Preferably, the image to be processed is input into a pre-trained network to extract features of five levels of different resolutions of the image to be processed;

[0020] The last three high-level features are respectively passed through the corresponding multi-scale mobile convolution modules to obtain the first target fusion feature, the second target fusion feature, and the third target fusion feature.

[0021] Preferably, the target fusion feature corresponding to the global feature map and each high-level feature is input into multiple axial hierarchical multi-scale cross-inverse attention modules, the polyp area is identified by erasing the foreground, and the output of the last axial hierarchical multi-scale cross-inverse attention module is passed through a Sigmoid activation function to obtain a polyp image segmentation prediction map, including:

[0022] After downsampling, the global feature map is input into the third axial hierarchical multi-scale cross inverse attention module with the third target fusion feature, and the output feature of the third axial hierarchical multi-scale cross inverse attention module is extracted;

[0023] The output features of the third axial hierarchical multi-scale cross-inverse attention module are added to the downsampled global feature map to obtain the third feature map;

[0024] The third feature map and the second target fusion feature are input into the second axial hierarchical multi-scale cross inverse attention module, and the output feature of the second axial hierarchical multi-scale cross inverse attention module is extracted;

[0025] After upsampling, the third feature map is added to the output feature of the second axial hierarchical multi-scale cross inverse attention module to obtain the second feature map;

[0026] The second feature map and the first target fusion feature are input into the first axial hierarchical multi-scale cross inverse attention module, and the output feature of the first axial hierarchical multi-scale cross inverse attention module is extracted;

[0027] After upsampling, the second feature map is added to the output feature of the first axial hierarchical multi-scale cross inverse attention module to obtain the first feature map;

[0028] The first feature map is activated by the Sigmoid function to generate a polyp image segmentation prediction map.

[0029] Preferably, each axial hierarchical multi-scale cross-inverse attention module comprises: a self-attention module, an axial patch-aware attention module and a lesion-aware cross-module.

[0030] Preferably, the self-attention module comprises: a multi-scale residual pyramid module and a first multi-head self-attention module;

[0031] The input features of the self-attention module are passed through the multi-scale residual pyramid module to capture and fuse features of different scales to obtain the output features of the multi-scale residual pyramid module;

[0032] Input the output features of the multi-scale residual pyramid module and the input features of the self-attention module into the first multi-head self-attention module, adjust the attention weights, and obtain the output features of the first multi-head self-attention module;

[0033] The dimension of the input feature of the self-attention module is adjusted to the dimension of the output feature of the first multi-head self-attention module, and after adding it to the output feature of the first multi-head self-attention module, a comprehensive attention feature is obtained;

[0034] The comprehensive attention features are added to the comprehensive attention features after feature fusion to obtain the output features of the self-attention module;

[0035] The axial patch-aware attention module includes: three parallel branches and an attention module. The three parallel branches include: local branch, serial convolution branch and global branch;

[0036] The input features of the axial patch-aware attention module are input into the local branch, serial convolution branch, and global branch to extract features, and the output features of each branch are obtained. After adding the output features of all branches, they are input into the attention module for adaptive feature enhancement to obtain the output features of the axial patch-aware attention module;

[0037] The lesion-aware cross-module includes: a second multi-head self-attention module;

[0038] After enhancing the features of the lesion-aware cross-module input by element-wise multiplication, the context information is extracted by the second multi-head self-attention module to obtain the output features of the second multi-head self-attention module;

[0039] The input features of the lesion perception cross module are dimensionally adjusted and added to the output features of the second multi-head self-attention module to obtain the module comprehensive features;

[0040] The module comprehensive features are added to the module comprehensive features after feature fusion and the dimensions are adjusted to obtain the lesion-aware cross-module output features.

[0041] Preferably, the downsampling of the global feature map and the input of the third target fusion feature into the third axial hierarchical multi-scale cross inverse attention module to extract the output features of the third axial hierarchical multi-scale cross inverse attention module comprises the following steps:

[0042] The third target fusion feature is passed through the self-attention module to extract and fuse features of different scales to obtain the output feature of the self-attention module;

[0043] The output features of the self-attention module are passed through the axial patch-aware attention module to extract local area information and obtain the output features of the axial patch-aware attention module;

[0044] The downsampled global feature map is passed through the Sigmoid activation function, fused with the output features of the self-attention module, and then input into the lesion-aware cross-attention module. The boundary features are enhanced through the lesion-aware attention to obtain the output features of the lesion-aware cross-attention module.

[0045] After expanding the number of channels of the output features of the lesion-aware cross-attention module, they are multiplied with the output features of the axial patch-aware attention module to obtain the output features of the third axial hierarchical multi-scale cross-inverse attention module.

[0046] Preferably, the pre-trained network is a Res2Net network.

[0047] The present invention also provides a polyp image segmentation system, comprising:

[0048] A model building module, used to build a polyp image segmentation model, the polyp image segmentation model comprising: a pre-trained network, a cascade fusion partial decoder, a plurality of multi-scale mobile convolution modules and an axial layered multi-scale cross inverse attention module;

[0049] The high-level feature extraction module is used to input the image to be processed into the pre-trained network and extract high-level features of different resolutions;

[0050] The global feature map extraction module is used to aggregate high-level features of different resolutions through cascade fusion partial decoders to obtain a global feature map;

[0051] The target fusion feature extraction module is used to input each high-level feature into the corresponding multi-scale mobile convolution module, extract the features of each high-level feature at different scales, and obtain the target fusion feature corresponding to each high-level feature, including:

[0052] After the high-level features are processed by batch normalization and PReLU activation function, they are reduced in dimension through the first 1x1 convolutional layer to obtain the high-level features after dimensionality reduction.

[0053] The high-level features after dimensionality reduction are input into multiple parallel moving inverted bottleneck convolution branches to extract features of different scales, and the output features of each branch are obtained;

[0054] Concatenate the output features of all branches along the channel dimension to obtain a comprehensive feature map;

[0055] After the comprehensive feature map is processed by batch normalization and PReLU activation function in sequence, it is passed through the second 1x1 convolution layer for dimensionality increase to obtain the comprehensive feature map after dimensionality increase;

[0056] Perform residual connection between high-level features and their corresponding dimensional integrated feature maps to obtain target fusion features corresponding to high-level features;

[0057] The prediction module is used to input the target fusion features corresponding to the global feature map and each high-level feature into multiple axial hierarchical multi-scale cross-inverse attention modules, identify the polyp area by erasing the foreground, and the output of the last axial hierarchical multi-scale cross-inverse attention module is passed through the Sigmoid activation function to obtain the polyp image segmentation prediction map.

[0058] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0059] The polyp image segmentation method and system described in the present invention inputs each high-level feature into a corresponding multi-scale moving convolution module, extracts features of different scales of each high-level feature respectively, and obtains target fusion features corresponding to each high-level feature, including: subjecting the high-level features to batch normalization and PReLU activation function processing in sequence, and then performing dimensionality reduction processing through a first 1x1 convolution layer to obtain high-level features after dimensionality reduction; respectively inputting the high-level features after dimensionality reduction into a plurality of parallel moving inverted bottleneck convolution branches to extract features of different scales, and obtain output features of each branch; splicing the output features of all branches along the channel dimension to obtain a comprehensive feature map; subjecting the comprehensive feature map to batch normalization and PReLU activation function processing in sequence, and then performing dimensionality increase processing through a second 1x1 convolution layer to obtain a comprehensive feature map after dimensional increase; performing residual connection between the high-level features and the corresponding comprehensive feature map after dimensional increase to obtain target fusion features corresponding to the high-level features; the multi-scale moving convolution module uses the first 1x1 convolution layer and the second 1x1 convolution layer to respectively perform dimensionality increase processing on the high-level features and the corresponding comprehensive feature map after dimensional increase to obtain target fusion features corresponding to the high-level features. The convolution layer performs dimensionality reduction and dimensionality increase, reducing the computational complexity of the polyp image segmentation model while maintaining the integrity of the polyp image features; extracting features of different scales through multiple parallel moving inverted bottleneck convolution branches helps to capture the multi-scale information of polyps in the image, improve the accuracy of polyp segmentation, and enhance the stability of the polyp image segmentation model.

[0060] In addition, the global feature map and the target fusion features corresponding to each high-level feature are input into multiple axial hierarchical multi-scale cross-inverse attention modules, and the polyp area is identified by erasing the foreground. The output of the last axial hierarchical multi-scale cross-inverse attention module is passed through the Sigmoid activation function to generate a polyp image segmentation prediction map. Each axial hierarchical multi-scale cross-inverse attention module includes: a self-attention module, an axial patch-aware attention module and a lesion-aware cross-attention module; the self-attention module in the axial hierarchical multi-scale cross-inverse attention module provides global context information, which helps to determine the approximate location of the polyp; the lesion-aware cross-attention module further refines these areas and enhances the boundary features through lesion-aware attention; the axial patch-aware attention module accurately captures and refines the edges of the polyps through parallel processing of local features. Through the cooperation of the three modules, the axial hierarchical multi-scale cross-inverse attention module can understand the image content more comprehensively and process information in detail from the global to the local, thereby more accurately refining the representation of the polyp boundary, significantly enhancing the accuracy of the polyp image segmentation prediction map. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0062] Figure 1It is a structural diagram of a polyp image segmentation method and system of the present invention.

[0063] Figure 2 It is a structural diagram of the decoder of the cascade fusion part.

[0064] Figure 3 This is the structural diagram of the mobile inverted bottleneck convolution module in the multi-scale mobile convolution module.

[0065] Figure 4 It is a schematic diagram of the multi-scale mobile convolution module. Figure 4 (a) in the figure is the structural diagram of the multi-scale mobile convolution module. Figure 4 (b) is a schematic diagram of three parallel moving inverted bottleneck convolution branches extracting features of different scales.

[0066] Figure 5 It is a structural diagram of the axial layered multi-scale cross-inverse attention module.

[0067] Figure 6 It is the structure diagram of the self-attention module.

[0068] Figure 7 This is the structural diagram of the lesion-aware cross-attention module.

[0069] Figure 8 It is the structure diagram of the axial patch-aware attention module.

[0070] Fig. 9 It is a schematic diagram of the polyp segmentation results of each model in different data sets.

[0071] Fig.10 This is a schematic diagram of the average Dice coefficient of each model on the CVC-ColonDB, ETIS, and CVC-300 datasets.

[0072] Fig.11 This is a schematic diagram of the mean intersection and union of various models on the CVC-ColonDB, ETIS, and CVC-300 datasets.

[0073] Fig.12 This is a schematic diagram of the average Dice coefficient of the polyp image segmentation model on five data sets.

[0074] Fig.13 Schematic diagram of the loss function of the polyp image segmentation model when the learning rate is 1e-4.

[0075] Fig.14 It is a flow chart of the steps of a polyp image segmentation method.

[0076] Fig.15 It is a system structure diagram of a polyp image segmentation system. DETAILED DESCRIPTION

[0077] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0078] Reference Figure 1 , 14 As shown, the first embodiment of the present invention provides a polyp image segmentation method, comprising the following steps:

[0079] Step S1: constructing a polyp image segmentation model (MdfraNet), wherein the polyp image segmentation model comprises: a pre-trained network, a cascade fusion partial decoder, a plurality of multi-scale mobile convolution modules and an axial layered multi-scale cross inverse attention module;

[0080] In this embodiment, preferably, the pre-trained network is a Res2Net network. The Res2Net network achieves efficient feature extraction and representation by constructing multi-scale hierarchical residual connections in a single residual block, and can accurately identify and segment small targets, which is particularly useful for distinguishing subtle anatomical structures in medical images. The flexibility of the Res2Net network allows the network structure to be adjusted according to the segmentation requirements of specific polyps and optimize network performance.

[0081] In addition, the depth and complexity of the Res2Net network help the model generalize better on limited data. In view of these advantages, the pre-trained Res2Net network was selected as the backbone network of the polyp image segmentation model. By using the Res2Net network to extract the features of the image to be processed, the polyp image segmentation model can inherit the powerful feature extraction capability of the Res2Net network and perform fast and effective segmentation on small medical target tasks. Using the Res2Net network pre-trained on the ImageNet dataset, good segmentation performance can be obtained on polyp images through fine-tuning even when there is less labeled data for polyp images.

[0082] Step S2: Input the image to be processed into the pre-trained network to extract high-level features of different resolutions;

[0083] The image to be processed is input into the pre-trained network to extract the features of five levels of different resolutions of the image to be processed. and is the low-level feature of the image to be processed, , and These are the high-level features of the image to be processed. These features have different resolutions, from low to high. , , , and , Indicates the height of the image to be processed. Indicates the width of the image to be processed.

[0084] Step S3: The high-level features of different resolutions are aggregated through the cascade fusion partial decoder to obtain the global feature map. ;

[0085] like Figure 2 As shown, Figure 2 This is the structural diagram of the cascade fusion partial decoder. In the cascade fusion partial decoder, three high-level features of the image to be processed extracted from the Res2Net network are first gradually aggregated through a series of upsampling and convolution operations. , and . Use upsampling and convolution layers to , After upsampling, Element-by-element multiplication is performed to achieve feature fusion; then, the fused features are further refined through a series of convolutional layers to enhance the expressiveness of the features; finally, the global feature map obtained by cascading the fusion partial decoders is , which can be used to generate accurate segmentation results. Compared with the traditional encoder-decoder framework, the cascade fusion partial decoder effectively reduces the computational cost while maintaining the segmentation accuracy by focusing on the aggregation of high-level features.

[0086] like Figure 4 As shown, Figure 4 It is a schematic diagram of the multi-scale mobile convolution module. Figure 4 (a) in the figure is the structural diagram of the multi-scale mobile convolution module. Figure 4 (b) is a schematic diagram of three parallel moving inverted bottleneck convolution branches extracting features of different scales;

[0087] Step S4: Input each high-level feature into the corresponding multi-scale mobile convolution module, extract the features of each high-level feature at different scales, and obtain the target fusion feature corresponding to each high-level feature, including:

[0088] After the high-level features are processed by batch normalization and PReLU activation function, they are reduced in dimension through the first 1x1 convolutional layer to obtain the high-level features after dimensionality reduction.

[0089] The high-level features after dimensionality reduction are input into multiple parallel moving inverted bottleneck convolution branches to extract features of different scales, and the output features of each branch are obtained;

[0090] Concatenate the output features of all branches along the channel dimension to obtain a comprehensive feature map;

[0091] After the comprehensive feature map is processed by batch normalization and PReLU activation function in sequence, it is passed through the second 1x1 convolution layer for dimensionality increase to obtain the comprehensive feature map after dimensionality increase;

[0092] Perform residual connection between high-level features and their corresponding dimensional integrated feature maps to obtain target fusion features corresponding to high-level features;

[0093] The last three high-level features , and After passing through the corresponding multi-scale mobile convolution modules, the first target fusion feature is obtained , the second target fusion feature , the third target fusion feature .

[0094] In this embodiment, preferably, each multi-scale moving convolution module includes: a first 1x1 convolution layer, a second 1x1 convolution layer, and a plurality of parallel moving inverted bottleneck convolution branches;

[0095] Each parallel branch includes multiple moving inverted bottleneck convolution modules (MBConv modules) connected in series along the forward propagation direction. The output of the last moving inverted bottleneck convolution module of each branch is concatenated with the outputs of other moving inverted bottleneck convolution modules in the branch along the channel dimension.

[0096] In this embodiment, three parallel moving inverted bottleneck convolution branches are used, each parallel branch includes three moving inverted bottleneck convolution modules connected in series along the forward propagation direction, and the output of the last moving inverted bottleneck convolution module of each branch is spliced ​​with the outputs of other moving inverted bottleneck convolution modules in the branch along the channel dimension.

[0097] The number of feature channels extracted by the three moving inverted bottleneck convolution modules in the first two branches is 1 / 16, 1 / 16 and 1 / 8 of the number of corresponding input feature channels respectively, and the number of feature channels extracted by the three moving inverted bottleneck convolution modules in the third branch is 1 / 8, 1 / 8 and 1 / 4 of the number of corresponding input feature channels respectively;

[0098] like Figure 3 As shown, Figure 3 This is a structural diagram of the mobile inverted bottleneck convolution module in the multi-scale mobile convolution module. In this embodiment, preferably, the mobile inverted bottleneck convolution module includes: an extended convolution layer, a first two-dimensional batch normalization layer, a first ReLU6 activation function layer, a depth convolution layer, a second two-dimensional batch normalization layer, a second ReLU6 activation function layer, a squeeze and excitation block, a projection convolution layer, and a third two-dimensional batch normalization layer connected in sequence along the forward propagation direction.

[0099] Among them, the squeeze and excitation block uses an adaptive average pooling layer to perform global average pooling on each channel of the input feature map of the squeeze and excitation block, compressing the information of the spatial dimension to the channel dimension to generate a global average pooling feature map , global average pooling feature map Through 1x1 convolution, the global average pooling feature map is fully connected. The number of channels is compressed; the compressed feature map is nonlinearly transformed by the Relu activation function to obtain a feature map with channel attention , feature map with channel attention Through 1x1 convolution and Sigmoid activation function in turn, the channel weights are output, and the channel weights are applied to each channel of the input feature map of the squeeze and excitation block for recalibration to obtain the output feature map of the squeeze and excitation block.

[0100] The input features of the mobile inverted bottleneck convolution module are expanded by the extended convolution layer to expand the number of channels, and then batch normalized by the first two-dimensional batch normalization layer in sequence, and the first ReLU6 activation function layer is nonlinearly transformed to obtain the processed input features of the mobile inverted bottleneck convolution module. The processed input features of the mobile inverted bottleneck convolution module are independently convolved channel by channel on each channel through the deep convolution layer, and then sequentially passed through the second two-dimensional batch normalization layer, the second ReLU6 activation function layer, and the squeeze and excitation block. The squeeze and excitation block adaptively recalibrates the weights on the feature channel. The output of the squeeze and excitation block is restored to the target channel number through the projection convolution layer. The output of the projection convolution layer is batch normalized through the third two-dimensional batch normalization layer, and the output of the third two-dimensional batch normalization layer is used as the output feature of the mobile inverted bottleneck convolution module;

[0101] When the number of channels of the input feature of the mobile inverted bottleneck convolution module is consistent with the number of channels of the output of the third two-dimensional batch normalization layer, and the stride is 1, the input feature of the mobile inverted bottleneck convolution module can be directly connected to the output of the third two-dimensional batch normalization layer through a skip connection, and used as the output feature of the mobile inverted bottleneck convolution module. This can promote the effective propagation of the gradient, prevent the gradient from gradually decreasing during the back propagation process, and thus cause the gradient disappearance problem. The use of residual connections allows the network to converge faster.

[0102] The mobile inverted bottleneck convolution module uses dilated convolution and depthwise separable convolution to effectively extract features. Dilated convolution expands the number of channels of the feature map and increases the expressiveness of the feature map. The depthwise convolution uses a 3x3 convolution kernel to perform independent spatial convolution on each channel, and then uses a 1x1 convolution to perform convolution on all input channels to generate the final output. The squeeze and excitation block adaptively recalibrates the weights of the channels, so that the network can enhance the focus on useful features and suppress unimportant features, thereby improving the feature representation ability and model performance. Then use 1×1 convolution to compress the high-dimensional feature map back to low dimensions to control the computational complexity and the number of parameters, while integrating high-dimensional feature information so that the number of channels of the output features of the mobile inverted bottleneck convolution module meets the network design requirements.

[0103] The multi-scale mobile convolution module uses the first 1x1 convolution layer and the second 1x1 convolution layer to reduce and increase the dimension, reducing the amount of calculation while maintaining the integrity of the features. Extracting features of different scales through multiple parallel mobile inverted bottleneck convolution branches helps capture the multi-scale information of polyps in the image, improve segmentation accuracy, enhance gradient flow, increase the depth of the polyp image segmentation model, and make it easier to train, improving the stability and performance of the model; through the batch normalization layer and PReLU activation function, the training process can be stabilized, and the convergence speed and performance of the polyp image segmentation model can be improved.

[0104] like Figure 5 As shown, Figure 5 The structure diagram of the axial hierarchical multi-scale cross-inverse attention module is shown in FIG. 5 . Step S5: The target fusion feature corresponding to the global feature map and each high-level feature is input into multiple axial hierarchical multi-scale cross-inverse attention modules, and the polyp area is identified by erasing the foreground. The output of the last axial hierarchical multi-scale cross-inverse attention module is passed through the Sigmoid activation function to obtain the polyp image segmentation prediction map.

[0105] In this embodiment, preferably, the global feature map The target fusion features corresponding to each high-level feature are input into multiple axial hierarchical multi-scale cross-inverse attention modules to identify the polyp area by erasing the foreground. The output of the last axial hierarchical multi-scale cross-inverse attention module is passed through the Sigmoid activation function to obtain the polyp image segmentation prediction map, including:

[0106] The global feature map After downsampling, the features are fused with the third target Inputting the third axial hierarchical multi-scale cross-inverse attention module, and extracting output features of the third axial hierarchical multi-scale cross-inverse attention module;

[0107] In this embodiment, preferably, the global feature map After downsampling, the features are fused with the third target Input the third axial hierarchical multi-scale cross-inverse attention module, and extract the output features of the third axial hierarchical multi-scale cross-inverse attention module, including:

[0108] The third target fusion feature Through the self-attention module, features of different scales are extracted and fused to obtain the output features of the self-attention module;

[0109] The output features of the self-attention module are passed through the axial patch-aware attention module to extract local area information and obtain the output features of the axial patch-aware attention module;

[0110] The downsampled global feature map is passed through the Sigmoid activation function, fused with the output features of the self-attention module, and then input into the lesion-aware cross-attention module. The boundary features are enhanced through the lesion-aware attention to obtain the output features of the lesion-aware cross-attention module.

[0111] After expanding the number of channels of the output features of the lesion-aware cross-attention module, they are multiplied with the output features of the axial patch-aware attention module to obtain the output features of the third axial hierarchical multi-scale cross-inverse attention module.

[0112] The output features of the third axial layered multi-scale cross-inverse attention module are added to the downsampled global feature map to obtain the third feature map ;

[0113] The third characteristic diagram Fusion of features with the second target Inputting the second axial hierarchical multi-scale cross-inverse attention module, and extracting output features of the second axial hierarchical multi-scale cross-inverse attention module;

[0114] In this embodiment, the third characteristic map Fusion of features with the second target Input the second axial hierarchical multi-scale cross-inverse attention module, and extract the output features of the second axial hierarchical multi-scale cross-inverse attention module, including:

[0115] The second target fusion feature Through the self-attention module, features of different scales are extracted and fused to obtain the output features of the second self-attention module;

[0116] The output feature of the second self-attention module is passed through the axial patch-aware attention module to extract local area information, thereby obtaining the output feature of the second axial patch-aware attention module;

[0117] The third characteristic diagram After passing through the Sigmoid activation function, it is fused with the output features of the second self-attention module, and then input into the lesion-aware cross-attention module. The boundary features are enhanced by the lesion-aware attention, and the output features of the second lesion-aware cross-attention module are obtained.

[0118] After expanding the number of channels of the output features of the second lesion-aware cross-attention module, they are multiplied with the output features of the second axial patch-aware attention module to obtain the output features of the second axial hierarchical multi-scale cross-inverse attention module.

[0119] After upsampling, the third feature map is added to the output features of the second axial layered multi-scale cross-inverse attention module to obtain the second feature map ;

[0120] Second feature map Fusion features with the first target Inputting the first axial hierarchical multi-scale cross-inverse attention module, extracting the output features of the first axial hierarchical multi-scale cross-inverse attention module;

[0121] In this embodiment, the third characteristic map Fusion of features with the second target Input the second axial hierarchical multi-scale cross-inverse attention module, and extract the output features of the second axial hierarchical multi-scale cross-inverse attention module, including:

[0122] The first target fusion feature Through the self-attention module, features of different scales are extracted and fused to obtain the output features of the first self-attention module;

[0123] The output feature of the first self-attention module is passed through the axial patch-aware attention module to extract local area information, thereby obtaining the output feature of the first axial patch-aware attention module;

[0124] Second feature map After passing through the Sigmoid activation function, it is fused with the output features of the first self-attention module, and then input into the lesion-aware cross-attention module. The boundary features are enhanced by the lesion-aware attention, and the output features of the first lesion-aware cross-attention module are obtained.

[0125] After expanding the number of channels of the output feature of the first lesion-aware cross-attention module, it is multiplied with the output feature of the first axial patch-aware attention module to obtain the output feature of the first axial hierarchical multi-scale cross-inverse attention module.

[0126] After upsampling, the second feature map is added to the output feature of the first axial layered multi-scale cross-inverse attention module to obtain the first feature map ;

[0127] First feature map The polyp image segmentation prediction map is generated through the Sigmoid activation function.

[0128] In this embodiment, preferably, each axial hierarchical multi-scale cross-inverse attention module includes: a self-attention module (ESA module), an axial patch-aware attention module (PPA module) and a lesion-aware cross module (LCA module).

[0129] like Figure 6 As shown, Figure 6 This is the structural diagram of the self-attention module. The self-attention module uses a multi-head self-attention mechanism and pyramid pooling to capture features of different scales, enhance the polyp image segmentation model's understanding of the global context, and more accurately determine the overall position and shape of the polyp.

[0130] In this embodiment, preferably, the input features of the self-attention module are passed through a multi-scale residual pyramid module to capture and fuse features of different scales to obtain output features of the multi-scale residual pyramid module;

[0131] Input the output features of the multi-scale residual pyramid module and the input features of the self-attention module into the first multi-head self-attention module, adjust the attention weights, and obtain the output features of the first multi-head self-attention module;

[0132] The dimension of the input feature of the self-attention module is adjusted to the dimension of the output feature of the first multi-head self-attention module, and after adding it to the output feature of the first multi-head self-attention module, a comprehensive attention feature is obtained;

[0133] The comprehensive attention features are added to the comprehensive attention features after feature fusion to obtain the output features of the self-attention module.

[0134] like Figure 8 As shown, Figure 8 This is the structural diagram of the axial patch-aware attention module. The axial patch-aware attention module uses a parallel multi-branch method. Each branch is responsible for extracting features of different scales and levels. It can capture detailed information of local areas and refine the boundaries of polyps, which helps to better obtain the boundary information of polyps.

[0135] In this embodiment, preferably, the axial patch perception attention module includes: three parallel branches and an attention module, and the three parallel branches include: a local branch, a serial convolution branch and a global branch;

[0136] The input features of the axial patch-aware attention module are respectively input into the local branch, serial convolution branch, and global branch to extract features, and the output features of each branch are obtained. After adding the output features of all branches, they are input into the attention module for adaptive feature enhancement to obtain the output features of the axial patch-aware attention module.

[0137] The input features of the axial patch-aware attention module are input into the local branch, serial convolution branch, and global branch to extract features, and the output features of each branch are obtained. After adding the output features of all branches, they are input into the attention module for adaptive feature enhancement, and the size is obtained. Output features of the axial patch-aware attention module.

[0138] in, represents the height of the input feature of the axial patch-aware attention module, represents the width of the input feature of the axial patch-aware attention module, Represents the width of the input features of the axial patch-aware attention module.

[0139] The distinction between local and global branches is achieved by controlling the patch size parameter P, which is achieved by aggregating and shifting non-overlapping patches in the spatial dimension.

[0140] like Figure 7 As shown, Figure 7 This is the structural diagram of the lesion-aware cross-attention module. The lesion-aware cross-attention module can improve the model's sensitivity to the local features of polyps and more clearly locate the tissue boundaries around polyps.

[0141] The lesion-aware cross-module includes: a second multi-head self-attention module;

[0142] After enhancing the features of the lesion-aware cross-module input by element-wise multiplication, the context information is extracted by the second multi-head self-attention module to obtain the output features of the second multi-head self-attention module;

[0143] The input features of the lesion perception cross module are dimensionally adjusted and added to the output features of the second multi-head self-attention module to obtain the module comprehensive features;

[0144] The module comprehensive features are added to the module comprehensive features after feature fusion and the dimensions are adjusted to obtain the lesion-aware cross-module output features.

[0145] The self-attention module provides global context information to help determine the approximate location of polyps; the lesion-aware cross-attention module further refines these areas and enhances boundary features through lesion-aware attention; the axial patch-aware attention module accurately captures and refines the edges of polyps through parallel processing of local features. Through the collaboration of the three modules, the polyp image segmentation model can more comprehensively understand the image content and process information carefully from global to local, thereby achieving higher accuracy and robustness in the polyp segmentation task, which is very effective for locating small polyps and identifying polyp boundary information.

[0146] In this embodiment, the polyp image segmentation model proposed in the present invention is tested on five polyp segmentation datasets, namely, the ETIS dataset, the CVC-ClinicDB dataset, the CVC-ColonDB dataset, the CVC-300 dataset, and the Kvasir dataset. These five datasets are specifically used for polyp segmentation. By training and testing on these datasets, the robustness and generalization ability of the segmentation algorithm can be ensured.

[0147] The ETIS dataset is a polyp detection dataset provided by ETIS and Larib, specifically used to evaluate the performance of automatic polyp detection algorithms; the CVC-ClinicDB dataset is a standardized polyp segmentation dataset provided by CVC, used to evaluate automatic segmentation algorithms; the CVC-ColonDB dataset is another polyp segmentation dataset provided by CVC; the CVC-300 dataset is an early polyp segmentation dataset provided by CVC, often used to evaluate the performance of polyp detection and segmentation algorithms; the Kvasir dataset is a widely used endoscopic image dataset, which includes polyp images, and its main goal is to help develop and evaluate detection and segmentation algorithms for digestive system diseases.

[0148] In this embodiment, 1459 intestinal polyp images were collected from Changshu Hospital Affiliated to Soochow University and Changshu Traditional Chinese Medicine Hospital, of which 467 were adenomatous polyps and 992 were serrated lesions. With the help of two senior endoscopists with more than ten years of work experience, each image was accurately segmented for testing the polyp image segmentation model proposed in the present invention.

[0149] In this embodiment, the training set consists of 900 images from KvasirSEG and 550 images from ClinicDB. The test set comes from 100 images from Kvasir-SEG, 62 images from CVC-ClincDB, 380 images from CVC-ColorDB, 60 images from CVC-300, 196 images from ETIS, and 1459 images from Changshu Hospital Affiliated to Soochow University and Changshu Traditional Chinese Medicine Hospital.

[0150] like Fig.13 As shown, Fig.13The figure shows the loss function of the polyp image segmentation model when the learning rate is 1e-4. The polyp image segmentation model is implemented on a single NVIDIA GeForce RTX 4070 GPU. Pytorch2.1.0 is configured in the Python3.9 environment, 50 training rounds (epochs) are used, the learning rate is adjusted to 1e-4, the learning rate decay value is 0.1, the decay period is 20, the Adam optimizer is selected for data enhancement, the number of samples per batch is set to 16, the size of the training image is set to 352, and the threshold of the gradient clipping is set to 0.5.

[0151] In this embodiment, the polyp image segmentation model uses multi-scale training to adjust the input image to different scales, including reducing the size of the input image to 75% of the original size and enlarging the size of the input image to 125% of the original size. While training, the polyp image segmentation model is tested for each training epoch.

[0152] Each time the model is trained, the training result of the first training epoch is set as the target polyp image segmentation model, and a test function is defined to test the average Dice coefficient of each training epoch. The average Dice coefficient of each training epoch is compared with the average Dice coefficient of the target polyp image segmentation model. If the average Dice coefficient of the training epoch is better than the average Dice coefficient of the target polyp image segmentation model, the training result of the training epoch is set as the target polyp image segmentation model. The average Dice coefficient of each model is recorded in txt, and the average Dice of the target model tested with the CVC-300 dataset and the historical average Dice are saved in the log.

[0153] After training, the target polyp image segmentation model is used to segment the test dataset, the prediction results are saved as image files through the imageio.imread() function, and the predicted result images are numerically evaluated with the true annotated images.

[0154] In this embodiment, Matlab is used to evaluate the segmentation performance of the target polyp image segmentation model. Six widely used evaluation indicators are analyzed, including mean Dice coefficient (mDice), mean intersection over union (mIoU), mean absolute error (MAE), weighted F-measure, S-measure, and enhanced matching index (E-measure).

[0155] The Dice coefficient is a statistic used to measure similarity, especially used in binary image segmentation tasks. , Dice represents the Dice coefficient, A represents the current polyp image segmentation prediction map, B represents the true label corresponding to the current polyp image segmentation prediction map, represents the intersection of the current polyp image segmentation prediction map and the true label, and They respectively represent the number of elements of the current polyp image segmentation prediction map and the corresponding true label. The average Dice coefficient (mDice) can measure the overlap between the current polyp image segmentation prediction map and the corresponding true label, and more comprehensively reflect the overall performance of the polyp image segmentation model.

[0156] The calculation formula of the average Dice coefficient (mDice) is:

[0157] ,

[0158] Among them, mDice represents the average Dice coefficient, Indicates Polyp image segmentation prediction map of samples, Indicates The true labels of samples, represents the total number of samples, .

[0159] IoU is the intersection over union ratio, which is used to calculate the ratio of the overlapping area to the joint area between the current polyp image segmentation prediction map and the true label corresponding to the polyp image segmentation prediction map. , A represents the current polyp image segmentation prediction map, B represents the true label corresponding to the current polyp image segmentation prediction map, represents the intersection of the current polyp image segmentation prediction map and the corresponding true label, It represents the union of the current polyp image segmentation prediction map and the corresponding true label. The mean intersection over union (mIoU) measures the degree of overlap between the polyp image segmentation prediction map and the true label, which can more comprehensively reflect the overall performance of the model.

[0160] The calculation formula for mean intersection over union (mIoU) is:

[0161] ,

[0162] Among them, mIoU means mean intersection over union, Indicates The segmentation prediction graph of the sample polyp image and the The intersection of the true labels of samples, Indicates The segmentation prediction map of the sample polyp image and the The union of the true labels of the samples.

[0163] The mean absolute error (MAE) can measure the average absolute error between the polyp image segmentation prediction map of multiple samples and the corresponding true label.

[0164] The mean absolute error (MAE) is calculated as:

[0165] ,

[0166] Among them, MAE stands for mean absolute error, Indicates The absolute error of the samples.

[0167] The F-Measure is a comprehensive indicator calculated based on precision and recall. The pixels predicted as polyp areas in the current polyp image segmentation prediction map are regarded as positive classes, and the pixels predicted as non-polyp areas are regarded as negative classes. The calculation formula of the F-Measure is:

[0168] ,

[0169] The calculation formula of Precision is: ;

[0170] The calculation formula of recall is: ;

[0171] in, represents the F value, Precision represents the precision, Recall represents the recall rate, Z represents the number of pixels that are actually positive and predicted as positive, O represents the number of pixels that are actually negative and predicted as positive, Q represents the number of pixels that are actually positive and predicted as negative, Represents weight.

[0172] The weighted F-measure is the weighted average of the F-measure. The weighted F-measure can be used to measure the comprehensive performance of the segmentation results.

[0173] The S-measure is a structural similarity metric specifically used to evaluate the structural similarity of binary segmentation results. The S-measure can be used to measure the structural similarity between the segmentation results and the true labels.

[0174] The enhanced matching index (E-measure) combines precision, recall, and structural similarity (S), taking into account both overall and local matching. The calculation formula is:

[0175] ,

[0176] Among them, E represents the enhanced matching index (E-measure), which can compare the performance of the model segmentation results in overall and local matching; S represents the structural similarity between the current polyp image segmentation prediction map and the true label corresponding to the polyp image segmentation prediction map, taking into account the brightness, contrast and structural information of the image. The closer the S value is to 1, the more similar the two are; It reflects the difference between the structural similarity between the current polyp image segmentation prediction map and the actual label corresponding to the polyp image segmentation prediction map and 1, indicating the difference between the current polyp image segmentation prediction result and the actual result.

[0177] As shown in Table 1, Table 1 compares the segmentation performance of the polyp image segmentation model provided by the present invention with that of U-Net, UNet++, random frontier model (SFA), parallel reverse attention network (PraNet), and context axial preserving attention network (CaraNet) on five data sets.

[0178] Table 1

[0179]

[0180] According to the experimental data in Table 1, it can be seen that the overall performance of the polyp image segmentation model proposed in the present invention on the five datasets of Kvasir, CVC-ClinicDB, CVC-ColonDB, ETIS and CVC-300 is better than the comparison model.

[0181] like Fig. 9 As shown, Fig. 9 The following are schematic diagrams of the polyp segmentation results of each model in different data sets. A polyp image is selected from each of the five data sets to demonstrate the segmentation effects of different models. It can be seen that the polyp image segmentation model proposed in the present invention is superior to the compared models in terms of overall segmentation effect.

[0182] like Fig.10 , 11 As shown, Fig.10 The figure is a schematic diagram of the average Dice coefficient of each model on the CVC-ColonDB, ETIS, and CVC-300 datasets. Fig.11It is a schematic diagram of the mean intersection and union of each model on the CVC-ColonDB, ETIS, and CVC-300 datasets. Through the visualized line graph, it can be seen that the polyp image segmentation model proposed in the present invention is superior to other compared models in terms of mDice and mIoU on the three datasets of CVC-ColonDB, ETIS, and CVC-300.

[0183] In this embodiment, the polyp image segmentation model is trained on 900 KvasirSEG images and 550 ClinicDB images, and the effect of testing on 196 ETIS datasets is greatly improved compared with other models. The ETIS dataset covers polyps of various shapes and sizes, with high annotation quality, and has a great testing effect on comprehensively evaluating the actual performance of the model in polyp detection and segmentation tasks. Since the polyp image segmentation model proposed in the present invention includes multiple multi-scale moving convolution modules and axial layered multi-scale cross-inverse attention modules, it can effectively identify the boundary information of polyps, thereby improving the segmentation performance, and thus has more advantages in the segmentation of complex small target datasets.

[0184] As shown in Table 2, Table 2 shows the test results of the polyp image segmentation model, parallel reverse attention network (PraNet), and context axial preservation attention network (CaraNet) proposed in the present invention on the datasets of Changshu Hospital Affiliated to Soochow University and Changshu Traditional Chinese Medicine Hospital.

[0185] Table 2

[0186]

[0187] According to the evaluation index data in Table 2, the polyp image segmentation model proposed in the present invention has better application effect in actual clinical environment than the parallel reverse attention network (PraNet) and the context axial preservation attention network (CaraNet). The polyp image segmentation model proposed in the present invention can effectively enhance the effect of polyp segmentation and enhance the segmentation effect on clinical data by extracting features at multiple scales and accurately capturing and refining the edges of polyps.

[0188] In order to verify the effectiveness of the polyp image segmentation model proposed in the present invention, we conducted ablation experiments on five datasets, namely Kvasir, CVC-ClinicDB, CVC-ColonDB, ETIS and CVC-300, to evaluate the influence of the two-output, multi-scale mobile convolution module and the axial layered multi-scale cross inverse attention module adopted in the present invention. The results of the ablation experiment are shown in Table 3:

[0189] Table 3

[0190]

[0191] From the experimental results, we can draw the following conclusions: When only two outputs are used instead of all four outputs, the average Dice coefficient (mDice) is improved on the five datasets, which shows that the middle layer only extracts features without output prediction, which can make the final result better. The output of the second axial layer multi-scale cross-inverse attention module in the middle layer , the output of the third axial hierarchical multi-scale cross-inverse attention module Redundant information is included, and there is no need to add the final polyp image segmentation calculation. After introducing the multi-scale mobile convolution module, the polyp segmentation effect of the polyp image segmentation model proposed in the present invention has been further improved. After further extracting the features of each layer trained by the backbone network, the context information can be better identified. When the axial layered multi-scale cross-inverse attention module is used, the model effect is better.

[0192] The polyp image segmentation model proposed in the present invention is significantly superior to the existing deep learning models in various indicators on five public datasets, the datasets of Changshu Hospital Affiliated to Soochow University and Changshu Hospital of Traditional Chinese Medicine, providing important insights for improving the performance of small object segmentation methods.

[0193] In order to verify the convergence of the method, we conducted a series of experiments on five datasets: ETIS, CVC-ClinicDB, CVC-ColonDB, CVC-300, and Kvasir. The experimental results are shown in the figure. Fig.12 As shown, Fig.12 The figure is a schematic diagram of the average Dice coefficient of the polyp image segmentation model on five data sets. The polyp image segmentation model proposed in the present invention almost converges after 25 iterations on the five data sets. These results together prove that the polyp image segmentation model proposed in the present invention has excellent convergence.

[0194] Reference Fig.15 As shown, this second embodiment also provides a polyp image segmentation system, including:

[0195] A model building module 10 is used to build a polyp image segmentation model, wherein the polyp image segmentation model includes: a pre-trained network, a cascade fusion partial decoder, a plurality of multi-scale mobile convolution modules and an axial layered multi-scale cross inverse attention module;

[0196] A high-level feature extraction module 20 is used to input the image to be processed into a pre-trained network to extract high-level features of different resolutions;

[0197] A global feature map extraction module 30 is used to aggregate high-level features of different resolutions through a cascade fusion partial decoder to obtain a global feature map;

[0198] The target fusion feature extraction module 40 is used to input each high-level feature into the corresponding multi-scale mobile convolution module, respectively extract the features of each high-level feature at different scales, and obtain the target fusion feature corresponding to each high-level feature, including:

[0199] After the high-level features are processed by batch normalization and PReLU activation function, they are reduced in dimension through the first 1x1 convolutional layer to obtain the high-level features after dimensionality reduction.

[0200] The high-level features after dimensionality reduction are input into multiple parallel moving inverted bottleneck convolution branches to extract features of different scales, and the output features of each branch are obtained;

[0201] Concatenate the output features of all branches along the channel dimension to obtain a comprehensive feature map;

[0202] After the comprehensive feature map is processed by batch normalization and PReLU activation function in sequence, it is passed through the second 1x1 convolution layer for dimensionality increase to obtain the comprehensive feature map after dimensionality increase;

[0203] Perform residual connection between high-level features and their corresponding dimensional integrated feature maps to obtain target fusion features corresponding to high-level features;

[0204] The prediction module 50 is used to input the target fusion features corresponding to the global feature map and each high-level feature into multiple axial hierarchical multi-scale cross-inverse attention modules, identify the polyp area by erasing the foreground, and obtain the polyp image segmentation prediction map through the output of the last axial hierarchical multi-scale cross-inverse attention module through the Sigmoid activation function.

[0205] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0206] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0209] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A polyp image segmentation method, characterized in that: The following steps are involved: Constructing a polyp image segmentation model, the polyp image segmentation model comprising: a pre-trained network, a cascade fusion partial decoder, a plurality of multi-scale mobile convolution modules and an axial layered multi-scale cross inverse attention module; Input the image to be processed into the pre-trained network to extract high-level features of different resolutions; High-level features of different resolutions are aggregated through cascaded fusion partial decoders to obtain a global feature map; Each high-level feature is input into the corresponding multi-scale mobile convolution module to extract the features of each high-level feature at different scales, and obtain the target fusion features corresponding to each high-level feature, including: After the high-level features are processed by batch normalization and PReLU activation function, they are reduced in dimension through the first 1x1 convolutional layer to obtain the high-level features after dimensionality reduction. The high-level features after dimensionality reduction are input into multiple parallel moving inverted bottleneck convolution branches to extract features of different scales, and the output features of each branch are obtained; Concatenate the output features of all branches along the channel dimension to obtain a comprehensive feature map; After the comprehensive feature map is processed by batch normalization and PReLU activation function in sequence, it is passed through the second 1x1 convolution layer for dimensionality increase to obtain the comprehensive feature map after dimensionality increase; Perform residual connection between high-level features and their corresponding dimensional integrated feature maps to obtain target fusion features corresponding to high-level features; The target fusion features corresponding to the global feature map and each high-level feature are input into multiple axial hierarchical multi-scale cross-inverse attention modules, and the polyp area is identified by erasing the foreground. The output of the last axial hierarchical multi-scale cross-inverse attention module is passed through the Sigmoid activation function to obtain the polyp image segmentation prediction map; wherein each axial hierarchical multi-scale cross-inverse attention module includes: a self-attention module, an axial patch-aware attention module and a lesion-aware cross module; The self-attention module includes: a multi-scale residual pyramid module and a first multi-head self-attention module; The input features of the self-attention module are passed through the multi-scale residual pyramid module to capture and fuse features of different scales to obtain the output features of the multi-scale residual pyramid module; Input the output features of the multi-scale residual pyramid module and the input features of the self-attention module into the first multi-head self-attention module, adjust the attention weights, and obtain the output features of the first multi-head self-attention module; The dimension of the input feature of the self-attention module is adjusted to the dimension of the output feature of the first multi-head self-attention module, and after adding it to the output feature of the first multi-head self-attention module, a comprehensive attention feature is obtained; The comprehensive attention features are added to the comprehensive attention features after feature fusion to obtain the output features of the self-attention module; The axial patch-aware attention module includes: three parallel branches and an attention module. The three parallel branches include: local branch, serial convolution branch and global branch; The input features of the axial patch-aware attention module are input into the local branch, serial convolution branch, and global branch to extract features, and the output features of each branch are obtained. After adding the output features of all branches, they are input into the attention module for adaptive feature enhancement to obtain the output features of the axial patch-aware attention module; The lesion-aware cross-module includes: a second multi-head self-attention module; After enhancing the features of the lesion-aware cross-module input by element-wise multiplication, the context information is extracted by the second multi-head self-attention module to obtain the output features of the second multi-head self-attention module; The input features of the lesion perception cross module are dimensionally adjusted and added to the output features of the second multi-head self-attention module to obtain the module comprehensive features; The module comprehensive features are added to the module comprehensive features after feature fusion and the dimensions are adjusted to obtain the lesion-aware cross-module output features.

2. A polyp image segmentation method according to claim 1, characterized in that: Each moving inverted bottleneck convolution branch includes multiple moving inverted bottleneck convolution modules connected in series along the forward propagation direction. The output of the last moving inverted bottleneck convolution module of each branch is concatenated with the outputs of other moving inverted bottleneck convolution modules in the branch along the channel dimension to obtain the output features of each branch.

3. A polyp image segmentation method according to claim 2, characterized in that: Each moving inverted bottleneck convolution module includes: an extended convolution layer, a first two-dimensional batch normalization layer, a first ReLU6 activation function layer, a depth convolution layer, a second two-dimensional batch normalization layer, a second ReLU6 activation function layer, a squeeze and excitation block, a projection convolution layer, and a third two-dimensional batch normalization layer connected in sequence along the forward propagation direction.

4. The polyp image segmentation method according to claim 1, characterized in that: Input the image to be processed into the pre-trained network to extract the features of five levels of different resolutions of the image to be processed; The last three high-level features are respectively passed through the corresponding multi-scale mobile convolution modules to obtain the first target fusion feature, the second target fusion feature, and the third target fusion feature.

5. A polyp image segmentation method according to claim 4, characterized in that: The target fusion feature corresponding to the global feature map and each high-level feature is input into multiple axial hierarchical multi-scale cross reverse attention modules, and the polyp area is identified by erasing the foreground. The output of the last axial hierarchical multi-scale cross reverse attention module is passed through the Sigmoid activation function to obtain a polyp image segmentation prediction map, including: After downsampling, the global feature map is input into the third axial hierarchical multi-scale cross inverse attention module with the third target fusion feature, and the output feature of the third axial hierarchical multi-scale cross inverse attention module is extracted; The output features of the third axial hierarchical multi-scale cross-inverse attention module are added to the downsampled global feature map to obtain the third feature map; The third feature map and the second target fusion feature are input into the second axial hierarchical multi-scale cross inverse attention module, and the output feature of the second axial hierarchical multi-scale cross inverse attention module is extracted; After upsampling, the third feature map is added to the output feature of the second axial hierarchical multi-scale cross inverse attention module to obtain the second feature map; The second feature map and the first target fusion feature are input into the first axial hierarchical multi-scale cross inverse attention module, and the output feature of the first axial hierarchical multi-scale cross inverse attention module is extracted; After upsampling, the second feature map is added to the output feature of the first axial hierarchical multi-scale cross inverse attention module to obtain the first feature map; The first feature map is activated by the Sigmoid function to generate a polyp image segmentation prediction map.

6. A polyp image segmentation method according to claim 5, characterized in that: The downsampling of the global feature map and the fusion feature of the third target are input into the third axial hierarchical multi-scale cross inverse attention module, and the output feature of the third axial hierarchical multi-scale cross inverse attention module is extracted, including the following steps: The third target fusion feature is passed through the self-attention module to extract and fuse features of different scales to obtain the output feature of the self-attention module; The output features of the self-attention module are passed through the axial patch-aware attention module to extract local area information and obtain the output features of the axial patch-aware attention module; The downsampled global feature map is passed through the Sigmoid activation function, fused with the output features of the self-attention module, and then input into the lesion-aware cross-attention module. The boundary features are enhanced through the lesion-aware attention to obtain the output features of the lesion-aware cross-attention module. After expanding the number of channels of the output features of the lesion-aware cross-attention module, they are multiplied with the output features of the axial patch-aware attention module to obtain the output features of the third axial hierarchical multi-scale cross-inverse attention module.

7. The polyp image segmentation method according to claim 1, characterized in that: The pre-trained network is Res2Net network.

8. A polyp image segmentation system, characterized in that: include: A model building module, used to build a polyp image segmentation model, the polyp image segmentation model comprising: a pre-trained network, a cascade fusion partial decoder, a plurality of multi-scale mobile convolution modules and an axial layered multi-scale cross inverse attention module; The high-level feature extraction module is used to input the image to be processed into the pre-trained network and extract high-level features of different resolutions; The global feature map extraction module is used to aggregate high-level features of different resolutions through cascade fusion partial decoders to obtain a global feature map; The target fusion feature extraction module is used to input each high-level feature into the corresponding multi-scale mobile convolution module, extract the features of each high-level feature at different scales, and obtain the target fusion feature corresponding to each high-level feature, including: After the high-level features are processed by batch normalization and PReLU activation function, they are reduced in dimension through the first 1x1 convolutional layer to obtain the high-level features after dimensionality reduction. The high-level features after dimensionality reduction are input into multiple parallel moving inverted bottleneck convolution branches to extract features of different scales, and the output features of each branch are obtained; Concatenate the output features of all branches along the channel dimension to obtain a comprehensive feature map; After the comprehensive feature map is processed by batch normalization and PReLU activation function in sequence, it is passed through the second 1x1 convolution layer for dimensionality increase to obtain the comprehensive feature map after dimensionality increase; Perform residual connection between high-level features and their corresponding dimensional integrated feature maps to obtain target fusion features corresponding to high-level features; The prediction module is used to input the target fusion features corresponding to the global feature map and each high-level feature into multiple axial hierarchical multi-scale cross-inverse attention modules, identify the polyp area by erasing the foreground, and obtain the polyp image segmentation prediction map through the output of the last axial hierarchical multi-scale cross-inverse attention module through the Sigmoid activation function; wherein each axial hierarchical multi-scale cross-inverse attention module includes: a self-attention module, an axial patch-aware attention module and a lesion-aware cross module; The self-attention module includes: a multi-scale residual pyramid module and a first multi-head self-attention module; The input features of the self-attention module are passed through the multi-scale residual pyramid module to capture and fuse features of different scales to obtain the output features of the multi-scale residual pyramid module; Input the output features of the multi-scale residual pyramid module and the input features of the self-attention module into the first multi-head self-attention module, adjust the attention weights, and obtain the output features of the first multi-head self-attention module; The dimension of the input feature of the self-attention module is adjusted to the dimension of the output feature of the first multi-head self-attention module, and after adding it to the output feature of the first multi-head self-attention module, a comprehensive attention feature is obtained; The comprehensive attention features are added to the comprehensive attention features after feature fusion to obtain the output features of the self-attention module; The axial patch-aware attention module includes: three parallel branches and an attention module. The three parallel branches include: local branch, serial convolution branch and global branch; The input features of the axial patch-aware attention module are input into the local branch, serial convolution branch, and global branch to extract features, and the output features of each branch are obtained. After adding the output features of all branches, they are input into the attention module for adaptive feature enhancement to obtain the output features of the axial patch-aware attention module; The lesion-aware cross-module includes: a second multi-head self-attention module; After enhancing the features of the lesion-aware cross-module input by element-wise multiplication, the context information is extracted by the second multi-head self-attention module to obtain the output features of the second multi-head self-attention module; The input features of the lesion perception cross module are dimensionally adjusted and added to the output features of the second multi-head self-attention module to obtain the module comprehensive features; The module comprehensive features are added to the module comprehensive features after feature fusion and the dimensions are adjusted to obtain the lesion-aware cross-module output features.

Images