Image segmentation method based on edge-to-global feature fusion network

Abstract

According to the EGANet polyp image segmentation method provided by the invention, the omission ratio and the error detection ratio can be reduced, and the colonoscopy accuracy is improved. The method is designed for three challenges: a ResNet50-SPD encoder is combined with space-to-depth layer downsampling, and a deconvolution decoder is matched, so that the problem of missing detection due to too small polyp size is solved; designing an edge attention module, supervising an advanced feature layer by using edge information, and coping with missing detection and under-segmentation caused by morphological differences; through data set reflection removal and color shift preprocessing, a global attention module is combined to capture long-distance dependence and channel context, and error detection caused by polyp and background similarity is processed. And finally fusing edge and global attention features to relieve over-segmentation. Experiments show that the segmentation precision of the scheme is superior to that of an existing model.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and medical imaging, and in particular to an image segmentation method based on an edge-to-global feature fusion network. Background Art

[0002] Colorectal cancer is a highly prevalent malignant tumor of the digestive system, and its lesions are mostly concentrated in the junction area of ​​the rectum and sigmoid colon. Therefore, in clinical practice, colorectal cancer is often collectively referred to as colorectal tumor together with rectal cancer. Epidemiological data show that the incidence and mortality rate of this malignant tumor rank third among malignant tumors in my country.

[0003] With the continuous development of medical imaging technology, modern colonoscopy can not only visually observe the appearance of tumors, including tumor volume, extent of invasion, and other superficial features, but also detect and identify pathological changes in the colonic mucosa, including polypoid changes, adenomatous hyperplasia, and primary tumor lesions. Therefore, colonoscopy is an effective means of cancer prevention and treatment.

[0004] However, there are multiple technical bottlenecks in the accurate segmentation of polyps based on colonoscopy images. The morphology of lesions varies greatly, and similar polyps lack uniform standards in color and surface structure, resulting in insufficient adaptability of traditional segmentation models. The boundary between polyps and normal tissue often presents a gradual transition. Coupled with light changes and mucus interference during image acquisition, the edge area has low contrast, making conventional algorithms prone to misjudging boundaries. Data shows that 68% of polyps are less than 1 cm in diameter. In the multi-layer feature extraction of deep learning models, the detailed information of these small targets is easily covered by background noise, resulting in missed detection or false detection. There are often multiple lesions in a single image, and the size difference can reach more than 100 times. This puts a double test on the multi-scale processing capabilities of the model: it is necessary to capture the texture features of microscopic lesions at the millimeter level, and to identify the macroscopic morphology of centimeter-level polyps. All of the above reasons make the task of polyp segmentation extremely complex and difficult.

[0005] Colonoscopic images are affected by equipment limitations and the acquisition environment, and often have three typical types of interference: edge black edges, highlight occlusion, and image blur. To address these problems, current preprocessing solutions revolve around three core directions: image enhancement uses histogram equalization and sharpening filtering to improve contrast and edge clarity; image denoising methods mainly include median filtering, Gaussian filtering, and non-local mean denoising; image standardization is mainly processed through normalization, whitening, and geometric transformation; Sánchez-González et al. proposed a preprocessing system for colonoscopic polyp images, which can effectively remove black edges, specular highlights, and standardize lighting; Nallala et al. Infrared spectral imaging was used for data preprocessing, and principal component analysis and cross-validation based on linear discriminant analysis were used for statistical analysis. Jha et al. first reshaped the size of the polyp images, then performed contrast enhancement and image scaling. Subsequently, in order to increase the training data set, data augmentation techniques were used on the existing polyp images, including but not limited to vertical or horizontal flipping, rotation, image cropping, and image translation. However, through the analysis of the polyp images, it was found that there were many highlights in the polyp images and they were scattered, which easily obscured the detailed information in the image. In addition, the polyps had diverse colors and there was no color contrast between the polyps and the surrounding tissues, which reduced the segmentation ability of the network model.

[0006] The self-attention mechanism is a variant of the attention mechanism that reduces reliance on external information and is better at capturing internal correlations within data or features. Its core idea is that when processing sequential data, each element can be associated with other elements in the sequence, rather than relying solely on elements in adjacent positions. It adaptively captures long-range dependencies between elements by calculating their relative importance. Specifically, for each element in the sequence, the self-attention mechanism calculates its similarity with other elements and normalizes these similarities into attention weights. The output of the self-attention mechanism is then obtained by weighted summing each element with its corresponding attention weight. Wang et al. proposed a non-local self-attention module that directly captures long-range dependencies by calculating the interaction between any two positions, rather than being limited to adjacent points, thereby retaining more information. Shen et al. integrated soft and hard attention into a context fusion module to design an enhanced self-attention network. ICGNet designed a new adaptive context module that adaptively extracts local and global information from the current layer and complements the previous layer's information, resulting in larger and denser features. However, the problem of missing polyp images due to the inability to effectively capture long-range dependencies between elements persists. Summary of the Invention

[0007] This invention aims to provide a method for accurately segmenting polyps. This method utilizes an encoder-decoder architecture, an attention mechanism, and a neural network model to segment polyps and compare the segmentation results with those of other models. The specific implementation of this invention is as follows: Step S1, collecting a data set of polyps under colonoscopy; Step S2: first divide the collected data set into a training set and a test set, and then perform highlight removal and color shift on the images in the training set; Step S3: input the preprocessed image into the ResNet50-SPD backbone network for encoding; Step S4, the outputs of the third, fourth and fifth layers are then input into the global attention module (GA) and the edge attention module (EA) to extract features; Step S5, the output of the second layer is passed through a dual-head attention (DA) module to extract detail features and input them into EA for edge guidance; Step S6, finally, the final prediction image is obtained through three decoder modules; In step S7, the global prediction graph is compared with the real data to obtain the accuracy of the corresponding segmentation graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the examples of the present invention and do not constitute a limitation of the present invention.

[0009] Figure 1 Flow chart of the steps of the method of the present invention.

[0010] Figure 2 The EGANet model architecture of the present invention.

[0011] Figure 3 The global attention module architecture of the present invention.

[0012] Figure 4 The edge attention module architecture of the present invention.

[0013] Figure 5 Comparison results of the test dataset of the present invention on Kvasir.

[0014] Figure 6 Comparison results of the test dataset of the present invention on CVC-ClinicDB.

[0015] Figure 7 Comparison results of the test dataset of the present invention on five datasets. DETAILED DESCRIPTION

[0016] The following is combined with Figure 1-7, the present invention is further described in detail. The specific implementation steps are as follows:

[0017] In step S1, data collection is performed: five public datasets, namely ETIS-LaribPolypDB, CVC-ClinicDB, CVC-ColonDB, CVC-T, and Kvasir, are downloaded.

[0018] In step S2, data preprocessing is performed: the training data consists of 1,450 polyp images, consisting of 900 images selected from Kvasir and 550 images from CVC-ClinicDB; the test set consists of the remaining 62 images from CVC-ClinicDB and the remaining 100 images from Kvasir. The input image data is processed by removing highlights and applying color shifting. The steps for removing highlights are as follows: first, grayscale the image to obtain the average grayscale value K; then multiply K by the weight W (default weight 0.3) to obtain the threshold T; then subtract the grayscale of the original image from the threshold T to obtain a value P. P greater than 0.75 is considered a highlight; then, values ​​greater than T are set to 1 and values ​​less than T are set to 0, resulting in a binary mask to generate a highlight location map; finally, after removing highlights using the illuminationChange function, the image is reconstructed. After obtaining the image with the highlights removed, the contrast, brightness, saturation, and hue of the image are randomly shifted using the color shift method (Color Jetter function). This color migration method not only prevents the network model from overly focusing on the color information of the polyps during training but also, to a certain extent, expands the detailed differences between the polyps and the surrounding normal tissues, thereby improving the segmentation accuracy. It is worth noting that image preprocessing is only performed during the training phase.

[0019] In step S3, the preprocessed data is input into the ResNet50-SPD backbone network for encoding: small object segmentation is a very challenging task, because small objects themselves have low resolution and the context information available for model learning is also limited; since the network will lose more or less image information when downsampling with cross-row convolution and pooling layers, ResNet50 is not suitable for segmentation tasks with low-resolution images and small objects; therefore, the present invention introduces ResNet50-SPD as an encoder, such as Figure 2As shown in the figure, this model is based on ResNet50, replacing the convolution with a stride of 2 with a space-to-depth (SPD) layer and a non-cross-row convolution (Conv) layer. The SPD layer downsamples the feature map X, but retains all information in the channel dimension, so there is no information loss; in addition, a non-cross-row convolution is set after each SPD to enable the network to retain as much discriminative feature information as possible; in the encoder-decoder network structure, deconvolution and upsampling operations are usually used to restore the features extracted by the encoder to the size of the original image. Compared with upsampling, deconvolution has parameter learning capabilities, and can achieve image upsampling by learning the weights of the deconvolution kernel, so that the network can better transfer feature information from the encoder to the decoder; in addition, the deconvolution operation can effectively utilize the feature information extracted by the encoder, and reconstruct these feature maps to the size of the original image through the deconvolution process, which helps to retain the spatial structure and semantic information of the image, which is also a beneficial supplement for image segmentation. Therefore Figure 2 As shown, the structure of the decoder is set as: a The convolution adjusts the relationship between the number of channels in the feature map, followed by two deconvolutions to reconstruct the feature map, and finally a The convolution integrates and compresses the features to improve the feature expression ability and accuracy of the model.

[0020] In step S4, the outputs of the third, fourth and fifth layers are then input into the global attention module (GA) and the edge attention module (EA) to extract features and analyze the polyp dataset. Due to the illumination angle of the colonoscope and the cancerous stage of the polyps, the size and shape of the polyps in different polyp images are constantly changing, so it is challenging for feature extraction. Zhang et al. have proved through experiments that local and global contextual information can help the network adapt to polyps of different sizes and enhance feature representation; the attention mechanism strengthens the information related to the optimization target and suppresses background noise; on this basis, this method proposes a global attention module (GA), which can extract global feature information and local feature information under different receptive fields from the encoder; the specific structure of the global attention module is as follows Figure 3 As shown in Figure 2, CA is the channel attention module and SA is the spatial attention module. Both modules are derived from the channel prior convolutional attention CPCA. First, the features output by the encoder are As the input feature of the GA module, the size of its feature map is ( ),Will conduct The channel convolution will The channel characteristics are adjusted to ( ); then the module is divided into two parts. The upper branch uses the non-local attention module (Non-Local) to capture the long-range spatial dependency of features and accurately extract the global dependency of each position, thereby obtaining global context information, and then passing it through the attention module CA to capture the display relationship between feature channels; the lower branch uses three different expansion rates The hole convolution is used to obtain the surrounding context information under different receptive fields, and then the local context information under different scales is captured through the channel attention module CA; secondly, the non-local attention module and the hole convolution module are connected and connected through Finally, it is combined with the convolution and BN layers of Original features of convolution , add them together, and then use the activation function to obtain the fused features. Finally, the spatial attention module SA is used to suppress irrelevant areas in the feature map. EGANet fuses the feature map with rich edge features obtained by the edge attention module and the feature map with global context information obtained by the global attention module, and then multiplies this edge-to-global fusion feature with the deep decoder feature map to achieve the complementarity of shallow detail features and deep semantic features at different scales, thereby achieving better segmentation accuracy. The specific structural design of the edge attention module is as follows: Figure 4 As shown; the specific operation process is: First, the feature map encoded by the ResNet50-SPD high-level feature layer With the downsampled edge feature map ; Perform feature multiplication, and then compare the multiplied feature map with , add them together to get multi-scale information The feature map is then Convolution obtains feature maps Secondly, The pooling size is The global average pooling and maximum pooling of The channel description row vectors Favg and Fmax share two fully connected layers, then perform channel convolution, and use the Relu activation function to fit the complex correlation between channels; then add the two channel description vectors obtained and get the Sigmod function The feature weight vector ,Will A full multiplication operation is performed with the weight vector to obtain feature maps with different importance of different channels; the edge attention module fuses shallow features rich in boundary information with deep features rich in semantic information, and through pooling and full connection operations, fits the complex correlation between channels, adaptively enhances important feature channels, and enables the network to display the importance of edges in the learning image, thereby more effectively capturing edge information in the image and effectively solving the problems of missed detection and under-segmentation of boundaries in multi-target images.

[0021] In step S5, the output of the second layer is passed through a dual-headed attention (DA) module to extract detailed features and input them into EA for edge guidance; edges are important boundaries between objects in the image or between objects and the background, and contain rich semantic information; through edge supervision, edge information can be used to provide additional supervision signals to help the network better understand the structure and semantics in the image, thereby improving the accuracy and stability of segmentation; many works have confirmed that edge information can be used as an additional constraint to guide the network to learn clearer and more accurate expressions; therefore, considering that shallow features contain rich detailed features (including color, edges and textures, etc.), this method uses a dual-headed attention module (Dual Attention) to learn edge attention representations in shallow encoders (second-layer encoders) to capture the spatial dependency between any two positions and the channel dependency between any two channels, thereby enhancing feature representation; then, the obtained feature map rich in edge information is fused with the deep feature map, thereby explicitly learning feature representations that focus on edges.

[0022] In step S6, the final prediction image is obtained after three decoder modules. The convolution adjusts the relationship between the number of channels in the feature map, followed by two deconvolutions to reconstruct the feature map, and finally a The convolution integrates and compresses the features to improve the feature expression ability and accuracy of the model.

[0023] In step S7, the test and results are compared: the global prediction map is compared with the real data to obtain the accuracy of the corresponding segmentation map; this method uses five metrics: Dice, IoU, Recall, Precision and F2; the test results on the Kvasir dataset are as follows Figure 5As shown in the table, the data demonstrates the excellent learning ability of the EGANet model. As can be seen from the table, the Dice coefficient of EGANet is 5.6% higher than that of the classic PraNet, and even 0.6% higher than the latest best-performing DUCK-Net. Although the recall rate is slightly lower than that of CFHA-Net, the precision is 1.7% higher than that of CFHA-Net. Overall, the design of the EGANet model is feasible. The test results on the CVC-ClinicDB dataset are shown in the figure below. Figure 6 As shown in the table, it can be seen that the Dice coefficient of EGANet is 2.1% higher than that of the classic PraNet, and even compared with the latest and best-performing PolypSegNet, there is an improvement of 0.4%; although the recall rate is slightly lower than that of PolypSegNet by about 0.1%, the accuracy is 1.1% higher than that of PolypSegNet; based on the performance of the EGANet model on the CVC-ClinicDB and Kvasir datasets, the EGANet designed in this invention has excellent learning ability. The model can learn appropriate feature representation and parameter adjustment from the training data to adapt to different data distributions and task requirements; in order to test the generalization ability of the EGANet model, three datasets that have not been trained are used: CVC-ColonDB, CVC-T and ETIS-LaribPolypDB; compared with the models in recent years, the measurement indicators are average Dice and average IoU; the results are as follows Figure 7As shown in the figure, the average Dice coefficient and average IoU coefficient of the EGANet model are significantly better than those of existing models: on the Kvasir dataset, the average Dice coefficient of the EGANet model is 4.0% higher than the classic PraNet model, and is about 0.8% better than the latest and best-performing PFD-Net; on the CVC-ClinicDB dataset, the average Dice coefficient of the EGANet model is 2.3% better than the classic CaraNet model and only 0.3% less than the best-performing DuAF model; it also shows good generalization ability on three untrained datasets; on the CVC-ColonDB dataset, the EGANet model has an average Dice of 1.2% higher than the best RFPA model and 13.6% higher than the classic PraNet model. The average Dice of , this figure well illustrates the feasibility of the EGANet model design; the performance on CVC-T is even better, 1.7% higher than the best PFD-Net, reaching the highest accuracy of the existing polyp image segmentation models at this stage, and also proving that the EGANet model can well realize the segmentation of small objects; on the ETIS-LaribPolypDB dataset, where the segmentation accuracy is generally low, very good segmentation results are also obtained, which is 0.5% higher than the best PFD-Net; this also illustrates the excellent generalization ability of the EGANet model. In summary, the EGANet model is very good in both learning ability and generalization ability compared with the new models and classic models proposed in the past two years, especially the average Dice of 0.928 on CVC-T.

[0024] The above content is a further detailed description of the present invention in combination with the preferred technical solution, and it cannot be determined that the specific implementation of the invention is limited to these descriptions; for ordinary technicians in the technical field to which the present invention belongs, simple deduction and replacement without departing from the concept of the present invention should be regarded as within the scope of protection of the present invention.

Claims

1. An image segmentation method based on edge-to-global feature fusion network. Characterized by: Collect the polyp image dataset that needs to be segmented; First, the image dataset is preprocessed, including specular highlight removal and color shift operations; The preprocessed image is input into the ResNet50-SPD backbone network for encoding; The output features of the third, fourth, and fifth layers in the encoder are respectively input into the global attention module (GA) and the edge attention module (EA) for feature extraction; The output features of the second layer are extracted through the dual-head attention module (DA) to extract detailed features and input into the edge attention module (EA) to achieve edge information guidance; The fused features output by the global attention module (GA) and the edge attention module (EA) are input into three cascaded decoder modules for multi-stage feature reconstruction; The final segmentation prediction map is generated by the decoder and the accuracy is evaluated with the true annotation.

2. The image segmentation method based on edge-to-global feature fusion network according to claim 1, characterized in that: For data preprocessing, we first need to remove the highlight part of the image and then perform color shift on the image. The steps to remove the highlight area are as follows: First, grayscale the image to obtain the average grayscale value K after grayscale; then multiply K by the weight W (the default weight is 0.3) to obtain the threshold T; then subtract the grayscale of the original image from the threshold T to obtain the value P, and P greater than 0.75 is considered a highlight; then, set the value greater than T to 1 and the value less than T to 0 to obtain a binary mask to generate a highlight location map; Finally, the illuminationChange function was used to remove highlights and reconstruct the image. After obtaining the highlight-removed image, color shift (ColorJetter function) was used to randomly shift the image's contrast, brightness, saturation, and hue. This color shift not only prevented the network model from focusing excessively on the color information of the polyps during training but also magnified the detailed differences between the polyps and surrounding normal tissue, improving segmentation accuracy. It is worth noting that image preprocessing was only performed during the training phase.

3. The polyp image segmentation method based on edge-to-global feature fusion network according to claim 1, characterized in that: ResNet50-SPD is introduced as the encoder to make the network retain as much discriminative feature information as possible; the decoder structure is set as: a The convolution adjusts the relationship between the number of channels in the feature map, followed by two deconvolutions to reconstruct the feature map, and finally a The convolution integrates and compresses the features to improve the feature expression ability and accuracy of the model.

4. The image segmentation method based on edge-to-global feature fusion network according to claim 1, characterized in that: Structural design of edge attention module: First, the feature map encoded by ResNet50-SPD high-level feature layer With the downsampled edge feature map ; Perform feature multiplication and then compare the multiplied feature map with ; Add together to get multi-scale information The feature map is then Convolution obtains feature maps Secondly, The pooling size is The global average pooling and maximum pooling of The channel description row vectors Favg and Fmax share two fully connected layers, then perform channel convolution, and use the Relu activation function to fit the complex correlation between channels; then add the two channel description vectors obtained and get the Sigmod function The feature weight vector ,Will Perform a full multiplication operation with the weight vector to obtain feature maps with different importance of different channels.

5. The image segmentation method based on edge-to-global feature fusion network according to claim 1, characterized in that: Structural design of the global attention module: First, the features output by the encoder As the input feature of the GA module, the size of its feature map is ( ),Will conduct The channel convolution will The channel characteristics are adjusted to ( ); The module is then divided into two parts. The upper branch uses the non-local attention module (Non-Local) to capture the long-range spatial dependencies of features and accurately extract the global dependencies of each position, thereby obtaining global context information. It is then transferred to the attention module CA to capture the explicit relationship between feature channels. The lower branch adopts three different expansion rates The dilated convolution is used to obtain the surrounding context information under different receptive fields, and then the channel attention module CA is used to capture the local context information at different scales; Secondly, the non-local attention module and the dilated convolution module are connected and Finally, it is combined with the convolution and BN layers of Original features of convolution , add them together, get the fused features, and then use the activation function on them. Finally, the spatial attention module SA is used to suppress irrelevant areas in the feature map.

6. As claimed in claim 1, an image segmentation method based on edge-to-global feature fusion network, characterized in that: Ten comparison models, including U-Net, PraNet, TGANet, MKDCNet, PolypSegNet, CFHA-Net, ECTransNet, DUCK-Net, AFA-Net, and PFD-Net, were trained and tested on five datasets, namely Kvasir, CVC-ClinicDB, ETIS-LaribPolypDB, CVC-ColonDB, and CVC-T. The accuracy was obtained by comparing the predicted results with the real data.

Images