Edge enhancement multi-scale feature fusion network based on neural network
Through edge enhancement of multi-scale feature fusion network based on neural network, the problems of polyp boundary complexity and detail segmentation difficulty in medical image segmentation are solved, and more accurate polyp edge extraction and segmentation effects are achieved.
Patent Information
- Application Number
- CN202411571387.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Medical image segmentation, especially polyp segmentation, faces complex boundary diversity and difficulty in segmenting details, resulting in fuzzy boundaries and precise extraction difficulties.
Adopt edge-enhanced multi-scale feature fusion network based on neural networks, combining pre-trained Res2Net-50 backbone network, shallow feature fusion module and edge-enhanced multi-scale feature fusion module to capture images through spatial channel attention mechanism and decoder Edge, texture and color features enhance boundary information and reduce the difficulty of detail segmentation.
The precise extraction ability of polyp boundaries is improved, the model's ability to capture features of different scales is enhanced, and the robustness and accuracy of the segmentation algorithm is improved.
Smart Images

Figure CN119090903A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to an edge enhancement multi-scale feature fusion network based on a neural network. Background Art
[0002] Image segmentation is a fundamental technology in the field of computer vision. It is the process of dividing an image into multiple regions or objects. Image segmentation is particularly important in medicine because it can help doctors and researchers identify and quantify lesions, organs, and other biomarkers. It is commonly used in X-ray, CT, MRI, and ultrasound images, which place higher demands on segmentation algorithms due to their complexity and high noise characteristics.
[0003] However, colonoscopy requires professional doctors to perform detection and segmentation, which is very time-consuming. However, manual colonoscopy is limited by the operator's status and experience, and polyps may be missed in video detection. Therefore, using machine deep learning to assist clinicians in polyp detection and segmentation is an effective solution. Since the size, shape, color and texture of polyps themselves are quite variable, and the boundary with the background intestinal wall is blurred, these complex situations make polyp segmentation very challenging.
[0004] In recent years, with the development of deep learning technology, the field of medical image segmentation has made significant progress. Convolutional neural networks (CNNs) have become the model of choice for many medical image segmentation tasks because they can automatically learn image features and effectively perform regional segmentation. In particular, in polyp segmentation, researchers have used the powerful ability of CNNs to identify subtle features of polyps, improving the accuracy and efficiency of segmentation. In addition, researchers have explored a variety of data augmentation and regularization techniques to improve the generalization ability of the model, as well as the use of multimodal data fusion to combine the advantages of different imaging technologies. These methods help to improve the robustness of segmentation algorithms in different patients and under different imaging conditions.
[0005] Despite these advances, medical image segmentation, especially polyp segmentation, still faces many challenges. The boundaries of polyps exist in mucosa with different resolutions, textures, states, and colors, which leads to the complex diversity of polyp boundaries. This diversity, coupled with downgraded decoding, poses challenges to the segmentation of detailed parts of the boundaries during the decoding process, including the difficulty of segmenting the detailed parts of the boundaries during the decoding process, which also leads to the formation of blurred polyp boundaries and makes it difficult to accurately extract the edges of polyps. For this reason, an edge-enhanced multi-scale feature fusion network based on neural networks is urgently needed. Summary of the invention
[0006] To this end, the present invention provides an edge-enhanced multi-scale feature fusion network based on a neural network to solve the problems raised in the background technology.
[0007] In order to achieve the above object, the present invention provides the following technical solutions: an edge-enhanced multi-scale feature fusion network based on a neural network, the network comprising a pre-trained backbone network Res2Net-50, a shallow feature fusion module and an edge-enhanced multi-scale feature fusion module; The shallow feature fusion module (SFF) uses the backbone network Resnet-50 to divide the input polyp images with blurred appearance, different sizes, and unclear boundaries into 5 feature layers, and uses the spatial channel attention mechanism to integrate the shallow features f 1 , f 2 After convolution, the feature map is obtained and the edges, textures, and colors in the image are captured. f 1 ’ and f 2 ’ , and then use the shallow feature fusion module to fuse the two to obtain the fused shallow feature sf 1 , thus obtaining a polyp image containing more feature and texture information; Edge-enhanced multi-scale feature fusion module, which combines high-level features { f i , i =3,4,5} three high-level feature layers and fused shallow features sf 1 , after the decoder PD, all features are aggregated through parallel connections. The decoder consists of S g =PD( sf 1 , f 3 , f 4 , f 5 ) to calculate and obtain the global mapping S g , which can capture the approximate location of the polyp tissue and the polyp image containing features and textures; global map S g It is mainly responsible for capturing the approximate location of the polyp tissue, and then through deep supervision, constraining the network at different scales to make it gain more attention on the boundary, thereby obtaining a polyp image with enhanced boundaries; finally, through the Sigmoid function, the final polyp image is obtained.
[0008] Res2Net is a convolutional neural network (CNN) architecture that enhances the network's multi-scale feature representation capabilities by building hierarchical residual-like connections within a single residual block. This design allows the network to capture features of different scales at a more fine-grained level, thereby improving the model's generalization capabilities for various visual tasks.
[0009] The core idea of Res2Net is to introduce an additional layer structure in the traditional residual block. In the standard residual network (ResNet), a residual block usually contains two paths: one is the shortcut of the identity connection (skipconnection), and the other is the convolutional layer.
[0010] Preferably, an additional hierarchical structure is introduced into the traditional residual block of Res2Net, in which the 3x3 convolution layer is replaced by a set of 3x3 convolution kernels, which are connected in a hierarchical manner to form multiple parallel sub-paths, and each sub-path captures features of different scales. Finally, these features are fused through 1x1 convolution, and the Res2Net-50 with the fused hierarchical structure is used as the pre-trained backbone network Res2Net-50.
[0011] Preferably, the spatial channel attention mechanism first separately f 1 , f 2 The Squeeze operation is performed on the feature map to perform global average pooling (Global AvgPool) to aggregate global information and obtain a 1*1*C feature vector; then the two FC layers form a Bottleneck structure to model the correlation between channels. The specific steps are as follows: (1) The feature map is transmitted through the first FC layer, compressing the number of channels C to C / 16, reducing the number of channels and extracting a more abstract polyp feature representation; then, through the nonlinear activation function of ReLU, the network's expressive power is increased, capturing more detailed feature differences, and reaching the second FC layer; (2) The feature map is transmitted to the second FC layer, and then the number of feature channels is restored from C / 16 to C, so as to learn a more complex polyp feature representation and obtain the weight parameters containing the attention mechanism; Finally, after the Sigmoid activation function, the normalized weights are weighted to the features of each channel through the Scale operation, thereby enhancing useful features and suppressing less important features. After the SE module is completed, the features from the shallow layer are f 1 , f 2 The features of are weighted and added element by element to generate the resulting fused shallow features sf 1 At this time, a polyp image with relatively clear boundaries and relatively obvious textures is obtained.
[0012] The present invention has the following advantages: Inspired by remote sensing detection, the present invention enhances the edge information of polyp boundaries. The network can learn to use global information to selectively emphasize information-rich features and suppress less useful features, thereby achieving accurate extraction of polyp edges in shallow features. Since low-level features often have greater resolution and computing resources than high-level features in feature extraction, in order to obtain more boundary information, low-level features are aggregated through a shallow feature fusion module, thereby proposing an edge enhancement multi-scale feature fusion module, which reduces the difficulty of segmenting the detailed parts of the boundary during the decoding process and strengthens the formation of polyp boundaries. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 Schematic diagram of the neural network-based edge enhancement multi-scale feature fusion network framework provided by the present invention.
[0014] Figure 2 This is a schematic diagram of the SFF module structure provided by the present invention. DETAILED DESCRIPTION
[0015] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0016] like Figure 2 As shown, this embodiment proposes an edge-enhanced multi-scale feature fusion network based on a neural network, which includes a pre-trained backbone network Res2Net-50, a shallow feature fusion module, and an edge-enhanced multi-scale feature fusion module; The shallow feature fusion module (SFF) uses the backbone network Resnet-50 to divide the polyp image input into five feature layers, and uses the spatial channel attention mechanism to integrate the shallow features. f 1 , f 2 After convolution, we get f 1 ’ and f 2 ’ , and then use the shallow feature fusion module to fuse the two to obtain the fused shallow feature sf 1 ; Taking a polyp image I of size h×w×c (where h represents height, w represents width, and c represents the number of channels) as input, we can extract a polyp image with a resolution of The five levels of features f i , i =1,...,5}.
[0017] This embodiment introduces an edge-enhanced multi-scale feature fusion module to aggregate features through parallel connections.
[0018] Edge-enhanced multi-scale feature fusion module, which combines high-level features { f i, i =3,4,5} three high-level feature layers and fused shallow features sf 1 , after the decoder PD, all features are aggregated through parallel connections. The decoder consists of S g =PD( sf 1 , f 3 , f 4 , f 5 ) to calculate and obtain the global mapping S g ; Then through deep supervision, the network is constrained at different scales to make it pay more attention to the boundaries; finally, after the Sigmoid function, the polyp image with enhanced boundaries is obtained.
[0019] Res2Net is a convolutional neural network (CNN) architecture that enhances the network's multi-scale feature representation capabilities by building hierarchical residual-like connections within a single residual block. This design allows the network to capture features of different scales at a more fine-grained level, thereby improving the model's generalization capabilities for various visual tasks.
[0020] The core idea of Res2Net is to introduce an additional layer structure in the traditional residual block. In the standard residual network (ResNet), a residual block usually contains two paths: one is the shortcut of the identity connection (skipconnection), and the other is the convolutional layer.
[0021] An additional hierarchical structure is introduced into the traditional residual block of Res2Net. The hierarchical structure replaces the 3x3 convolution layer with a set of 3x3 convolution kernels. These convolution kernels are connected in a hierarchical manner to form multiple parallel sub-paths, and each sub-path captures features of different scales. Finally, these features are fused through 1x1 convolution, and then the Res2Net-50 with the fused hierarchical structure is used as the pre-trained backbone network Res2Net-50.
[0022] The SFF module is composed of two SE modules in a symmetrical manner. The spatial channel attention mechanism firstly f 1 , f 2 The Squeeze operation is performed on the feature map to perform global average pooling (Global AvgPool) to aggregate global information and obtain a 1*1*C feature vector; then the two FC layers form a Bottleneck structure to model the correlation between channels, such as Figure 1 As shown, the specific steps are as follows: (1) After being transmitted through the first FC layer, the number of channels C is compressed to C / 16, reducing the number of parameters, and then it reaches the second FC layer through the nonlinear activation function of ReLU; (2) Transmit to the second FC layer, and then restore the number of feature channels from C / 16 to C to obtain the weight parameters containing the attention mechanism; Finally, the Sigmoid activation function is used, and then the Scale operation is used to weight the normalized weights on the features of each channel. After the SE module is completed, the shallow features are f 1 , f 2 The features of are weighted and added element by element to generate the resulting fused shallow features sf 1 .
[0023] The training set of this embodiment consists of 900 images of Kvasir-SEG and 550 images of CVC-ClinicDB. The images of the Kvasir and CVC-ClinicDB datasets are randomly divided into three parts, 80% for training, 10% for verification, and 10% for testing.
[0024] The above network is modeled through various deep learning frameworks and tools. In order to verify the effectiveness of the model in this case, this embodiment conducts comparative experiments on seven benchmark colonoscopy datasets. The specific situation of each dataset is as follows: Kvasir: This dataset contains 1,000 polyp images and their corresponding GroundTruth from colonoscopy video sequences. The Ground Truth is manually annotated by medical doctors and then verified by experienced gastroenterologists. The resolution of the images ranges from 332×487 to 1920×1072 pixels.
[0025] CVC-ClinicDB: This dataset contains 612 images collected from 29 colonoscopy video sequences with a resolution of 288 × 384. The images contain multiple examples of polyps, and masks corresponding to the areas covered by polyps in the images are provided to support image segmentation tasks.
[0026] CVC-ColonDB: This dataset consists of 380 polyp images and their corresponding ground truths with a resolution of 500 × 570. The images were generated from 15 different videos, and experts ensured that the extracted frames showed significantly different perspectives by excluding similar frames and manually annotated the 380 frames.
[0027] ETIS: This dataset contains 196 polyp images extracted from 34 colonoscopy videos with a resolution of 966 × 1225. The dataset includes variations in the size and shape of the coral polyps, blurred boundaries, and complex backgrounds, making the segmentation task extremely challenging.
[0028] CVC-300: This dataset is a test dataset from EndoScene, including 60 polyp images with a resolution of 500 × 574. This example selects CVC-300 as the test set, which contains a total of 60 samples for evaluation.
[0029] NICE_1 and NICE_2: These two datasets come from Hospital B affiliated to University A and Hospital C of Traditional Chinese Medicine, respectively, and include 1,459 intestinal polyp images, of which dataset 1 contains 467 serrated lesion polyps and dataset 2 contains 992 adenomatous polyps, which are used for external testing of the model. These datasets include images of different sizes. The external test set is specifically used to evaluate the performance of the model. It is not used in the training and parameter adjustment process to ensure the objectivity and independence of its evaluation results. In order to improve the adaptability and stability of the model to different situations, this embodiment selects endoscopic images with different intestinal cleanliness, includes patients who underwent colonoscopy within the specified period, and all polyps found have been removed and histopathologically analyzed, excluding those with inflammatory bowel disease, active colitis, coagulopathy, familial polyposis, and those who undergo emergency colonoscopy.
[0030] In this embodiment, in order to comprehensively evaluate the model performance, a variety of quantitative evaluation indicators are used. First, this embodiment uses two basic indicators, average Dice and average IoU. In addition, four metrics widely used in the field of object detection are further introduced, including weighted Dice metric , which corrects the problem of equal importance of all pixels in the traditional Dice metric; Mean absolute error (MAE), used to evaluate pixel-level accuracy, the lower the value, the better; Enhanced alignment metric , used to evaluate pixel-level and global-level similarity; and the structural metric Sα, used to measure the structural similarity between the predicted result and the true label. This embodiment follows the existing state-of-the-art (SOTA) method and uses these six metrics - mDice, mIoU, Sα, and MAE to evaluate the performance of the method in this embodiment and facilitate comparative analysis with other existing technologies. These metrics have been applied in a large number of literatures and have been proven to be effective in polyp segmentation tasks. Through these comprehensive evaluations, this embodiment can fully understand the performance of the system relative to the ground truth labels and make effective comparisons with other technologies.
[0031] In the data sets NICE_1 and NICE_2 of this embodiment, as shown in Tables 1 to 3, this embodiment compares several extremely challenging segmentations including small polyps, large polyps and multiple polyps, and the model performance of this embodiment is still outstanding, wherein Ours in Tables 1 to 3 is the algorithm model of the present invention.
[0032] In this embodiment, six advanced algorithms, U-Net, U-Net++, ResUnet++, SFA, Pranet and SANet, are used as controls for testing on five open source datasets, Kvasir, CVC-ClinicDB, CVC-300, ETIS, CVC-ColonDB, and two private datasets, NICE_1 and NICE_2. From the data results in Tables 1 to 3, it can be seen that this embodiment exceeds the existing algorithms in various indicators such as accuracy.
[0033] This is because the SFF module and edge-enhanced multi-scale feature fusion module proposed in this embodiment can effectively enhance the edge information of the polyp image, thereby capturing more edge features.
[0034] Table 1. Performance comparison of test datasets on Kvasir and CVC-ClinicDB
[0035] Table 2. Performance comparison of test datasets on CVC-300, ETIS, and CVC-ColonDB
[0036] Table 3. Performance comparison of the test dataset on the private datasets NICE_1 and NICE_2
[0037] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.
Claims
1. Edge-enhanced multi-scale feature fusion network based on neural network, characterized by: The network includes a pre-trained backbone network Res2Net-50, a shallow feature fusion module, and an edge-enhanced multi-scale feature fusion module; The shallow feature fusion module uses the pre-trained backbone network Resnet-50 to divide the input polyp images with blurred appearance, different sizes, and unclear boundaries into 5 feature layers. fi , i =1,2} is classified as a shallow feature, and the shallow feature is focused using the spatial channel attention mechanism. f 1 , f 2 After convolution, the feature map is obtained and the edges, textures, and colors in the image are captured, and the shallow features after convolution are obtained. f 1 ’ and f 2 ’ ; Then use the shallow feature fusion module to convolute the shallow features f 1 ’ and f 2 ’ Fusion obtains fusion shallow features sf 1 , obtain a polyp image containing feature and texture information; Edge-enhanced multi-scale feature fusion module, which combines high-level features { fi , i =3,4,5} three high-level feature layers and fused shallow features sf 1 , after the decoder PD, all features are aggregated through parallel connections. The decoder consists of S g =PD( sf 1 , f 3 , f 4 , f 5 ) to calculate and obtain the global mapping S g , capturing the location of polyp tissue and polyp images containing features and textures; then through deep supervision, the network is constrained at different scales to make it pay more attention to the boundaries, thereby obtaining a polyp image with enhanced boundaries; finally, after the Sigmoid function, the final polyp image is obtained.
2. The edge-enhanced multi-scale feature fusion network based on neural network according to claim 1, characterized in that: An additional hierarchical structure is introduced into the traditional residual block of Res2Net. The hierarchical structure replaces the 3x3 convolution layer with a group of 3x3 convolution kernels. These convolution kernels are connected in a hierarchical manner to form multiple parallel sub-paths, and each sub-path captures features of different scales. Finally, these features are fused through 1x1 convolution, and then the Res2Net-50 with the fused hierarchical structure is used as the pre-trained backbone network Res2Net-50.
3. The edge-enhanced multi-scale feature fusion network based on neural network according to claim 1, characterized in that: The spatial channel attention mechanism first separately f 1 , f 2 The Squeeze operation is performed to perform global average pooling on the feature map to aggregate global information and obtain a 1*1*C feature vector; then two FC layers form a Bottleneck structure to model the correlation between channels. The specific steps are as follows: (1) The feature map is transmitted through the first FC layer, where the number of channels C is compressed to C / 16, and the abstract polyp feature representation is extracted. Then, the feature differences of the details are captured through the nonlinear activation function of ReLU, and the feature map reaches the second FC layer. (2) The feature map is transmitted to the second FC layer, and then the number of feature channels is restored from C / 16 to C, so as to learn the complex polyp feature representation and obtain the weight parameters containing the attention mechanism; Finally, the Sigmoid activation function is used, and then the Scale operation is used to weight the normalized weights on the features of each channel. After the SE module is completed, the shallow features are f 1 , f 2 The features of are weighted and added element by element to generate the resulting fused shallow features sf 1 , that is, a polyp image with clear boundaries and obvious texture is obtained.
Citation Information
Patent Citations
Road image multi-scale edge detection model and method based on residual network
CN112085756A
Medical image processing method and device, electronic equipment and storage medium
CN113506310A
Polyp image segmentation method based on superficial layer supervision and attention feedback
CN115393283A
Pathological image segmentation network model, method, device and medium
CN117953341A
Method and system for adaptive feature fusion in convolutional neural network
WO2023077998A1
Cited By
Medical image segmentation method based on edge feature capture and multi-scale feature fusion
CN120070477A
