A deep learning and image fusion collaborative learning enhanced colon polyp segmentation method

Abstract

The application belongs to the field of intelligent medical computer-aided diagnosis application, and relates to a colon polyp segmentation method based on deep learning fusion and collaborative learning enhancement. The method comprises a feature extraction model, a fusion module and multi-view collaborative learning. The feature extraction model is divided into two branches. One branch uses DeiT-Small to extract global feature information and establish the correlation between each pixel. The other branch uses HardNet-MSEG to extract local feature information and obtain more low-level detail information. In order to improve the segmentation accuracy of small target colon polyp images, based on the public colon polyp image dataset and the initial deep learning single-branch segmentation method, a deep learning technology fusion method is proposed, and multi-view collaborative learning is used to enhance colon polyp segmentation. Compared with the feature extracted by a single deep learning method before improvement, the feature is richer, and the information omission defect of a single branch is compensated.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical computer-aided diagnosis applications, specifically involving a colon polyp segmentation method based on the fusion of deep learning methods. Technical Background

[0002] Preventing colon cancer is a global concern.

[0003] Clinical treatment shows a close relationship between colon cancer and colon polyps. Colon polyps are benign (non-cancerous) tissues inside the colon, but if not detected and treated in time, they can cause tissue lesions. It takes about five to ten years for these lesions to develop into colorectal cancer, but most patients are diagnosed at an advanced stage. Because early clinical symptoms of colon cancer are not obvious, as the disease progresses, symptoms such as dizziness, headaches, weight loss, abdominal bloating, and abdominal pain begin to appear, but by this time the disease has generally reached the middle or late stage. Colon polyps are an early sign of colorectal cancer; the earlier colon polyps are detected, the higher the survival rate for colon cancer patients.

[0004] Currently, colonoscopy is the key method for screening for colorectal polyps. Colonoscopy allows observation of the shape, extent, and size of polyps, and allows for the extraction of tissue samples for examination to determine if the polyp has become malignant, facilitating further diagnosis and treatment. However, due to the varying sizes and shapes of colorectal polyps, as well as their low contrast with the background, some polyps may be missed during colonoscopy. Missed polyps can affect the diagnosis, ultimately leading to disease progression and death. Therefore, utilizing computer-aided techniques to improve the detection rate of colorectal polyps is highly significant for the diagnosis and treatment of colorectal cancer.

[0005] Currently, methods for segmenting colon polyps mainly fall into two categories: traditional segmentation methods based on low-level features and deep learning-based segmentation methods. Early traditional segmentation methods primarily relied on shape, color, and texture features to distinguish target and background regions. However, this method requires manual feature extraction by personnel with medical knowledge, thus limiting its application to complex and varied polyp segmentation tasks. With the continuous development of deep learning, some deep learning methods based on convolutional neural networks have begun to be applied to colon polyp segmentation tasks. While this has improved the segmentation accuracy of some colon polyp datasets, its performance remains poor for datasets containing small-diameter polyps. Effectively improving the segmentation accuracy of datasets containing small-diameter polyps remains a significant challenge. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a co-learning-enhanced colon polyp segmentation method that integrates deep learning. Based on the varying sizes of colon polyps and the respective advantages of deep learning methods, this invention proposes a colon polyp segmentation method that integrates deep learning methods, aiming to improve the segmentation accuracy of datasets containing a large number of small colon polyps.

[0007] A collaborative learning-enhanced method for segmenting colonic polyps using deep learning integration includes the following steps:

[0008] Step 1: Standardizing the Resolution of the Training Dataset: The colonoscopy images in the dataset used in this invention have varying resolutions, while the input to the DeiT branch in the Transformer branch needs to be of a fixed size. Furthermore, a large difference in resolution can also have an impact when fusing the feature information extracted from the Transformer branch and the CNN branch. Therefore, this invention standardizes the resolution of the training dataset before model training.

[0009] Step 2: Input the training set into the Transformer branch and the CNN branch respectively.

[0010] Step 3: Transformer Branch: The Transformer branch first preprocesses the input image into a two-dimensional sequence, then inputs it into DeiT-small for global feature extraction, and finally processes the output sequence T of DeiT-small. l Remodeled into d model 2D feature image of the channel Finally, feature map T1 is obtained by performing two progressive upsampling operations (alternating between convolution and upsampling).

[0011]

[0012] Step 4: CNN Branch: In the CNN branch, the input image is first fed into the HardNet-MSEG network to extract local feature information. Then, the output U of HardNet-MSEG is processed... l Downsampling is performed to obtain U0, and then U0 is upsampled by 2x, convolution, BatchNorm, and ReLU operations are performed to obtain... Next, U2 is upsampled and then downsampled to obtain... Similarly, update U0 based on U1.

[0013]

[0014] Step 5: Fusion Module: This module is designed to fully integrate the feature information extracted from the Transformer branch and the CNN branch to make up for the defects of missing information.

[0015] 5.1 Global-Local Fusion Submodule: This module integrates the feature maps T extracted from the Transformer branch. i Feature maps U extracted from CNN branches i The input is fed into the global-local fusion submodule to fuse feature maps from the same scale from the two branches, and the final output F is... i This represents the fused feature map.

[0016] 5.2 Dense Fusion Submodule: This module combines the output F of the global-local fusion submodule. i The input is fed into the dense fusion submodule for multi-scale feature fusion, with the aim of obtaining a more comprehensive feature representation.

[0017] Step 6: Multi-view collaborative training: This invention includes Transformer branch, CNN branch, and fusion branch. Each branch has a predicted output graph. If the weights of the branches are given by humans or each branch is treated equally, the entire process will obviously not achieve the best performance of the network. Therefore, this invention regards the Transformer branch, CNN branch, and fusion branch as three perspectives and uses a multi-view collaborative objective function for training to achieve adaptive evaluation of the weights of each perspective.

[0018]

[0019]

[0020] loss k =loss(Pre k ,G)

[0021] Where w represents the viewpoint weight coefficient, Θ represents the model parameters, and loss k The loss represents the difference between the predicted value and the true label for the k-th viewpoint, where λ represents the hyperparameter. k G represents the predicted output from the k-th viewpoint, and G represents the true label.

[0022] Step 7: Comprehensive Decision Making: In the network structure of this invention, three perspectives output three prediction maps. Choosing any one of them as the final result is unreliable. Since this invention saves the weight of each branch during the training process, the weight of each perspective is multiplied with its respective prediction map and then added together to obtain the final prediction map, thereby achieving comprehensive decision making and hoping to achieve the best segmentation effect.

[0023]

[0024] Where Out represents the output result after comprehensive decision-making, w k This represents the weight of the k-th viewpoint.

[0025] Furthermore, in step 6, the optimal value of the λ hyperparameter can be obtained using cross-validation or other strategies.

[0026] Furthermore, the multi-view training used in step seven treats the Transformer branch, CNN branch, and fusion branch as three views, employing a multi-view collaborative objective function to adaptively evaluate the importance of each view. The objective function is solved using a cross-iterative approach, fixing the network parameters Θ, and utilizing... Solve for w k In this invention, λ is set to 1. w is fixed. k The Adam optimizer is used to update the model parameters Θ. k The weights for each k-th viewpoint are given, and the sum of the weights for the three viewpoints is required to be 1.

[0027] The advantages of this invention include the following:

[0028] (1) Model Structure Design: Existing methods for segmenting colon polyps include those based on convolutional neural networks and those based on Transformer methods. Although some existing methods are effective in segmenting colon polyps, they also have certain drawbacks. This invention uses a combination of Transformer and CNN to take into account the global feature learning ability of Transformer networks and the local feature learning ability of CNN networks, and can achieve information complementarity from different aspects.

[0029] (2) Construction of the feature fusion module: To fuse the feature information obtained from the two deep learning techniques, a global-local feature fusion (GLFF) module was first designed to fuse feature maps of multiple scales obtained from the two deep learning branches, reducing the loss of location and detail information. Then, a dense fusion of multi-scale features (DFM) module was designed to densely fuse the fused feature maps of different scales, fully combining the feature information of high-level and low-level images to enhance information representation capabilities.

[0030] (3) Construction of multi-view objective function: By treating the CNN branch, Transformer branch, and fusion branch as three perspectives, a multi-view collaborative learning training method is proposed to optimize model parameters. This invention uses a multi-view learning weighting mechanism to automatically obtain the weight coefficients of each perspective. By utilizing the trained model parameters and the weight coefficients of each perspective, the final multi-view collaborative decision can be achieved.

[0031] (4) Through sufficient experiments, it is demonstrated that the proposed Fu-TransHNet model has significantly improved the segmentation accuracy on datasets containing many small targets compared with existing related methods, and has also achieved significant performance improvements on several other public benchmark datasets. Attached Figure Description

[0032] Figure 1 This is a diagram of the overall algorithm framework in Example 1;

[0033] Figure 2 This is a diagram illustrating the preprocessing framework for colonoscopy images in Example 1.

[0034] Figure 3a This is a flowchart of the Transformer branching process;

[0035] Figure 3b This is a DeiT-small structure diagram;

[0036] Figure 4a It is a global-local fusion submodule;

[0037] Figure 4b It is a densely integrated submodule;

[0038] Figure 5 This is a framework diagram of the fusion module in Example 1;

[0039] Figure 6 This is a comparison chart of the prediction graph in Example 1 and prediction graphs from other methods;

[0040] Figure 7 This is the overall framework diagram without the fusion module in Example 2;

[0041] Figure 8 This is an overall framework diagram of Example 2, which includes a single Dense Fusion Submodule (DFM).

[0042] Figure 9 This is the overall framework diagram of Example 2, which includes a single Global-Local Information Fusion (GLFF) submodule. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and embodiments:

[0044] like Figures 1-5As shown, this invention implements a co-learning enhancement method for colon polyp segmentation using deep learning fusion. This method comprises three parts: a Transformer model and a CNN model to extract colon polyp feature information, an information fusion module, and multi-view training. The two deep learning methods used in this invention are: first, DeiT-small to obtain global feature information of colon polyps, achieving correlation between arbitrary pixels; and second, HardNet-MSEG to obtain local feature information of colon polyps, achieving short-distance feature dependencies. The fusion module integrates the feature maps from the two branches, allowing the feature information obtained by the two methods to complement each other, reducing the omission of information about small targets, thus helping to improve the segmentation accuracy of colon polyps with small diameters. Multi-view training utilizes a Shannon entropy-based multi-view weighting approach to adaptively evaluate the importance of each viewpoint. Adaptively determining the weight of each viewpoint is more reasonable and effective than directly assuming each viewpoint is equally important or manually assigning weights, and is expected to achieve optimal network model performance.

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] Example 1:

[0047] A method for segmenting colonic polyps using a deep learning-integrated, collaborative learning-enhanced approach, characterized by the following steps:

[0048] (1) Standardized resolution of training dataset: The colonoscopy images in the dataset used in this invention have different resolutions, while the input of DeiT in the Transformer branch needs to be of a fixed size. Furthermore, the large difference in resolution when fusing the feature information extracted by the Transformer branch and the CNN branch can also have a certain impact. Therefore, this invention standardizes the resolution of the training dataset to 352×352 before model training.

[0049] (2) Input the training set into the Transformer branch and the CNN branch respectively. The Transformer branch and the CNN branch are shown in Figure 3-4.

[0050] (3) Transformer branch: The Transformer branch first preprocesses the input image into a two-dimensional sequence, then inputs it into DeiT-small for global feature extraction, and then processes the output sequence T of DeiT-small. l Remodeled into d modelThe 2D feature image T0 (22×22×384) of the channel is finally obtained by performing two progressive upsampling operations (alternating between convolution and upsampling operations) to obtain feature maps T1 (44×44×128) and T2 (88×88×64).

[0051] (4) CNN Branch: In the CNN branch, the input image is first fed into the HardNet-MSEG network to extract local feature information, and then the output U of HardNet-MSEG is processed. l We downsample to obtain U0, then upsample U0 by 2x, convolution, BatchNorm, and ReLU operations to obtain U2 (88×88×64), then upsample and downsample U2 to obtain U1 (44×44×128), and similarly update U0 (22×22×256) based on U1.

[0052] (5) Integration module, such as Figure 5 As shown: This module is designed to fully integrate the feature information extracted by the Transformer branch and the CNN branch, and make up for the defects of missing information.

[0053] 1) Global-Local Fusion Submodule: This module integrates the feature maps T extracted from the Transformer branch. i Feature maps U extracted from CNN branches i The input is fed into the global-local fusion submodule to fuse feature maps from the same scale from the two branches, and the final output F is... i This represents the fused feature map.

[0054] 2) Dense Fusion Submodule: This module combines the output F of the global-local fusion submodule. i The input is fed into the dense fusion submodule for multi-scale feature fusion, with the aim of obtaining a more comprehensive feature representation.

[0055] (6) Multi-view collaborative training: This invention includes Transformer branch, CNN branch, and fusion branch. Each branch has a predicted output graph. If the weights of the branches are given by humans or each branch is treated equally, the entire process will obviously not achieve the best performance of the network. Therefore, this invention regards the Transformer branch, CNN branch, and fusion branch as three perspectives and uses a multi-view collaborative objective function for training to achieve adaptive evaluation of the weights of each perspective.

[0056]

[0057]

[0058] loss k =loss(Pre k,G)

[0059] Where w represents the viewpoint weight coefficient, Θ represents the model parameters, and loss k The value represents the loss between the predicted value and the true label at the k-th viewpoint, where λ represents the hyperparameter set to 1 in this invention. k G represents the predicted output from the k-th viewpoint, and G represents the true label.

[0060] (7) Comprehensive decision-making: In the network structure of this invention, three prediction maps are output from three perspectives. Choosing any one of them as the final result is unreliable. Since this invention saves the weight of each branch during the training process, the weight of each perspective is multiplied with its respective prediction map and then added together to obtain the final prediction map, thereby achieving comprehensive decision-making and hoping to achieve the best segmentation effect.

[0061]

[0062] Where Out represents the output result after comprehensive decision-making, w k This represents the weight of the k-th viewpoint.

[0063] In segmenting colonic polyps, this invention first trains Fu-TransHNet on the Kvasir and CVC-ClinicDB datasets to enable the model to acquire a certain learning ability. Then, a test set is used to test and verify the model's learning ability. The datasets used in Example 1 are shown in Table 1 (test set).

[0064] Table 1: Test set information used in Example 1:

[0065]

[0066] Table 2 summarizes the segmentation results of this invention on the test dataset of Example 1. This invention uses two commonly used evaluation metrics for semantic segmentation: average Dice coefficient (mDice) and average intersection-union ratio (mIoU). To analyze the model more deeply, four commonly used evaluation metrics in the field of object detection are introduced: weighted F-measure. S-measure(S α ), max-E-measure The mean absolute error (MAE) and other metrics show that this algorithm achieved excellent segmentation results on five public datasets.

[0067] Table 2: Performance metrics of the present invention on various datasets in Example 1

[0068]

[0069] Table 3-7 summarizes the research results on colon polyp segmentation in recent years. All studies used the same training dataset, but the experimental methods differed. The table details the specific methods of each experiment. The results show that our invention achieved good segmentation accuracy on all five publicly available datasets. Our segmentation accuracy was higher than other algorithms on the first four test sets, especially on the CVC-ColonDB dataset, where our algorithm achieved an accuracy of 81%, while other algorithms did not exceed 80%. Furthermore, on datasets containing many small-diameter colon polyps, our algorithm achieved a segmentation accuracy of 79.3%, a 12.7% improvement over the previous HardNet-MSEG and a 6.5% improvement over TransFuse-s.

[0070] Table 3: Experimental results of different methods on the CVC-ClinicDB dataset

[0071]

[0072]

[0073] Table 4: Experimental results of different methods on the CVC-ColonDB dataset

[0074]

[0075] Table 5: Experimental results of different methods on the CVC-EndoScene dataset

[0076]

[0077]

[0078] Table 6: Experimental results of different methods on the ETIS-LaribpolypDB dataset

[0079]

[0080] Table 7: Experimental results of different methods on the Kvasir dataset

[0081]

[0082]

[0083] Example 2

[0084] To verify that the fusion module is helpful in improving the segmentation accuracy of colon polyps, this invention separately tested the results of experiments without the fusion module (…). Figure 7 ), has a single fusion submodule ( Figure 8 , 9) and possess the entire fusion module ( Figure 1 A comparison was made. Finally, the segmentation results of each method were calculated and plotted in Table 8. Table 8 shows that the method containing the entire fusion module outperforms the other three methods in terms of metrics on all datasets, and the method containing a single fusion sub-module outperforms the method without a fusion module. This proves that the invented fusion module is effective.

[0085] Table 8: The impact of different methods on the results

[0086]

Claims

1. A colon polyp segmentation method enhanced by deep learning and collaborative learning, characterized in that, The steps are as follows: Step 1: unify the resolution of the colonoscopy image training data; Step 2: input the training set into the Transformer branch and the CNN branch respectively; Third step: first, the input image is preprocessed into a two-dimensional sequence in the Transformer branch, then input into DeiT-small to extract global feature information, and finally the output sequence of DeiT-small is reshaped into a 2D feature map of the channel , and then two progressive up-sampling is used to obtain a feature map ; Step 4: In the CNN branch, the input image is first fed into the HardNet-MSEG network to extract local feature information. Then, the output of HardNet-MSEG is processed... Downsampling was performed to obtain Then to Perform 2x upsampling, convolution, BatchNorm, and ReLU operations to obtain Then on Upsampling and then downsampling are performed to obtain Similarly, based on renew ; Step 5: 5.1 Feature maps extracted by the Transformer branch and the CNN branch are input into the global-local fusion sub-module to realize the fusion of feature maps from the same scale of the two branches, and the final output represents the fused feature map; 5.2 The output of the global-local fusion sub-module is input into the dense fusion sub-module for multi-scale feature fusion, aiming to obtain a more comprehensive feature representation. Input into the dense fusion sub-module for multi-scale feature fusion, aiming to obtain a more comprehensive feature representation. Step 6: consider the Transformer branch, the CNN branch and the fusion branch as three perspectives, use a multi-perspective collaborative objective function for training, and realize adaptive evaluation of the weight of each perspective; Step 7: the network structure outputs three prediction maps from the three perspectives, multiplies each prediction map by the weight of the respective perspective, and adds them together as the final prediction map to realize comprehensive decision-making.

2. The deep learning and image fusion collaborative learning enhanced colon polyp segmentation method of claim 1, wherein: The specific operation of the sixth step is as follows: Consider the Transformer branch, the CNN branch and the fusion branch as three perspectives, use a multi-perspective collaborative objective function for training, and realize adaptive evaluation of the weight of each perspective; wherein, represents a view angle weight coefficient, represents a model parameter, represents a loss between a prediction value of the kth view angle and a real label, represents a hyperparameter, represents a predicted output of the kth view angle, represents a real label.

3. The deep learning and image fusion synergistic learning enhanced colon polyp segmentation method of claim 2, wherein: In the sixth step, The optimal values of the hyperparameters are obtained using a cross-validation strategy.

4. The deep learning and image fusion synergistic learning enhanced colon polyp segmentation method of claim 2 or 3, wherein: The seventh step specifically operates as follows: the training process saves the weight of each branch, so the weight of each view is multiplied by the respective prediction map and added as the final prediction map to achieve comprehensive decision-making, which is expected to achieve the best segmentation effect. wherein, represents the output result after the comprehensive decision, represents the weight of the kth view.

5. The deep learning and image fusion synergistic learning enhanced colon polyp segmentation method of claim 4, wherein: The multi-view training used in the seventh step is solved by using cross iteration to solve the target function, and the network parameters are fixed , by using solving , setting to 1, fixing , updating the model parameters by using the Adam optimizer ; is the weight of each k view, and the sum of the weights of the three views is required to be 1.

Images