An enteroscopy image classification, detection and segmentation method combining medical information recovery and momentum contrast
By combining medical information recovery and momentum comparison methods, and using large-scale unlabeled colonoscopy image data for self-supervised pre-training, the problems of insufficient data and poor generalization performance in colonoscopy image analysis are solved, and efficient colonoscopy image classification, detection and segmentation are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-03-13
- Publication Date
- 2026-05-19
AI Technical Summary
Existing deep learning methods lack large-scale labeled data for colonoscopy image analysis, have poor generalization performance, and conventional contrastive learning methods are not suitable for medical images.
By combining medical information recovery and momentum comparison methods, data augmentation and medical information recovery are performed through large-scale acquisition of unlabeled colonoscopy image data. The momentum comparison network is used for self-supervised pre-training to improve the feature extraction capability of the backbone network.
With limited labeled data, the performance of colonoscopy image classification, detection, and segmentation models was improved, the dependence on labeled data was reduced, and the generalization ability of the models was enhanced.
Smart Images
Figure CN116612064B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning, and more specifically, relates to a method for classifying, detecting, and segmenting colonoscopy images by combining medical information recovery and momentum comparison. Background Technology
[0002] Colorectal cancer is a leading cause of cancer-related death in both Eastern and Western countries. Colonoscopy is a crucial clinical tool for diagnosing and completely removing early lesions such as colorectal polyps, thereby reducing the incidence and mortality of colorectal cancer. However, the quality of colonoscopy varies depending on the endoscopist's expertise. Inexperienced doctors, as well as fatigue and lack of concentration from long hours, can easily miss polyps.
[0003] Computer-aided techniques based on deep convolutional neural networks help endoscopists detect polyps during real-time colonoscopies. The performance of these methods is positively correlated with the amount of labeled image data used for training. However, labeling colonoscopy images is a time-consuming and expensive task. Unlike natural images, which can be annotated through crowdsourcing, medical images rely on strong professional knowledge for annotation. This process requires the involvement of specialized clinicians. Therefore, the number of labeled colonoscopy images is always limited. To alleviate this problem, a common practice in the medical field is to first perform supervised pre-training on large labeled natural datasets such as ImageNet and MS COCO, and then fine-tune the model on a medical dataset with limited labels. However, models obtained in this way often struggle to generalize well to images from colonoscopy systems from different manufacturers, as well as polyps of varying morphology, pathological nature, and size.
[0004] Through self-supervised learning, models can acquire good visual representations from large-scale unannotated datasets, benefiting the training process for different downstream tasks and enhancing the model's generalization and performance. Contrastive learning is the most cutting-edge self-supervised learning approach. The basic idea of contrastive learning is that different augmented versions of the same sample image have similar representations, which should differ from different sample images. Thus, positive and negative sample pairs can be constructed by augmenting unlabeled data, minimizing the contrastive loss function to train the backbone network to learn the feature representations of the samples. However, current conventional contrastive learning methods are designed for the characteristics of natural images and are not suitable for medical images. Summary of the Invention
[0005] To address the shortcomings of existing deep learning methods in colonoscopy image analysis, such as the lack of large-scale labeled data and poor generalization performance, this invention proposes a colonoscopy image classification, detection, and segmentation method that combines medical information recovery and momentum contrast analysis. This invention performs medical information recovery on the data augmentation portion of the momentum contrast analysis structure, improving the backbone network's ability to extract colonoscopy image features. Furthermore, this invention uses a pre-trained backbone network based on contrastive learning as a feature extractor for downstream classification, detection, and segmentation tasks, mitigating the impact of insufficient labeled colonoscopy image data on model performance.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for classifying, detecting, and segmenting colonoscopy images by combining medical information recovery and momentum comparison, comprising the following steps:
[0007] Step 1, Collect colonoscopy data
[0008] We selected colonoscopy systems from mainstream manufacturers on the market and collected large-scale images of the inside of the intestines without labeling them, thus creating a pre-training dataset.
[0009] Step 2, Data Augmentation
[0010] A certain number of batch data are sampled, each image is cropped and scaled to a specified size, and then two random appearance data augmentations are performed to obtain appearance-enhanced images. Then, medical information restoration methods are used to locally restore the appearance-enhanced images, and then random spatial data augmentation is performed to obtain positive and negative sample pairs.
[0011] The steps for local restoration of the appearance enhancement image using the medical information restoration method are as follows:
[0012] 1) Set the restoration ratio ε, which is the ratio of the length and width of the restoration region rectangle to the length and width of the appearance enhancement image. The value of ε ranges from 0 to 1.
[0013] 2) Calculate the length and width of the restored region rectangle based on the restoration ratio ε and the length and width of the apparent enhanced image;
[0014] 3) Randomly select the starting point of the restoration region rectangle in the image to determine the position of the restoration region rectangle, and replace the RGB values of the corresponding pixels in the image after random appearance data augmentation with the original RGB values of the image.
[0015] Step 3: Pre-train the feature extraction network
[0016] Set hyperparameters, input all positive and negative sample pairs into the Momentum Contrast (MoCo) network, train the Momentum Contrast network and save the parameters of the backbone network used for feature extraction;
[0017] The hyperparameters include backbone network structure, temperature coefficient τ, momentum coefficient m, queue length, batch size, number of iterations (epoch), optimizer-related parameters, etc.
[0018] Step 4: Train downstream classification, detection, and segmentation models
[0019] The parameters of the pre-trained backbone network in step 3 are used to initialize the backbone network of the classification, detection, and segmentation models, and then a small amount of labeled colonoscopy image data is used to fine-tune the model.
[0020] Step 5: Use the trained model to classify, detect, or segment the new colonoscopy images and evaluate its performance metrics.
[0021] Preferably, the apparent data enhancement method described in step 2 includes one or more of color distortion, random grayscale, and Gaussian blur. The robustness of the algorithm can be enhanced by increasing the richness of positive and negative sample pairs.
[0022] Preferably, the spatial data augmentation method described in step 2 includes one or more of random horizontal flipping and random rotation. The robustness of the algorithm can be enhanced by increasing the richness of positive and negative sample pairs.
[0023] Preferably, the specific implementation of training the momentum contrast network in step 3 is as follows:
[0024] The momentum comparison network consists of a key network and a query network, each comprising an encoding layer and a mapping layer. It also maintains a key queue and updates the encoding and mapping layers of the key network using momentum updates.
[0025] Assume x q and x k For positive sample pairs obtained from the same image in a batch of data after data augmentation, x q Input into the query network, x k Input to the key network; where x q The feature representation h is obtained after querying the network coding layer. q The query q;x is obtained by querying the network mapping layer. k The feature representation h is obtained after the key network encoding layer. k The key k is obtained through a key network mapping layer; the contrastive loss is calculated using key k and query q. The dot product is used to measure the similarity between key k and query q; InfoNCE is used as the contrastive loss function.
[0026]
[0027] in, This represents the contrastive loss function, where q is the query and k is the input.+ k is the key for the corresponding positive sample, and k- is the key for the negative sample. For the current image, its query q has only one key k generated from different augmented versions of the same image. + To form positive sample pairs, there are a total of N (N is the length of the sample queue) k... - To form negative sample pairs, N is the length of the key queue. τ is an adjustable temperature coefficient.
[0028] The training process of the momentum contrast network can be summarized as follows:
[0029] In each training step, the keys of the current batch of image data are added to the key queue, while the most outdated keys are dequeued. During gradient backpropagation, only the parameters of the encoding and mapping layers of the query network are updated, while the gradients of the key network are ignored. Then, the parameters of the encoding and mapping layers of the key network are updated more smoothly using momentum.
[0030] θ k =mθ k +(1-m)θq
[0031] Where, θ k Let θ be the set of parameters for the encoding and mapping layers of the key network. q To query the set of parameters for the coding and mapping layers of the network, m is the momentum coefficient parameter, with a value in the range [0,1).
[0032] After training is complete, the coding layer parameters of the query network are taken as the initialization parameters of the backbone network used for feature extraction in step 4.
[0033] Preferably, the backbone network in step 3 can be any deep convolutional neural network, serving as the encoding layer for the query network and the key network during the training of the momentum contrast network.
[0034] Preferably, the model described in step 4 can be a common baseline, such as ResNet, MobileNet, etc. for classification; Faster RCNN, Mask RCNN, RetinaNet, YOLO, etc. for detection; FCN, UNet, DeepLab, GCNet, PSPNet, etc. for segmentation.
[0035] The beneficial effects of this invention are:
[0036] 1) This invention introduces a self-supervised learning method based on momentum contrast structure into the field of colonoscopy image analysis. Specifically, it uses the MoCo algorithm for self-supervised pre-training of the backbone network of the colonoscopy image analysis baseline. This surpasses the currently popular ImageNet pre-training and fine-tuning paradigm when downstream tasks have limited labeled data.
[0037] 2) This invention combines medical information recovery methods with the MoCo algorithm, improving its self-supervised pre-training performance in the field of colonoscopy images. Specifically, by restoring a portion of the appearance-enhanced image, it balances the need for contrastive learning to enhance image richness with the need for feature extractors to achieve medical realism in the image.
[0038] 3) Only a small amount of labeled colonoscopy data is needed to train the model, and the dependence on labels is low. Attached Figure Description
[0039] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0040] Figure 1 This is a flowchart of a colonoscopy image classification, detection, and segmentation method that combines medical information recovery and momentum comparison, provided in an embodiment of the present invention.
[0041] Figure 2 The classification results of this method and other comparative methods provided in this embodiment of the invention on the SUN Colonoscopy VideoDatabase dataset are shown.
[0042] Figure 3 These are the detection results of the method provided in this embodiment of the invention and other comparative methods on the CVC-VideoClinicDB dataset.
[0043] Figure 4 These are the segmentation results of the method provided in this embodiment of the invention and other comparative methods on the Kvasir-SEG dataset. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be further described and illustrated below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] This invention provides an embodiment: a method for classifying, detecting, and segmenting colonoscopy images by combining medical information recovery and momentum comparison. For example... Figure 1 As shown, the embodiments of the present invention mainly include the following steps:
[0046] Step 1: Collect colonoscopy data. Select a colonoscopy system from a mainstream manufacturer on the market and collect large-scale images of the inside of the intestines without annotation to form a pre-training dataset.
[0047] Step 2, Data Augmentation. A certain number of batch data are sampled, each image is cropped and scaled to a specified size, then random appearance data augmentation is performed twice, followed by medical information recovery methods, and then random spatial data augmentation is performed to obtain positive and negative sample pairs.
[0048] Step 3: Pre-train the feature extraction network. Set the hyperparameters, input all positive and negative sample pairs into the Momentum Contrast (MoCo) network, train the Momentum Contrast network, and save the backbone network parameters used for feature extraction.
[0049] Step 4: Train the classification, detection, and segmentation models. Use the pre-trained parameters to initialize the backbone network of the classification, detection, and segmentation models, and then fine-tune the models using a small amount of labeled colonoscopy image data.
[0050] Step 5: Use the trained model to classify, detect, or segment the new colonoscopy images and evaluate its performance metrics.
[0051] In step 2, the number of batch data is 384, the specified size is 320x320, and the scaling factor ranges from [0.5, 1.0].
[0052] Furthermore, the specific implementation method of the random appearance data augmentation is as follows:
[0053] This includes color distortion, random grayscale, and Gaussian blur. The robustness of the algorithm can be enhanced by increasing the richness of positive and negative sample pairs.
[0054] The specific implementation method of the random space data augmentation is as follows:
[0055] This includes random horizontal flipping, random rotation, etc. The robustness of the algorithm can be enhanced by increasing the richness of positive and negative sample pairs.
[0056] Furthermore, the specific implementation method of the medical information recovery is as follows:
[0057] 1) Set the recovery ratio ε to 0.6, which is the ratio of the length and width of the recovery region rectangle to the length and width of the image.
[0058] 2) Calculate the length and width of the restored rectangle.
[0059] 3) Randomly select the starting point of a rectangle in the image, and replace the RGB values of the corresponding pixels in the image after random appearance data augmentation with the original RGB values of the image.
[0060] After enhancement, the image x i Generate sample pairs (x) k x q ).
[0061] Step 3, the specific implementation of training the momentum contrast network, is as follows:
[0062] In this example, the hyperparameters are set as follows:
[0063] The backbone network structure is ResNet50, with a temperature coefficient τ of 0.2, a momentum coefficient m of 0.999, a queue size of 65664, a batch size of 384, and 500 epochs. The SGD optimizer is used, with an initial learning rate (lr) of 0.03, optimizer momentum of 0.9, and weight decay of 0.0001. Milestone learning rate planning is employed, reducing the learning rate to 1 / 10 of its original value at epochs 200, 300, and 400. The key and query networks use the same backbone network structure.
[0064] The momentum comparison network consists of a key network and a query network, each comprising an encoding layer and a mapping layer. It also maintains a key queue and updates the key encoder via momentum updates.
[0065] Assume x q and x k For positive sample pairs obtained from the same image in a batch of data after data augmentation, x q Input into the query network, x k Input to the key network; where x q The feature representation h is obtained after querying the network coding layer. q The query q;x is obtained by querying the network mapping layer. k The feature representation h is obtained after the key network encoding layer. k The key k is obtained through a key network mapping layer; the contrastive loss is calculated using key k and query q. The dot product is used to measure the similarity between key k and query q, and InfoNCE is used as the contrastive loss function.
[0066]
[0067] Where q represents the query, k+ represents the key of the corresponding positive sample, and k- represents the key of the negative sample. For the current image, the query q has only one key k+ generated from different enhanced versions of the same image to form a positive sample pair, while there are a total of N (N is the length of the sample queue) k- to form a negative sample pair. τ is an adjustable temperature coefficient hyperparameter.
[0068] The training process of the momentum contrast network can be summarized as follows:
[0069] In each training step, the keys of the current batch of image data are added to the key queue, while the most outdated keys are dequeued. During gradient backpropagation, only the parameters of the encoding and mapping layers of the query network are updated, while the gradients of the key network are ignored. Then, the parameters of the encoding and mapping layers of the key network are updated more smoothly using momentum.
[0070] θ k =mθ k +(1-m)θq
[0071] Where, θ k θ represents the parameters of the coding and mapping layers of the key network. q To query the coding and mapping layer parameters of the network, m is the momentum coefficient parameter, which takes values in the range [0,1).
[0072] After training is complete, the coding layer parameters of the query network are taken as the initialization parameters of the backbone network used for feature extraction in step 4.
[0073] Step 4, which involves fine-tuning the model using a small amount of labeled colonoscopy image data, is implemented as follows:
[0074] The training set portion and labels from the corresponding downstream task dataset are input into the model that has been initialized with the backbone network parameters described in step 3, for further training. In this embodiment, the corresponding downstream task dataset is:
[0075] The classification task used the SUN Colonoscopy Video Database. Due to the large amount of training data in this dataset, 1%, 2%, 5%, 10%, 20%, 50%, and 100% of its total training set were used for training.
[0076] The detection task used CVC-VideoClinicDB. Since the training set of this dataset is relatively small, the entire training set was used for training.
[0077] The segmentation task uses Kvasir-SEG. Since the training set of this dataset is relatively small, the entire training set is used for training.
[0078] In step 5, the new colonoscopy image is a portion of the test set in the corresponding downstream task dataset. In this embodiment, the corresponding downstream task dataset is:
[0079] The classification task used the SUN Colonoscopy Video Database; the detection task used the CVC-VideoClinicDB; and the segmentation task used the Kvasir-SEG.
[0080] Furthermore, the specific implementation method for evaluating its performance indicators is as follows:
[0081] The test set is input into the trained model, the output results are compared with the labels, and the corresponding performance evaluation indicators are calculated.
[0082] In this embodiment, for the classification model, the area under the ROC curve (AUC), accuracy, precision, recall, and F1 score are calculated; for the detection model, the mean average precision (mAP) of each category is calculated; and for the segmentation model, the mean intersection over union (mIoU) is calculated.
[0083] To test the effectiveness of our method, we compared it with popular paradigms in the field of medical image analysis, based on several mainstream backbone networks and baselines.
[0084] The mainstream backbone network and baseline are as follows:
[0085] In the classification task, ResNet18, ResNet50 and ResNet101 backbone networks were used.
[0086] In the detection task, Faster R-CNN, Mask R-CNN, Cascade R-CNN and RetinaNet baselines were used, all with ResNet50 as their backbone network.
[0087] In the segmentation task, only FCN, DeepLab v3, DeepLab v3 plus, GCNet, and PSPNet baselines were used, and their backbone networks were all ResNet50.
[0088] The prevailing paradigm in the field of medical image analysis is as follows: first, the model is pre-trained in a supervised manner on large labeled natural datasets such as ImageNet and MS COCO, and then fine-tuned on a target task medical dataset with limited labels.
[0089] Depend on Figure 2 As can be seen, for the SUN Colonoscopy Video Database dataset, compared with conventional methods in the field of medical image analysis, the structure proposed in this method achieves better performance in all evaluation metrics for three different backbone networks.
[0090] Depend on Figure 3It can be seen that, for the CVC-VideoClinicDB dataset, compared with conventional methods in the field of medical image analysis, the proposed structure achieves better performance in the mAP metric for four different baseline networks.
[0091] Depend on Figure 4 It can be seen that, for the Kvasir-SEG dataset, compared with conventional methods in the field of medical image analysis, the proposed structure achieves better performance in the mIoU metric for five different baseline networks.
[0092] The method of this invention, compared with existing technologies, realizes a simple, efficient, low-cost, and high-performance colonoscopy image classification, detection, and segmentation method that combines medical information recovery and momentum contrast. Addressing the problem of the scarcity of labeled colonoscopy images, the Momentum Contrast (MoCo) algorithm is used for self-supervised feature extraction in colonoscopy image analysis. This requires only a small number of labeled data samples to complete the training of downstream task models, resulting in low dependence on labeled data. To address the issue of the poor performance of self-supervised learning in colonoscopy image analysis, a method is designed to balance the need for image enhancement richness in contrastive learning with the need for medical realism in feature extractors by restoring a portion of the appearance-enhanced image. This encourages the model to extract more representative features, effectively improving the performance of downstream task models and surpassing conventional methods in the field of medical image analysis.
Claims
1. A method for classifying, detecting, and segmenting colonoscopy images by combining medical information recovery and momentum comparison, characterized in that, Includes the following steps: Step 1: Collect colonoscopy images to form a pre-training dataset; Step 2, Data Augmentation Batch data is obtained by sampling from the pre-training dataset. Each image in the batch data is cropped and scaled to a specified size. Then, two random appearance data augmentations are performed to obtain appearance-enhanced images. Then, medical information restoration methods are used to locally restore the appearance-enhanced images. Finally, random spatial data augmentation is performed to obtain positive and negative sample pairs. The steps for local restoration of the appearance enhancement image using the medical information restoration method are as follows: 1) Set the restoration ratio ε, which is the ratio of the length and width of the restoration region rectangle to the length and width of the appearance enhancement image. The value of ε ranges from 0 to 1. 2) Calculate the length and width of the restored region rectangle based on the restoration ratio ε and the length and width of the apparent enhanced image; 3) Randomly select the starting point of the rectangular restoration region in the image to determine the position of the restoration region rectangle, and replace the RGB values of the corresponding pixels in the image after random appearance data augmentation with the original RGB values of the image; Step 3: Pre-train the feature extraction network Set hyperparameters, input all positive and negative sample pairs into the momentum contrast network, train the momentum contrast network and save the backbone network parameters for feature extraction; The hyperparameters include backbone network structure, temperature coefficient, momentum coefficient, queue length, batch size, number of iterations, and optimizer-related parameters. Step 4: Train the classification, detection, and segmentation models. The parameters of the pre-trained backbone network in step 3 are used to initialize the backbone network of the classification, detection, and segmentation models, and then the model is fine-tuned using labeled colonoscopy image data. Step 5: Use the trained model to classify, detect, or segment the new colonoscopy images.
2. The colonoscopy image classification, detection, and segmentation method combining medical information recovery and momentum comparison according to claim 1, characterized in that, The apparent data enhancement described in step 2 includes one or more of color distortion, random grayscale, and Gaussian blur.
3. The colonoscopy image classification, detection, and segmentation method combining medical information recovery and momentum comparison according to claim 1, characterized in that, The spatial data augmentation described in step 2 includes one or more of random horizontal flipping and random rotation.
4. The colonoscopy image classification, detection, and segmentation method combining medical information recovery and momentum comparison according to claim 1, characterized in that, The specific method for training the momentum contrast network in step 3 is as follows: The momentum comparison network consists of a key network and a query network, both of which include an encoding layer and a mapping layer. It also includes a key queue, which updates the encoding and mapping layers of the key network through momentum updates. Assume x q and x k For positive sample pairs obtained from the same image in a batch of data after data augmentation, x q Input into the query network, x k Input to the key network; where x q The feature representation h is obtained after querying the network coding layer. q The query q;x is obtained by querying the network mapping layer. k The feature representation h is obtained after the key network encoding layer. k The key k is obtained through a key network mapping layer; the contrastive loss is calculated using key k and query q; the dot product is used to measure the similarity between key k and query q; InfoNCE is used as the contrastive loss function. in, Let q represent the contrastive loss function, k+ be the key of the corresponding positive sample, and k- be the key of the negative sample. For the current image, the query q has only one key k generated from different augmented versions of the same image. + Positive sample pairs are formed, while a total of N k- form negative sample pairs, where N is the length of the key queue; τ is an adjustable temperature coefficient; The training process of the momentum contrast network can be summarized as follows: In each training step, the keys of the current batch of data are added to the key queue, while the most outdated keys are dequeued. During gradient backpropagation, only the parameters of the encoding and mapping layers of the query network are updated, while the gradients of the key network are ignored. Then, the parameters of the encoding and mapping layers of the key network are updated more smoothly using momentum. i k =mθ k +(1-m)θq Where, θ k Let θ be the set of parameters for the encoding and mapping layers of the key network. q To query the set of parameters for the coding and mapping layers of the network, m is the momentum coefficient, which takes values in the range [0,1); After training is complete, the coding layer parameters of the query network are taken as the initialization parameters of the backbone network used for feature extraction in step 4.
5. The colonoscopy image classification, detection, and segmentation method combining medical information recovery and momentum comparison according to claim 1, characterized in that, The backbone network described in step 3 is a deep convolutional neural network, which serves as the encoding layer for the query network and the key network during the training of the momentum contrast network.
6. The colonoscopy image classification, detection, and segmentation method combining medical information recovery and momentum comparison according to claim 1, characterized in that, The model described in step 4 is any one of the following: ResNet or MobileNet for classification; Faster RCNN, Mask RCNN, RetinaNet, or YOLO for detection; or FCN, UNet, DeepLab, GCNet, or PSPNet for segmentation.