Deep learning-based polyp detection model training method
By using colonoscopy report images to train the basic model and combining time window filters and asymmetric mixing algorithms, pseudo-label samples are extracted for fine-tuning of the model, solving the problem of high cost of data set establishment and field gap in colon polyp detection, and achieving more Efficient polyp detection and dataset establishment.
Patent Information
- Application Number
- CN202410997788.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-24
AI Technical Summary
In the prior art, colon polyp detection depends on the vision and experience of endoscopists, and there is still room for improvement in accuracy and efficiency. Due to the high data set size and labeling cost, deep learning models have low model training and application efficiency.
Using a deep learning-based polyp detection model training method, the basic model is trained by using colonoscopy to report images, pseudo-label samples are extracted and hard-negative samples are combined for model fine-tuning, and the image is constructed using time window filters and asymmetric mixing algorithms-video - Connector network to reduce model training costs and domain gaps.
The accuracy and reliability of the establishment of the polyp detection data set is improved, the labeling cost is reduced, the adaptability and accuracy of the model is improved, and the field gap between colonoscopy report images and real-time videos is solved.
Smart Images

Figure CN118966308A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a medical image processing method, and in particular to a colorectal polyp detection model training method based on an artificial intelligence deep learning algorithm, belonging to the technical field of colonoscopic polyp detection. Background Art
[0002] Colorectal cancer (CRC) is a general term for a series of tumors that occur in the colon and rectum. It is the third most common cancer in the world and the second most common cancer in terms of mortality. It is also the fourth most common cancer in China and the fifth most common cancer in terms of mortality. Colonoscopy is the core technical means and gold standard for early diagnosis and screening of colorectal cancer. High-quality colonoscopy can effectively reduce the incidence and mortality of colorectal cancer. Before endangering health, CRC manifests as colon polyps, which can be detected and treated early through colonoscopy. Therefore, accurate detection of polyps is crucial.
[0003] Currently, polyp detection mainly relies on the vision and experience of endoscopists. However, due to the different morphologies of colon polyps, it is very difficult to distinguish them from normal structures. Even for professional endoscopists, there is an inevitable rate of polyp missed detection. Although computer-aided detection (CAD) systems exist to improve the detection rate, the CAD systems in the prior art still have room for improvement in accuracy and efficiency.
[0004] The typical process of discovering colon polyps can be summarized in four stages: Stage 1: The endoscopist scans the colon to look for polyps while withdrawing the scope; Stage 2: The polyps appear in the field of view; Stage 3: The endoscopist notices the potential polyps and stops removing the tube; Stage 4: The endoscopist adjusts the field of view to observe carefully and take photos for reporting. The correct judgment of the endoscopist in the third stage is crucial and is the prerequisite for observation and recording in the fourth stage. Therefore, the stage in clinical practice where computer-aided diagnosis of colon polyps is most needed is the second stage when the polyps just appear. In other words, CAD needs to be performed in a real-time video stream simultaneously with the colonoscopy.
[0005] Recently, deep learning has been widely used in colon polyp CAD due to its excellent performance. Directly using colonoscopy videos to train polyp detection models is an intuitive solution. However, it is a cumbersome task for professional endoscopists to annotate polyp areas in colonoscopy videos. Since video recording is not a routine procedure for colonoscopy, collecting sufficient data (especially positive samples) is also very time-consuming. These two difficulties in clinical practice limit the scale of video datasets, and the lack of diverse datasets may lead to overfitting of deep learning models. In short, manually annotating polyp areas in large-scale video datasets is time-consuming and expensive, which limits the development of deep learning technology.
[0006] A compromise approach is to train the target model using annotated colonoscopy report images and perform inference on live videos. Since these images can be extracted from medical report databases, it is easy for researchers to obtain a much larger dataset than recording videos from scratch. However, there are some problems between image-based training and video-based inference, including domain differences and a lack of positive video samples. Summary of the invention
[0007] The purpose of the present invention is to reduce the cost of establishing a training data set for a polyp detection model, to address the field gap between colonoscopy report images and real-time videos, to improve the accuracy and reliability of polyp detection and its data set establishment, and to reduce annotation costs.
[0008] In order to achieve the above object, the technical solution of the present invention is to provide a polyp detection model training method based on deep learning, which is characterized by comprising the following steps:
[0009] Step 1: Use source domain report images to train a basic polyp detection model;
[0010] Step 2: Use the trained basic polyp detection model to extract pseudo-label samples from the unlabeled target domain video, where the pseudo-label samples include positive pseudo-label samples and hard negative samples, where the positive pseudo-label samples contain polyps under different shooting angles and lighting conditions, and the hard negative samples contain various misleading noises;
[0011] Step 3: Retrain the basic polyp detection model using source domain report images mixed with hard negative samples, and then fine-tune the basic polyp detection model using positive pseudo-label samples.
[0012] Preferably, in step 2, a time window filter is used to extract the positive pseudo-label sample and the hard-negative sample. During the extraction, the positive pseudo-label sample or the hard-negative sample is determined by judging whether a polyp prediction in any frame of the target domain video is surrounded by other polyp predictions with a high density: if a polyp prediction is surrounded by other polyp predictions with a high density, the current frame of video is the positive pseudo-label sample; otherwise, the current frame of video is the hard-negative sample.
[0013] Preferably, in step 2, the time window filter extracts the positive pseudo-label samples and the hard negative samples by the following steps:
[0014] The target domain video is inferred by the basic polyp detection model to obtain the output Det({x i}) is represented as:
[0015] Det({x i})={p i}={{(s ij ,b ij )}}
[0016] In the formula, p i is the i-th frame video x in the target domain video i Box-level prediction, s ij With b ij They are p i The confidence score and bounding box coordinates of the j-th polyp prediction in ;
[0017] According to the box-level prediction p i The maximum confidence score s i ,s i =max(s ij ), assigning a binary classification prediction c to each frame of unlabeled transformed training video i , as shown below:
[0018]
[0019] In the formula, the constant T cls is the classification threshold;
[0020] Calculate the i-th frame unlabeled conversion training video x i The surrounding positive density d i , as shown below:
[0021]
[0022] Where t is the start time of the time window, and the constant L is the length of the time window;
[0023] High confidence predictions with high surrounding positive density are extracted as positive pseudo-label samples, as shown in the following formula:
[0024] f pos ({p i})={(x i ,y i )|d i ≥D∧s i ≥T bbox}
[0025] y i = {b ij ∣∣s ij ≥T bbox}
[0026] In the formula, y i is the i-th frame of the unlabeled transformed training video x i The positive pseudo-label of , the constant D is the density threshold of the time window filter, and the constant T bboxis the bounding box extraction threshold, f pos ({p i}) is a positive pseudo-label sample with high confidence prediction and high surrounding positive density, (x i ,y i ) represents a frame image and its positive pseudo label;
[0027] High confidence predictions with low surrounding positive density are extracted as hard negative samples as shown in the following formula:
[0028]
[0029] In the formula, f hardNeg ({p i}) are hard negative samples with high confidence predictions with low surrounding positive density, Indicates that the frame has no positive pseudo-label.
[0030] Preferably, in step 3, when mixing the hard negative sample with the source domain report image, an asymmetric mixing algorithm is used to utilize the negative pseudo-label sample.
[0031] Preferably, in step 3, after the hard negative sample is mixed with the source domain report image, the constructed virtual training sample as follows:
[0032]
[0033] In the formula, is any positive input frame in the source domain report image, yes Tags, is a hard negative frame randomly selected from the hard negative pseudo-label samples extracted in step 2, and λ is a random variable controlling the mixing ratio.
[0034] Preferably, an asymmetric discrete distribution is used to limit the impact of hard negative samples, and the mixing strength and probability are controlled separately, as shown in the following formula:
[0035] P(λ=γ)=ρ,λ∈{0,γ}
[0036] Where λ can be 0 or γ, where the probability that λ is γ is ρ, γ and ρ are two constant hyperparameters, the hyperparameter γ is used to control the mixing strength, and the hyperparameter ρ is used to control the mixing probability.
[0037] The present invention provides an image-video-joint polyp detection network (Evo-Pod Network), i.e., a training method for unsupervised domain adaptation, which improves the accuracy and reliability of polyp detection and its data set establishment, while reducing the labeling cost.
[0038] Compared with the prior art solutions, the present invention has the following beneficial effects:
[0039] 1) Reduce the cost of establishing the polyp detection model training data set;
[0040] 2) A training method called Image-Video-Pod Network was proposed to address the domain gap between colonoscopy report images and real-time videos. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of the scheme of the present invention;
[0042] Figure 2 The time window confirmation algorithm is illustrated. DETAILED DESCRIPTION
[0043] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.
[0044] I) Foundation: A basic polyp detection model is trained using colonoscopy report images (hereinafter referred to as “source domain report images”) to learn polyps with morphological diversity.
[0045] II) Extraction: Extract pseudo-label samples from unlabeled target domain videos using the basic polyp detection model. The present invention believes that there are two types of samples that are valuable for domain adaptation because they usually do not appear in the source domain report images: one is positive samples, also known as positive pseudo-label samples, which contain polyps under different shooting angles and lighting conditions; the other is hard negative samples, which contain various misleading noises such as feces and bubbles.
[0046] Due to the temporal consistency of colonoscopy videos, adjacent frames should contain similar content. If one frame contains a polyp, its adjacent frame may also contain a polyp, and vice versa. In other words, if a polyp prediction is surrounded by a high density of other polyp predictions, then it is likely to be a true positive, otherwise it is a false positive. The present invention refers to the false positive samples found by this method as hard negative samples.
[0047] This paper proposes a simple time window filtering (TWF) to accurately extract positive pseudo-label samples and hard negative samples. The specific strategy is as follows:
[0048] The unlabeled transformed training video (i.e., the “unlabeled target domain video” mentioned above) is inferred from the output Det({x i}) is represented as:
[0049] Det({x i})={p i}={{(s ij ,b ij )}}
[0050] In the formula, p i For the i-th frame, the unlabeled transformed training video x i Box-level prediction, s ij With b ij They are p i Confidence score and bounding box coordinates of the j-th polyp prediction in .
[0051] According to the box-level prediction p i The maximum confidence score s i ,s i =max(s ij ), assigning a binary classification prediction c to each frame of unlabeled transformed training video i (positive is 1, negative is 0), as shown in the following formula:
[0052]
[0053] In the formula, the constant T cls is the classification threshold.
[0054] Calculate the i-th frame unlabeled conversion training video x i The surrounding positive density d i :
[0055]
[0056] Where t is the start time of the time window, and the constant L is the length of the time window. The present invention defines d by the maximum density obtained in a set of sliding time windows [t, t+L–1] i , rather than a single fixed time window.
[0057] High confidence predictions with high surrounding positive density are extracted as positive pseudo-label samples, as shown in the following formula:
[0058] f pos ({p i})={(x i ,y i )|d i ≥D∧s i ≥T bbox}
[0059] y i = {b ij ∣∣s ij ≥T bbox}
[0060] In the formula, y i is the i-th frame of the unlabeled transformed training video x i The positive pseudo-label of , the constant D is the density threshold of the time window filter, and the constant T bbox is the bounding box extraction threshold, f pos ({p i}) is a positive pseudo-label sample with high confidence prediction and high surrounding positive density, (x i ,y i ) represents a frame image and its positive pseudo label.
[0061] High confidence predictions with low surrounding positive density are extracted as false positive samples, i.e., as hard negative samples, as shown in the following formula:
[0062]
[0063] In the formula, f hardNeg ({p i}) are hard negative samples with high confidence predictions with low surrounding positive density, Indicates that the frame has no positive pseudo-label.
[0064] In order to support a larger time window length L, the embodiment of the present invention does not directly calculate d i , but use Figure 2 The algorithm shown is used to determine whether d i ≥D, the complexity is from O(NL 2 ) is reduced to O(N), where N is the number of video frames, and specifically includes the following steps:
[0065] Step 1, initialization: set {v i}=0, and initialize the sum variable S and position variable P.
[0066] Step 2: Traverse the video frames: loop from frame 1 to frame N and perform the following steps:
[0067] Step 201: Accumulate predicted values: add the predicted value c of the current frame i Add to the sum variable S.
[0068] Step 202, sliding window: if the current frame index is greater than the time window length L, then subtract the predicted value c of the iL frame {i-L} To maintain the length of the sliding window.
[0069] Step 203, density calculation and verification: If the density S / L in the current window is greater than or equal to the threshold D, then:
[0070] Update the positive sample flag: set the flag v of all frames from max(P+1,i-L+1) to i j Set to 1.
[0071] Update position variable: Update the position variable P to the current frame index i.
[0072] III) Application: Use the pseudo-label samples obtained in step II) to improve the polyp detection performance.
[0073] The present invention first retrains the basic polyp detection model using source domain report images mixed with hard negative samples to reduce false positives caused by various noises in real-time video. Then, the basic polyp detection model is fine-tuned using positive pseudo-label samples to adapt to various shooting conditions of real-time video. The specific strategy is as follows:
[0074] The present invention utilizes the positive pseudo-label samples and hard negative pseudo-label samples extracted in step II) to adapt the basic polyp detection model of the target domain.
[0075] Since negative samples cannot be used directly to train the basic polyp detection model, the present invention proposes an asymmetric hybrid algorithm to utilize negative pseudo-label samples, and uses the source domain report images mixed with hard negative samples to retrain the basic model to reduce false positives. That is, the present invention randomly adds hard negative samples to positive samples at the pixel level (after adjusting to the same size), and uses the background noise in the hard negative samples to enhance the positive training data. Formally, the virtual training samples constructed by the present invention are as follows:
[0076]
[0077] In the formula, is any positive input frame in the training dataset, yes Tags, is a hard negative frame randomly selected from the hard negative pseudo-label samples extracted in step II), and λ is a random variable controlling the mixing ratio.
[0078] The present invention adopts an asymmetric discrete distribution to limit the influence of hard negative samples and independently controls the mixing strength and probability, as shown in the following formula:
[0079] P(λ=γ)=ρ,λ∈{0,γ}
[0080] Where λ can be 0 or γ, where the probability that λ is γ is ρ. γ and ρ are two constant hyperparameters that allow us to adjust the impact of hard negative samples, where the hyperparameter γ is used to control the mixing strength and the hyperparameter ρ is used to control the mixing probability.
[0081] Then, the present invention uses positive pseudo-label samples to fine-tune the basic polyp detection model to adapt to various shooting conditions of real-time video.
Claims
1. A polyp detection model training method based on deep learning, characterized in that: The following steps are involved: Step 1: Use source domain report images to train a basic polyp detection model; Step 2: Use the trained basic polyp detection model to extract pseudo-label samples from the unlabeled target domain video, where the pseudo-label samples include positive pseudo-label samples and hard negative samples, where the positive pseudo-label samples contain polyps under different shooting angles and lighting conditions, and the hard negative samples contain various misleading noises; Step 3: Retrain the basic polyp detection model using source domain report images mixed with hard negative samples, and then fine-tune the basic polyp detection model using positive pseudo-label samples.
2. A polyp detection model training method based on deep learning as claimed in claim 1, characterized in that: In step 2, a time window filter is used to extract the positive pseudo-label sample and the hard negative sample. During the extraction, the positive pseudo-label sample or the hard negative sample is determined by judging whether there are other polyp predictions with high density around a polyp prediction in any frame of the target domain video: if there are other polyp predictions with high density around a polyp prediction, the current frame of video is the positive pseudo-label sample; otherwise, the current frame of video is the hard negative sample.
3. A polyp detection model training method based on deep learning as claimed in claim 2, characterized in that: In step 2, the time window filter extracts the positive pseudo-label samples and the hard negative samples using the following steps: The target domain video is inferred by the basic polyp detection model to obtain the output Det({x i }) is represented as: It({x i })={p i }={(s ij ,b ij )}} In the formula, p i is the i-th frame video x in the target domain video i Box-level prediction, s ij With b ij They are p i The confidence score and bounding box coordinates of the j-th polyp prediction in ; According to the box-level prediction p i The maximum confidence score s i ,s i =max(s ij ), assigning a binary classification prediction c to each frame of unlabeled transformed training video i , as shown below: In the formula, the constant T cls is the classification threshold; Calculate the i-th frame unlabeled conversion training video x i The surrounding positive density d i , as shown below: Where t is the start time of the time window, and the constant L is the length of the time window; High confidence predictions with high surrounding positive density are extracted as positive pseudo-label samples, as shown in the following formula: f pos ({p i })={(x i ,y i )∣d i ≥D∧s i ≥T bbox } y i ={b ij ∣∣s ij ≥T bbox } In the formula, y i is the i-th frame of the unlabeled transformed training video x i The positive pseudo-label of , the constant D is the density threshold of the time window filter, and the constant T bbox is the bounding box extraction threshold, f pos ({p i }) is a positive pseudo-label sample with high confidence prediction and high surrounding positive density, (x i ,y i ) represents a frame image and its positive pseudo label; High confidence predictions with low surrounding positive density are extracted as hard negative samples as shown in the following formula: In the formula, f hardNeg ({p i }) are hard negative samples with high confidence predictions with low surrounding positive density, Indicates that the frame has no positive pseudo-label.
4. The method for training a polyp detection model based on deep learning according to claim 1, characterized in that: In step 3, when the hard negative samples are mixed with the source domain report image, an asymmetric mixing algorithm is used to utilize negative pseudo-label samples.
5. A polyp detection model training method based on deep learning as claimed in claim 4, characterized in that: In step 3, after the hard negative sample is mixed with the source domain report image, the constructed virtual training sample as follows: In the formula, is any positive input frame in the source domain report image, yes Tags, is a hard negative frame randomly selected from the hard negative pseudo-label samples extracted in step 2, and λ is a random variable controlling the mixing ratio.
6. A polyp detection model training method based on deep learning as claimed in claim 5, characterized in that: An asymmetric discrete distribution is used to limit the impact of hard negative samples, and the mixing strength and probability are controlled separately, as shown in the following formula: P(λ=γ)=ρ,λ∈{0,γ} Where λ can be 0 or γ, where the probability that λ is γ is ρ, γ and ρ are two constant hyperparameters, the hyperparameter γ is used to control the mixing strength, and the hyperparameter ρ is used to control the mixing probability.
Citation Information
Patent Citations
Domain adaptive behavior recognition system and method based on data enhancement and pseudo tag
CN114638995A
Intestinal polyp detection system and method based on deep learning
CN115880266A
Polyp segmentation method capable of being used under different data distributions
CN116309312A
MCI conversion intelligent prediction method and system based on brain network and deep learning
CN116468677A
Self-training domain adaptive remote sensing image semantic segmentation method based on pseudo label quality
CN116863136A
Cited By
Enteroscope polyp size evaluation method and system based on multi-frame matching
CN119722775A
