A wireless capsule endoscope lesion automatic detection method, device and storage medium
Through keyframe recognition and automatic tracking, the labeled data set is generated, combined with active learning and data synthesis optimization model, the problem of difficulty in labeling and low detection accuracy in wireless capsule endoscopic lesions detection is solved, and the detection accuracy and efficiency are improved.
Patent Information
- Application Number
- CN202210103187.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-27
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-01-27
AI Technical Summary
In the automatic detection of lesions, wireless capsule endoscopes have problems such as difficulty in obtaining labeled data sets, small number of positive cases, small target areas and strong image diversity, resulting in low detection accuracy and high false detection rate.
Keyframe recognition technology is used to combine expert annotation and automatic tracking to generate annotated data sets, and the lesion detection model is optimized through active learning and data synthesis methods to reduce the rate of missed detection and false detection.
It improves the accuracy and efficiency of lesion detection, reduces the cost of expert labeling, enhances model performance, and reduces the rate of missed detection and false detection.
Smart Images

Figure CN114581375B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and intelligent disease diagnosis, and in particular to a method, device and storage medium for automatic detection of lesions using a wireless capsule endoscope. Background Art
[0002] Wireless Capsule Endoscopy (WCE) is an instrument used to detect abnormalities in the human stomach and esophagus. It enters the human body through a microcapsule, moves along the digestive direction with the movement of the gastrointestinal muscles, and takes pictures, which are then transmitted to an imaging workstation for doctors to diagnose digestive tract diseases.
[0003] At present, the clinical use of wireless capsule endoscopes for diagnosis mainly relies on doctors' manual film reading. However, it takes 8-12 hours to collect videos of a subject and collect about 120,000 pictures. Manual screening by experts is highly subjective, time-consuming and labor-intensive, and prone to missed or false detection. There are also studies that use deep learning methods to automatically diagnose lesions with wireless capsule endoscopes, but there is still a certain gap with actual clinical needs and applications. There are mainly the following defects and technical challenges:
[0004] First, it is difficult to obtain a large number of annotated data sets for training. In medical tasks, lesion annotation relies on expert knowledge, and even requires the comprehensive conclusions of multiple experts to obtain correct annotations. The annotation cost is time-consuming and expensive, and it is difficult to meet the requirements of model training for large annotated data sets.
[0005] Second, the number of positive examples is small and the target in the lesion area is small. For target detection tasks, the number of positive examples containing lesions is small, and the lesion area is small relative to the area occupied by the entire image, which is easy to miss.
[0006] Third, the image diversity is strong and the background is complex, so the accuracy of lesion detection is low. The photos taken by wireless capsule endoscopes are easily affected by residues, bubbles, etc., which may cause false detection and reduce the accuracy of lesion detection. Summary of the invention
[0007] In order to obtain a high-performance lesion detection model with as little expert-labeled data as possible, the present invention provides a wireless capsule endoscope lesion automatic detection method, device and storage medium to provide doctors with relatively accurate lesion detection support.
[0008] The technical means adopted by the present invention are as follows:
[0009] A wireless capsule endoscope lesion automatic detection method, characterized in that it comprises the following steps:
[0010] S1. Obtain a video file captured by a wireless capsule, convert the video file into an image sequence with a timestamp, perform key frame recognition on the image sequence with a timestamp, obtain an expert's annotation results for the key frames, and use an automatic tracking method to obtain an annotation result for a non-key frame, thereby obtaining an annotated data set;
[0011] S2, using the labeled data set as training data to train a lesion detection model, obtaining a wireless capsule endoscope video to be detected, and performing lesion detection on the wireless capsule endoscope video to be detected based on the trained lesion detection model, and the obtained lesion detection results include the timestamps, lesion locations, lesion categories and confidences of all images with lesions, wherein the confidences represent the probability that the images with lesions belong to the corresponding lesion category;
[0012] S3, according to the lesion detection result of the wireless capsule endoscope video to be detected, all images with lesions are sorted from small to large according to the confidence, and images with confidence less than a certain threshold range are used as "difficult example" samples, and the correction results of experts on the "difficult example" samples are obtained to obtain the "missed" lesion information and the "falsely detected" lesion information, and then generate a "missed detection library" and a "falsely detected library", and at the same time, the lesion detection model is actively learned based on the expert's correction results on the "difficult example" samples;
[0013] S4. Performing synthesis processing on the data in the "missed detection library" and the "false detection library" to generate a new synthetic data set, and iteratively optimizing the lesion detection model based on the synthetic data set to fine-tune the lesion detection model weights.
[0014] Furthermore, S1 converts the video file into an image sequence with a timestamp, including:
[0015] S111, converting the video into an image according to a preset video frame counting interval frequency;
[0016] S112: Automatically extract the shooting time in the image based on OCR technology, so as to obtain an image sequence with a time stamp.
[0017] Furthermore, in S1, key frame recognition is performed on the image sequence with the timestamp, including:
[0018] S121, performing feature extraction on each frame of the image in the image sequence with the timestamp, so as to obtain image features;
[0019] S122, calculating the similarity of the image features of two adjacent frames based on the image features, and when the similarity is less than a set threshold, taking the current frame image as a key frame; otherwise, taking the current frame image as a similar frame of the previous key frame and moving to the next frame;
[0020] S123, repeating the similarity calculation and comparison steps until the entire image sequence is traversed, thereby obtaining a key frame and a similar interval sequence.
[0021] Furthermore, in S1, an automatic tracking method is used to obtain the annotation results of non-key frames, including:
[0022] S131, based on the expert's annotation results of the key frames, filter out the key frames with lesions;
[0023] S132. For each key frame with lesions, a target tracking algorithm is used to automatically match all lesion regions of the key frame in a similar interval sequence, thereby achieving automatic labeling of lesions in all similar intervals.
[0024] Furthermore, in S2, lesion detection is performed on the wireless capsule endoscope video to be detected based on the trained lesion detection model, including:
[0025] S211, converting the wireless capsule endoscope video to be detected into a sequence of images to be detected with a timestamp according to the method described in S1;
[0026] S212. Load the trained lesion detection model, perform lesion detection on all images in the image sequence to be detected, and then output the detection result with only the lesion image as the lesion detection result of the wireless capsule endoscope video to be detected, wherein the detection result with the lesion image includes the image timestamp, lesion location, lesion category and confidence.
[0027] Furthermore, S3 obtains the correction results of the experts for the "difficult" samples, obtains the "missed" lesion information and the "falsely detected" lesion information, and then generates the "missed detection library" and "falsely detected library", including:
[0028] S311, extracting the "missed" lesion information, cropping the corresponding area of the image according to the lesion position coordinates and storing it in the "missed library", wherein the "missed" lesion information includes the lesion position and lesion category manually marked by experts;
[0029] S312, extracting the "falsely detected" lesion information, cropping the corresponding area of the image according to the lesion position coordinates and storing it in the "falsely detected library", wherein the "falsely detected" lesion information includes the lesion position predicted by the lesion detection model.
[0030] Furthermore, in S4, the data in the "missed detection library" and the "false detection library" are synthesized to generate a new synthetic data set, including:
[0031] S411, randomly selecting an image from the image sequence to be detected as a background image;
[0032] S412, randomly selecting M images from the "missed detection library" and randomly selecting N images from the "false detection library" to generate a target image;
[0033] S413, performing random data enhancement on the target image, wherein the data enhancement method is scaling, flipping or rotating;
[0034] S414, determining an available synthesis area that meets the brightness requirement in the background image by color space conversion, and randomly selecting a synthesis position in the available synthesis area;
[0035] S415, using an image fusion method to fuse the target image into the background image to generate a composite image, wherein the label of the composite image includes the original lesion in the background image and the lesion information newly integrated from the "missed detection library";
[0036] S416. Repeat S411 to S415K times to obtain a synthetic data set consisting of K images.
[0037] The present invention also discloses a wireless capsule endoscope lesion automatic detection device, which is used to implement any one of the above automatic detection methods, including:
[0038] A pre-processing module is used to obtain the video file collected by the wireless capsule, convert the video file into an image sequence with a time stamp, perform key frame recognition on the image sequence with a time stamp, obtain the annotation results of the key frames by experts, and use an automatic tracking method to obtain the annotation results of non-key frames, so as to obtain an annotated data set;
[0039] A lesion detection module is used to train a lesion detection model using the annotated data set as training data, obtain a wireless capsule endoscope video to be detected, and perform lesion detection on the wireless capsule endoscope video to be detected based on the trained lesion detection model. The obtained lesion detection results include the timestamps, lesion locations, lesion categories and confidence levels of all images with lesions, wherein the confidence level indicates the probability that the image with lesions belongs to the corresponding lesion category;
[0040] An active learning module, which is used to sort all images with lesions from small to large according to the confidence level according to the lesion detection results of the wireless capsule endoscope video to be detected, and use images with confidence levels less than a certain threshold range as "difficult example" samples, obtain the correction results of experts on the "difficult example" samples, obtain the "missed" lesion information and the "falsely detected" lesion information, and then generate a "missed detection library" and a "falsely detected library", and at the same time, actively learn the lesion detection model based on the expert's correction results on the "difficult example" samples;
[0041] The data synthesis module synthesizes the data in the "missed detection library" and the "false detection library" to generate a new synthetic data set, and iteratively optimizes the lesion detection model based on the synthetic data set to fine-tune the lesion detection model weight.
[0042] The present invention also provides a computer-readable storage medium, in which a computer instruction set is stored; when the computer instruction set is executed by a processor, a wireless capsule endoscope lesion automatic detection method as described in any one of the above is implemented.
[0043] Compared with the prior art, the present invention has the following advantages:
[0044] 1. The present invention provides a wireless capsule endoscope lesion automatic detection method, which can assist experts in completing automatic lesion detection and improve film reading efficiency and detection performance.
[0045] 2. The present invention adopts key frame recognition technology to eliminate redundant images and solve the problem of a large number of images after video conversion and high similarity between adjacent frames. On this basis, the present invention combines expert annotation of key frames with automatic tracking of similar intervals to achieve a larger sample size of annotated data set with less expert annotation cost.
[0046] 3. The present invention adopts deep learning end-to-end training to obtain a lesion detection model. Through iterative optimization of the model, the lesion position and lesion type can be located with high precision.
[0047] 4. The present invention uses active learning to select "difficult" samples and have experts correct the prediction results, providing sample support for subsequent data synthesis to gradually improve model performance.
[0048] 5. The present invention designs a data synthesis method to solve the problems of few positive examples in training samples and small target areas. At the same time, the synthesized images are used to retrain and fine-tune the network weights, which greatly reduces the model's missed detection rate and false detection rate and improves detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0050] Figure 1 The present invention is a flow chart of the automatic lesion detection method of wireless capsule endoscope.
[0051] Figure 2 It is an overall execution flow chart of the automatic detection method in the embodiment.
[0052] Figure 3 Schematic diagram of a key frame recognition method in an embodiment.
[0053] Figure 4 Schematic diagram of the YOLO v4 network structure in the embodiment.
[0054] Figure 5 It is the training set loss curve in the embodiment.
[0055] Figure 6 This is the validation set loss curve in the embodiment.
[0056] Figure 7 Schematic diagram of the test set ground-truth in the embodiment.
[0057] Figure 8 This is a schematic diagram of predicting TP and FP for the test set in the embodiment.
[0058] Fig. 9 Schematic diagram of mAP of the test set in the embodiment.
[0059] Fig.10a This is Example 1 of the easily divisible sample in the embodiment.
[0060] Fig.10b This is Example 2 of the easily divisible sample in the embodiment.
[0061] Fig.10c This is Example 3 of the easily divisible sample in the embodiment.
[0062] Fig.11a This is Example 1 of the missed sample in the embodiment.
[0063] Fig.11b This is Example 2 of the missed sample in the embodiment.
[0064] Fig.11c This is Example 3 of the missed sample in the embodiment.
[0065] Fig.12a The example 1 of the misdetected sample in the embodiment is shown in FIG.
[0066] Figure 12b The example 2 of the misdetected sample in the embodiment is shown in FIG.
[0067] Fig.12c The example 3 of the misdetected sample in the embodiment is shown in FIG.
[0068] Fig.13a This is an example of a background image in the embodiment.
[0069] Fig.13b 2 is an example of a target map in the embodiment.
[0070] Fig.14 This is an example of binarization of the background image V channel in the embodiment.
[0071] Fig.15 This is an example of the synthesized image in the embodiment.
[0072] Fig.16 This is an example of a synthetic data set sample in the embodiment.
[0073] Fig.17 This is a display of the model detection results after loading and fine-tuning in the embodiment.
[0074] Fig.18 This is a schematic diagram of the mAP of the test set after fine-tuning the model in the embodiment.
[0075] Fig.19 This is a structural diagram of the automatic lesion detection device of the wireless capsule endoscope of the present invention. DETAILED DESCRIPTION
[0076] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0077] like Figure 1 As shown, the present invention discloses a method for automatically detecting lesions with a wireless capsule endoscope, comprising the following steps:
[0078] A wireless capsule endoscope lesion automatic detection method comprises the following steps:
[0079] S1. Obtain a video file captured by a wireless capsule, convert the video file into an image sequence with a timestamp, perform key frame recognition on the image sequence with a timestamp, obtain the annotation results of the key frames by experts, and use an automatic tracking method to obtain the annotation results of non-key frames, thereby obtaining an annotated data set.
[0080] This step is mainly used to convert the video into an image sequence with a timestamp, perform key frame recognition and data labeling. Specifically:
[0081] Converting a video to a time-stamped image sequence involves the following steps:
[0082] S111, reading the video and frame rate, and converting the video into an image according to a certain video frame counting interval frequency;
[0083] S112 , then applying OCR technology to automatically extract the shooting time in the image to obtain an image sequence with a time stamp.
[0084] Furthermore, since the amount of image data after video conversion is large and the similarity between images is high, it is necessary to remove redundant frames through key frame recognition, and the representative sequences retained are used as key frames of the video. The image set with high similarity to the key frame is called the similar interval of the key frame. Key frame recognition includes the following steps:
[0085] S121, extracting features from each frame of the image sequence with the timestamp, thereby obtaining image features. Optionally, extracting features from each frame of the image may be performed by calculating an image hash value, or by using a pre-trained convolutional neural network, etc.;
[0086] S122, based on the image features, calculate the similarity of the features of two adjacent frames of images. When the similarity is less than a set threshold, the current frame image is used as a key frame; otherwise, the current frame image is used as a similar frame of the previous key frame and moved to the next frame. Optionally, the similarity measurement method can be a difference hash algorithm (DHash), cosine distance (CosineDistance), Euclidean distance (Euclidean Distance), Pearson Correlation coefficient (Pearson Correlation), etc.;
[0087] S123, repeating the similarity calculation and comparison steps until the entire image sequence is traversed, thereby obtaining a key frame and a similar interval sequence.
[0088] Furthermore, this application designs the use of the "expert annotation + automatic tracking" method to obtain annotated data sets, with the goal of obtaining annotated data of greater value at a lower expert annotation cost. The data set includes all images with lesions and corresponding label files. The label files include the names of the images with lesions, the coordinates of the lesion locations, and the lesion categories. The specific annotation steps are as follows:
[0089] S131, perform key frame identification based on the above method. Then perform expert annotation of key frames, and screen out key frames with lesions based on the expert's annotation results of key frames. Specifically, for the key frame sequence, the "expert annotation" method is used to manually annotate the lesion information. Among them, the lesion information includes the location coordinates of the lesion area and the lesion category;
[0090] S132, automatic tracking of similar interval sequences. For each key frame with lesions, a target tracking algorithm is used to automatically match all lesion areas of the key frame in the similar interval sequence, thereby achieving automatic labeling of lesions in all similar intervals. The target tracking algorithm may be a template matching algorithm, a deep network end-to-end target tracking algorithm, etc.
[0091] Output the labeled data set. Save all images with lesions and corresponding labels to obtain the labeled data set.
[0092] S2. Use the labeled data set as training data to train a lesion detection model, obtain the wireless capsule endoscope video to be detected, and perform lesion detection on the wireless capsule endoscope video to be detected based on the trained lesion detection model. The obtained lesion detection results include the timestamp, lesion location, lesion category and confidence of all images with lesions, and the confidence indicates the probability that the image with lesions belongs to the corresponding lesion category.
[0093] Among them, based on the trained lesion detection model, lesion detection is performed on the wireless capsule endoscope video to be detected, including:
[0094] S211, converting the wireless capsule endoscope video to be detected into a sequence of images to be detected with a timestamp according to the method described in S1;
[0095] S212. Load the trained lesion detection model, perform lesion detection on all images in the image sequence to be detected, and then output the detection result with only the lesion image as the lesion detection result of the wireless capsule endoscope video to be detected, wherein the detection result with the lesion image includes the image timestamp, lesion location, lesion category and confidence.
[0096] Specifically, the role of S2 is to train the lesion detection model and output the lesion detection results of the wireless capsule endoscope video to be detected. Lesion detection can be defined as a target detection problem, which is used to detect the lesion information of the image, including the lesion location, lesion category and confidence. The target detection model used in training can be a single-stage target detection algorithm (such as YOLO, SSD, etc.) or a two-stage target detection algorithm (such as R-CNN, FastR-CNN, FasterR-CNN, Mask R-CNN, etc.).
[0097] After the model training is completed, the video shot by the wireless capsule endoscope to be detected is input, and the video is converted into an image sequence with a timestamp according to the method described in the pre-processing; secondly, the lesion detection model is loaded to predict the lesion information of the entire image sequence; finally, only the detection results of the lesion image are output, including the timestamp, lesion location, lesion category and confidence.
[0098] S3. According to the lesion detection results of the wireless capsule endoscope video to be detected, all images with lesions are sorted from small to large according to the confidence level, and images with confidence levels less than a certain threshold range are used as "difficult example" samples. The correction results of experts on the "difficult example" samples are obtained to obtain the "missed" lesion information and the "falsely detected" lesion information, and then a "missed detection library" and a "falsely detected library" are generated. At the same time, the lesion detection model is actively learned based on the expert's correction results on the "difficult example" samples.
[0099] Specifically, the purpose of step S3 is to mark a small number of samples to maximize the performance benefits of the model. That is, to obtain "difficult" samples through confidence ranking, and to hand them over to experts for correction and marking, and then obtain the missed detection library and the false detection library composed of "missed detection" and "false detection" areas.
[0100] According to the results of lesion detection, all images with lesions are sorted from small to large according to the confidence level. The smaller the confidence level, the smaller the probability of predicting that the target box is a certain lesion, and the greater the uncertainty of the prediction result. Therefore, images with confidence levels less than a certain threshold range are regarded as "difficult examples" and handed over to experts for manual correction.
[0101] Experts manually correct the annotations of "difficult" samples. When experts determine that the lesion detection model prediction is incorrect, there are mainly the following two types of misdetection situations: the first is "missed detection", that is, the area determined by the expert to be the lesion, but the model did not detect it); the second is "false detection", that is, the model mistakenly identifies non-lesion areas such as residues and bubbles as lesions. For these two situations, the "missed" lesion information (lesion location and lesion category manually marked by experts) and the "falsely detected" lesion information (lesion location predicted by the lesion detection model) are recorded, and the corresponding area of the image is cropped according to the lesion position coordinates to obtain the "missed detection library" and "false detection library" respectively. The steps for obtaining the "missed detection library" and "false detection library" in this application include:
[0102] S311, extracting the "missed" lesion information, cropping the corresponding area of the image according to the lesion position coordinates and storing it in the "missed library", wherein the "missed" lesion information includes the lesion position and lesion category manually marked by experts;
[0103] S312, extracting the "falsely detected" lesion information, cropping the corresponding area of the image according to the lesion position coordinates and storing it in the "falsely detected library", wherein the "falsely detected" lesion information includes the lesion position predicted by the lesion detection model.
[0104] S4. Synthesize the data in the "missed detection library" and the "false detection library" to generate a new synthetic data set, and iteratively optimize the lesion detection model based on the synthetic data set to fine-tune the lesion detection model weight. By iteratively optimizing and fine-tuning the model weight, the missed detection rate and false detection rate are continuously reduced, thereby improving the model performance.
[0105] Specifically, step S4 is mainly used to expand the number of labeled samples, thereby solving the problem of small number of positive examples and small target area in target detection. The steps for generating the synthetic data set are as follows:
[0106] S411, select a background image. Randomly select an image from the original image sequence as the background image;
[0107] S412, select the target image. Randomly select M images from the missed detection library and randomly select N images from the false detection library as the target image;
[0108] S413, target graph data enhancement step. The target graph is randomly enhanced by data enhancement, and the data enhancement method may be scaling, flipping, rotating, etc.;
[0109] S414, step of determining the synthesis position. In order to facilitate the target detection model to identify the target area, it should be avoided to synthesize the target image into the background area with too dark brightness. Therefore, the background image is first converted from the RGB color space to the HSV color space, and the V channel is extracted, because the V parameter represents the brightness value, that is, the brightness of the color. Through the brightness binarization processing, the area with lower brightness in the background image can be removed, and the remaining area with higher brightness in the background image is used as the available synthesis area. The target image randomly selects the synthesis position in the background image, and it only needs to ensure that the target image is completely within the available area;
[0110] S415, image fusion. In the process of synthesizing the target image into the background image, in order to make the synthesized image more natural and keep the synthesis boundary seamless, the present invention adopts an image fusion method to achieve a smooth transition between images. The image fusion algorithm can be Poisson Blend, Laplacian Pyramid fusion, etc. Among them, in addition to the original lesions in the background image, the label of the synthesized image should include the newly synthesized lesion information from the missed detection library;
[0111] S416, obtaining a synthetic data set. Repeat steps S411 to S415 for K times, and obtain a synthetic data set consisting of K images. The synthetic data set will be reused for training the lesion detection model, and the model weights will be fine-tuned through iterative optimization to continuously reduce the missed detection rate and false detection rate, thereby improving the model performance.
[0112] The solution of the present invention is further described below through specific application examples.
[0113] like Figure 2The figure shows the overall execution flow of the embodiment. It mainly includes the pre-processing stage, the training and prediction stage of the lesion detection model, the active learning and data synthesis stage, and the fine-tuning stage of the lesion detection model. The specific description is as follows:
[0114] The pre-processing stage mainly includes:
[0115] Step 1: Convert video to image sequence with timestamp. First read the input video file, get the video frame rate FPS = 15, set the time interval interval = 1 / 15s, and use the VideoCapture class provided by OpenCV to convert the video into an image sequence. Then convert the image into a grayscale image, call the Pytesseract library to automatically identify the shooting time in the image, and obtain a total of 120,000 image sequences with timestamps.
[0116] Step 2: Key frame identification. This embodiment uses Average Hash, one of the perceptual hash algorithms, to extract image features, and generates an 8-byte hash value for each image. The first image is set as the key frame, and the similarity between images is measured by calculating the average hash value difference between adjacent frames. Images similar to the key frame are called similar intervals of the key frame. Figure 3 The key frame recognition method shown in the figure identified 735 key frame sequences in total.
[0117] Step 3: Data labeling. For the key frame sequence, experts label the lesion location coordinates and lesion categories. The labeling tool uses LabelMe. Each image with a lesion will generate a .json file. The label and points information in the file are used to record the lesion location coordinates and lesion category. Among them, the lesion location coordinate bbox is marked in the form of a rectangular box, and the i-th bbox format is a list bbox i =[xmin i ,ymin i ,xmax i ,ymax i ],xmin i and ymin i Indicates the coordinates of the upper left corner of the i-th rectangle, xmax i and ymax i Indicates the coordinates of the lower right corner of the i-th rectangular box. Batch extract lesion information from .json files, and finally output all .txt annotation files with lesion keyframe images. Each line record format is [img_name, [bbox1], lesion_cls1, ..., [bbox i ],lesion_cls i ] Among them, img_name represents the image name, bbox iand lesion_cls i Indicates the i-th lesion location coordinates and the corresponding lesion type. In this example, the lesion categories contained in the dataset are mainly ulcers, so lesion_cls i =1, indicating that the lesion type is ulcer.
[0118] For similar interval sequences, when the corresponding key frame has a lesion, the target tracking algorithm of template matching is used to find the area that best matches the key frame lesion in the similar interval sequence, so as to realize the automatic labeling of lesions in similar intervals. Specifically, the original image I (x, y) is the image in the similar interval sequence, and the template T (x', y') is the key frame lesion area. By sliding, T is covered at each position of I, and the comparison calculation results of the template and the covered image are stored in the matrix image R (x, y). The value of each position (x, y) in R represents the matching metric value calculated after the pixel of the covered image and the template is calculated with the point as the upper left corner. The matching metric method adopts the standardized square sum of difference matching, as shown in formula (1). The closer the value is to 0, the higher the matching degree.
[0119]
[0120] In summary, by combining expert annotation and automatic tracking, the annotation of key frame sequences and similar interval sequences was completed, and finally a labeled data set of 3883 images was obtained.
[0121] The training and prediction phases of the lesion detection model mainly include:
[0122] Step 1: Training of lesion detection model. In this embodiment, the amount of collected data is 3883, and the data set is randomly divided according to the ratio of 8:1:1 to obtain 3105 training set samples, 389 validation set samples and 389 test set samples. The lesion detection model adopts the single-stage target detection algorithm YOLO v4, and the network structure is as follows: Figure 4 As shown. The input image size is 416×416×3, and the three main components of the YOLO v4 architecture include:
[0123] (1) Backbone: CSPDarkNet53 is selected as the backbone network to extract features;
[0124] (2) Neck: SPP and PANet are used. The SPP (Spatial Pyramid Pooling) structure is mixed into the convolution of the last feature layer of CSPdarknet53, and pooling kernels of different sizes (13×13, 9×9, 5×5, 1×1) are used for multi-scale fusion. At the same time, the PANet (Path aggregation network) structure is used for the feature layer, and the feature pyramid is used to repeatedly extract features and pass them to the prediction layer;
[0125] (3) Head: Use the YOLO v3 classification regression layer to predict image features and generate bounding boxes and predicted categories.
[0126] Figure 5 and Figure 6 They respectively represent the changes in the loss curves of the training set and the validation set during the model training process, and the number of iterations Epoch = 100.
[0127] Step 2: Lesion detection. The test set sample size is 389 images, and the ground-truth contains 543 real lesion frames, all of which are ulcers. The performance of the lesion detection model is verified in the test set, and mAP (mean average precision) = 64.71%, TP (True Positive) = 384, FP (False Positive) = 71, as shown in the figure. Figure 7-9 shown.
[0128] The active learning and data synthesis phase mainly includes:
[0129] Step 1: Confidence sorting. Sort the model prediction confidence, and take samples with confidence greater than 0.5 as easy-to-classify samples, as shown in Figure 10. The lesion results automatically detected by the model in the figure are basically the same as the lesion results annotated by the expert, indicating that the confidence of the model output prediction result is high, that is, samples that are easy to detect correctly. Such samples will not significantly improve the model performance. Therefore, for high-confidence easy-to-classify samples, no expert intervention is required.
[0130] Step 2: Expert correction. Samples with a confidence level less than or equal to 0.5 will be considered difficult to classify, and experts are required to correct the prediction results to improve model performance. Figures 11 and 12 show examples of misdetected samples and missed samples after expert correction, respectively. Fig.11a , Fig.11c as well as Fig.11b The box area on the right side is the real result corrected by the expert. Fig.11b The box area on the left side of the middle is both the actual result corrected by the expert and the result automatically predicted by the model. Figure 12b , Fig.12c as well as Fig.12a The box area on the left is the model's automatic prediction result. Fig.12a The box area on the right side of the figure is both the real result corrected by the expert and the result automatically predicted by the model. By cutting according to the position of the box, we can get the "missed detection library" and "false detection library" respectively.
[0131] Step 3: Synthesize the data set. Use the original image as the background image, and the images in the "missed detection library" and "false detection library" as the target images. The image synthesis process is as follows:
[0132] (1) Randomly select background images and target images, as shown in Figure 13(a) and Figure 13(b);
[0133] (2) The target image is randomly augmented with probability p = 0.5. The data augmentation methods include horizontal flipping, vertical flipping, and random scaling of width and height by 0.8 to 2.0 times;
[0134] (3) Convert the background image to HSV color space and extract the brightness V channel for binary processing V[V>150]=255, V[V<=150]=0, that is, set the brightness value greater than 150 to white 255, and set other areas to black 0, such as Fig.14 As shown in the figure, the white area is the available synthesis area mapped back to the original background image;
[0135] (4) The target image after data enhancement is synthesized with the available area of the original background image, and the Poisson fusion algorithm is used to seamlessly fuse the images through the Poisson equation. The goal is to minimize the gradient change at the boundary, as shown in formula (2).
[0136]
[0137] Among them, Ω represents the original image, v represents the original image gradient, represents the original image boundary, f and Respectively represent the fused image and its gradient, f * represents the target image;
[0138] (5) The synthesized image is as follows Fig.15 As shown in FIG. 1 , the rectangular frame represents the newly added lesion frame after synthesis, and its position coordinates and lesion category are added to the original image label to obtain new annotation data.
[0139] Fig.16 Represents a sample example of the synthetic dataset. The solid-line frame is the original lesion, the dashed-line frame is the missed-detection frame after synthesis, and the dashed-line frame is the false-detection frame after synthesis. In particular, only the solid-line frame and the dashed-line frame represent positive examples, i.e., the lesion area, and its position coordinates and lesion type will be used as labels for the synthetic image. Therefore, the synthetic dataset will greatly increase the number and diversity of positive samples.
[0140] The stage of fine-tuning the lesion detection model mainly includes:
[0141] The synthesized dataset is used to fine-tune the weights of the original lesion detection model. Fig.17 The fine-tuned model is loaded as shown, and the detection results of some samples are displayed. The figure includes two box areas, one of which and the text information on the box indicate the model's prediction of the lesion box location, lesion type and confidence; the other box area and the text information on the box indicate the lesion box location and lesion type of the lesion ground-truth. The model obtained after retraining has an evaluation index of mAP=74.02% on the test set, which is 14.39% higher than the original model mAP=64.71%. Fig.18 After fine-tuning, the model can not only correctly detect small and inconspicuous lesions, but also reduce the misjudgment rate of negative examples such as residues, greatly improving the detection accuracy, further proving the beneficial effect of the method proposed in the present invention.
[0142] The present invention also discloses a wireless capsule endoscope lesion automatic detection device, such as Fig.19 As shown, including:
[0143] A pre-processing module is used to obtain the video file collected by the wireless capsule, convert the video file into an image sequence with a time stamp, perform key frame recognition on the image sequence with a time stamp, obtain the annotation results of the key frames by experts, and use an automatic tracking method to obtain the annotation results of non-key frames, so as to obtain an annotated data set;
[0144] A lesion detection module, which is used to train a lesion detection model using the annotated data set as training data, and to detect an image sequence to be detected by the trained lesion detection model to obtain lesion information of the image sequence to be detected, wherein the image sequence to be detected is an image sequence with a timestamp obtained by converting a video captured by a wireless capsule endoscope, and the lesion detection model outputs a detection result of an image with a lesion, wherein the detection result of an image with a lesion includes a timestamp, a lesion location, a lesion category and a confidence level of all images with a lesion, and the confidence level indicates the probability that the image with a lesion belongs to a corresponding lesion category;
[0145] Active learning module, which is used to sort all images with lesions from small to large according to the confidence level according to the detection results, and take images with confidence levels less than a certain threshold range as "difficult examples" samples, obtain the correction results of experts on the "difficult examples" samples, obtain the information of "missed" lesions and "falsely detected" lesions, and then generate a "missed detection library" and a "falsely detected library";
[0146] The data synthesis module is used to generate a synthetic data set based on the data in the "missed detection library" and the "false detection library", retrain the lesion detection model based on the synthetic data set, and fine-tune the model weights through iterative optimization.
[0147] As for the embodiments of the present invention, since they correspond to the embodiments above, the description is relatively simple. For the relevant similarities, please refer to the description of the above embodiments, which will not be described in detail here.
[0148] The present invention also discloses a computer-readable storage medium, in which a computer instruction set is stored; when the computer instruction set is executed by a processor, a wireless capsule endoscope lesion automatic detection method as described in any one of the above items is implemented.
[0149] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0150] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0151] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A wireless capsule endoscope lesion automatic detection method, characterized in that: The following steps are involved: S1. Obtain a video file collected by a wireless capsule, convert the video file into an image sequence with a timestamp, perform key frame recognition on the image sequence with a timestamp, obtain an expert's annotation results for the key frames, and use an automatic tracking method to obtain an annotation result for a non-key frame, thereby obtaining an annotated data set; S2, using the labeled data set as training data to train a lesion detection model, obtaining a wireless capsule endoscope video to be detected, and performing lesion detection on the wireless capsule endoscope video to be detected based on the trained lesion detection model, and the obtained lesion detection results include the timestamps, lesion locations, lesion categories and confidences of all images with lesions, wherein the confidences represent the probability that the images with lesions belong to the corresponding lesion category; S3, according to the lesion detection result of the wireless capsule endoscope video to be detected, all images with lesions are sorted from small to large according to the confidence, and images with confidence less than a certain threshold range are used as "difficult example" samples, and the correction results of experts on the "difficult example" samples are obtained to obtain the "missed" lesion information and the "falsely detected" lesion information, and then generate a "missed detection library" and a "falsely detected library", and at the same time, the lesion detection model is actively learned based on the expert's correction results on the "difficult example" samples; S4. Performing synthesis processing on the data in the "missed detection library" and the "false detection library" to generate a new synthetic data set, and iteratively optimizing the lesion detection model based on the synthetic data set to fine-tune the lesion detection model weights.
2. The method for automatic lesion detection using wireless capsule endoscope according to claim 1, characterized in that: S1 converts the video file into an image sequence with a timestamp, including: S111, converting the video into an image according to a preset video frame counting interval frequency; S112: Automatically extract the shooting time in the image based on OCR technology, so as to obtain an image sequence with a time stamp.
3. The method for automatic lesion detection by wireless capsule endoscope according to claim 1, characterized in that: In S1, key frame recognition is performed on the image sequence with the timestamp, including: S121, performing feature extraction on each frame of the image in the image sequence with the timestamp, so as to obtain image features; S122, calculating the similarity of the image features of two adjacent frames based on the image features, and when the similarity is less than a set threshold, taking the current frame image as a key frame; otherwise, taking the current frame image as a similar frame of the previous key frame and moving to the next frame; S123, repeating the similarity calculation and comparison steps until the entire image sequence is traversed, thereby obtaining a key frame and a similar interval sequence.
4. The method for automatic lesion detection by wireless capsule endoscope according to claim 3, characterized in that: In S1, an automatic tracking method is used to obtain the annotation results of non-key frames, including: S131, based on the expert's annotation results of the key frames, filter out the key frames with lesions; S132. For each key frame with lesions, a target tracking algorithm is used to automatically match all lesion regions of the key frame in a similar interval sequence, thereby achieving automatic labeling of lesions in all similar intervals.
5. The method for automatic lesion detection using wireless capsule endoscope according to claim 1, characterized in that: In S2, lesion detection is performed on the wireless capsule endoscope video to be detected based on the trained lesion detection model, including: S211, converting the wireless capsule endoscope video to be detected into a sequence of images to be detected with a timestamp according to the method described in S1; S212. Load the trained lesion detection model, perform lesion detection on all images in the image sequence to be detected, and then output the detection result with only the lesion image as the lesion detection result of the wireless capsule endoscope video to be detected, wherein the detection result with the lesion image includes the image timestamp, lesion location, lesion category and confidence.
6. The method for automatic lesion detection using wireless capsule endoscope according to claim 1, characterized in that: S3 obtains the correction results of experts on "difficult examples" and obtains the information of "missed" and "falsely detected" lesions, and then generates a "missed detection library" and a "falsely detected library", including: S311, extracting the "missed" lesion information, cutting the corresponding area of the image according to the lesion position coordinates and storing it in the "missed library", wherein the "missed" lesion information includes the lesion position and lesion category manually marked by experts; S312, extracting the "falsely detected" lesion information, cropping the corresponding area of the image according to the lesion position coordinates and storing it in the "falsely detected library", wherein the "falsely detected" lesion information includes the lesion position predicted by the lesion detection model.
7. The method for automatic lesion detection using wireless capsule endoscope according to claim 1, characterized in that: In S4, the data in the "missed detection library" and the "false detection library" are synthesized to generate a new synthetic data set, including: S411, randomly selecting an image from the image sequence to be detected as a background image; S412, randomly select M images from the "missed detection library" and randomly select N images from the "false detection library" to generate a target image; S413, performing random data enhancement on the target image, wherein the data enhancement method is scaling, flipping or rotating; S414, determining an available synthesis area that meets the brightness requirement in the background image by color space conversion, and randomly selecting a synthesis position in the available synthesis area; S415, using an image fusion method to fuse the target image into the background image to generate a composite image, wherein the label of the composite image includes the original lesion in the background image and the lesion information newly integrated from the "missed detection library"; S416. Repeat S411 to S415K times to obtain a synthetic data set consisting of K images.
8. A wireless capsule endoscope lesion automatic detection device, used to implement the automatic detection method according to any one of claims 1 to 7, characterized in that: include: A pre-processing module is used to obtain the video file collected by the wireless capsule, convert the video file into an image sequence with a time stamp, perform key frame recognition on the image sequence with a time stamp, obtain the annotation results of the key frames by experts, and use an automatic tracking method to obtain the annotation results of non-key frames, so as to obtain an annotated data set; A lesion detection module is used to train a lesion detection model using the annotated data set as training data, obtain a wireless capsule endoscope video to be detected, and perform lesion detection on the wireless capsule endoscope video to be detected based on the trained lesion detection model. The obtained lesion detection results include the timestamps, lesion locations, lesion categories and confidence levels of all images with lesions, wherein the confidence level indicates the probability that the image with lesions belongs to the corresponding lesion category; An active learning module, which is used to sort all images with lesions from small to large according to the confidence level according to the lesion detection results of the wireless capsule endoscope video to be detected, and use images with confidence levels less than a certain threshold range as "difficult example" samples, obtain the correction results of experts on the "difficult example" samples, obtain the "missed" lesion information and the "falsely detected" lesion information, and then generate a "missed detection library" and a "falsely detected library", and at the same time, based on the expert's correction results on the "difficult example" samples, actively learn the lesion detection model; The data synthesis module synthesizes the data in the "missed detection library" and the "false detection library" to generate a new synthetic data set, and iteratively optimizes the lesion detection model based on the synthetic data set to fine-tune the lesion detection model weight.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer instruction set; when the computer instruction set is executed by the processor, the method for automatic detection of lesions by wireless capsule endoscope is implemented as described in any one of claims 1-7.
Citation Information
Patent Citations
Interaction method and device based on multi-feature recognition and computer equipment
CN112509148A
Gastric cancer focus detection method and device based on convolutional neural network
CN112634261A
Method for searching difficult case images, model training method, equipment and storage medium
CN112733666A
KR20190090150A
