A capsule endoscope image lesion recognition device and training method
Through the multi-teacher distillation method, the knowledge of complex models is efficiently passed to the student network, solving the problem of low diagnostic accuracy of lightweight models, and achieving high-precision and high-efficiency capsule endoscopic image lesion recognition, suitable for multiple types of diseases real-time diagnosis and lesion segmentation.
Patent Information
- Application Number
- CN202110640667.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-09
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2041-06-09
AI Technical Summary
The existing lightweight image recognition models have low diagnostic accuracy in clinical disease diagnosis, cannot meet the needs of real-time diagnosis, and have low knowledge distillation efficiency, making it difficult to efficiently pass on the knowledge of complex models to student networks.
The multi-teacher distillation method is used to input the capsule endoscopic images into multiple complex teacher networks to generate feature maps and significance maps. The student network is trained through cross-entropy loss and knowledge distillation loss functions to make it close to the diagnostic accuracy of the complex model.
Achieve high-precision and high-efficiency lightweight model, which can achieve high accuracy in the diagnosis of multiple types of diseases, reduce overfitting problems caused by limited training samples, and support disease diagnosis and lesion areas of capsule endoscopic images segmentation.
Smart Images
Figure CN113538334B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to image recognition, and in particular to a capsule endoscope image lesion recognition device and a training method. Background Art
[0002] At present, deep learning models have achieved high accuracy in a variety of medical disease diagnosis tasks. However, high-precision models have high complexity and require a lot of computing power, memory, and inference time. However, in clinical disease diagnosis, hospitals have limited computing resources and cannot meet the needs of real-time diagnosis. In order to improve inference efficiency and achieve real-time diagnosis, lightweight models need to be deployed in clinical practice. However, existing lightweight models generally have poor feature expression capabilities and low diagnostic accuracy. In order to truly achieve real-time diagnosis in clinical diagnosis, it is necessary to develop lightweight models with high diagnostic accuracy. The knowledge distillation framework is also called the teacher-student network, and the complex teacher network has stronger fitting ability. Existing methods transfer the knowledge of the teacher network to the lightweight student network in the form of prediction results, feature maps, attention maps, and relationships between samples, so that the lightweight model can achieve diagnostic accuracy close to that of the complex model. How to efficiently distill the knowledge of the teacher network into the student network and what knowledge to distill into the student network are the main research topics of knowledge distillation.
[0003] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention
[0004] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a capsule endoscope image lesion recognition device and training method to achieve high-precision and high-efficiency disease diagnosis.
[0005] To achieve the above object, the present invention adopts the following technical solutions:
[0006] A capsule endoscope image lesion recognition device comprises a computer program storage medium and a processing device, wherein the processing device implements a training process and a testing process when executing the computer program;
[0007] The training process includes the following steps:
[0008] S1, input capsule endoscopy images to the model’s teacher networks 1 to N;
[0009] S2. For each capsule endoscopy image, each teacher network generates its own feature map h*w*c, where h, w, c are the height, width and number of channels of the feature map respectively, and obtains the saliency map of the image by averaging different channels of the feature map; the model performs classification or segmentation tasks;
[0010] S3, calculate the cross entropy loss for the model's prediction results and the true labels;
[0011] S4, minimize the cross entropy loss and continuously update the model parameters until the cross entropy loss no longer decreases and the model converges;
[0012] S5, merge the feature maps, saliency maps and prediction results generated by all teacher networks as guidance information for the integrated teacher network;
[0013] S6, inputting the capsule endoscopy image into the student network, and passing the feature map, saliency map and prediction result of the integrated teacher network to the student network through the knowledge distillation algorithm;
[0014] S7, obtaining the loss function of the student network according to the cross entropy function and the knowledge distillation loss functions at all levels, and training the student network by optimizing the loss function until convergence;
[0015] The testing process includes the following steps:
[0016] T1. Input the capsule endoscopy image into the trained student network to obtain the predicted probability of each category of the image. The category with the highest probability is the predicted category to which the test image belongs.
[0017] T2. Generate a saliency map for each test image. The lesion area of the image can be roughly marked according to the saliency value, providing explanation for the model's prediction.
[0018] Further:
[0019] In step S1, the teacher network is selected from any one of DenseNet121, ResNet50, and NASNet.
[0020] In step S2, the disease-related areas and background areas are determined according to the saliency of different areas in the saliency map; for the classification task, the last layer feature map of the model is subjected to global average pooling and full connection layer mapping to obtain the category prediction probability (p 1 ,p 2 ,…,p C ), where p iis the probability that the image is judged as category i, C is the total number of categories, the predicted probabilities of all categories add up to 1, and the category with the largest predicted probability is the category that the image is finally judged to be, as the prediction result; for segmentation tasks, the model generates a prediction result for each pixel.
[0021] In step S4, the cross entropy loss is minimized based on a gradient descent algorithm.
[0022] In step S7, the loss function of the student network is obtained by adding the cross entropy function and the knowledge distillation loss functions at all levels.
[0023] The testing process also includes the following steps:
[0024] T2. Generate a saliency map for each test image; preferably, also mark the lesion area of the image according to the saliency value to provide explanation for the model's prediction.
[0025] A training method for capsule endoscopy image lesion recognition network based on multi-teacher distillation, comprising the following steps:
[0026] S1, input capsule endoscopy images to the model’s teacher networks 1 to N;
[0027] S2. For each capsule endoscopy image, each teacher network generates its own feature map h*w*c, where h, w, c are the height, width and number of channels of the feature map respectively, and obtains the saliency map of the image by averaging different channels of the feature map; the model performs classification or segmentation tasks;
[0028] S3, calculate the cross entropy loss for the model's prediction results and the true labels;
[0029] S4, minimize the cross entropy loss and continuously update the model parameters until the cross entropy loss no longer decreases and the model converges;
[0030] S5, merge the feature maps, saliency maps and prediction results generated by all teacher networks as guidance information for the integrated teacher network;
[0031] S6, inputting the capsule endoscopy image into the student network, and passing the feature map, saliency map and prediction result of the integrated teacher network to the student network through the knowledge distillation algorithm;
[0032] S7. According to the cross entropy function and the knowledge distillation loss functions at each level, the loss function of the student network is obtained, and the student network is trained by optimizing the loss function until convergence.
[0033] A capsule endoscope image lesion recognition device includes a computer program storage medium and a processing device. When the processing device executes the computer program, a test process is implemented. The test process includes the following steps:
[0034] T1. Input the capsule endoscopy image into the student network trained using the method described above to obtain the predicted probability of each category of the image. The category with the largest probability is the predicted category to which the test image belongs.
[0035] The testing process also includes the following steps:
[0036] T2. Generate a saliency map for each test image; preferably, roughly mark the lesion area of the image according to the saliency value to provide explanation for the model's prediction.
[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps performed by the device or the method.
[0038] The present invention has the following beneficial effects:
[0039] The present invention provides a lightweight capsule endoscope image lesion recognition device and training method based on multi-teacher distillation, wherein capsule endoscope images are input into multiple complex teacher networks, and each network outputs a diagnosis result, a feature map and a saliency map. Multiple teacher networks are integrated to obtain a more powerful lesion recognition network. Afterwards, the diagnosis results, feature maps and saliency maps of the integrated teacher networks are used as supervisory signals to provide additional supervision for the student network, thereby transferring the knowledge of the teacher network to the student network. Thus, the present invention can obtain a lightweight model with high diagnostic accuracy. In the present invention, the knowledge distillation algorithm is used to compress the expression ability of the complex network into the lightweight network, thereby achieving high-precision and high-efficiency diagnosis. Each teacher network in the present invention can learn the diagnosis of a class of diseases, so that a more accurate diagnosis effect can be achieved on the class of diseases, and the student network can achieve high-precision real-time diagnosis of multiple types of diseases by learning multiple teacher networks that are proficient in the diagnosis of different diseases. The present invention can be used for disease diagnosis of capsule endoscope images and segmentation of lesion areas, such as bleeding, ulcers, and polyps. By using the present invention, existing high-precision algorithms can also be deployed on lightweight networks for clinical applications through knowledge distillation. In addition, through knowledge distillation, the teacher network can provide soft labels for the student network, thereby alleviating the overfitting problem caused by limited training samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 The figure is a schematic diagram of an implementation principle of an embodiment of the present invention. DETAILED DESCRIPTION
[0041] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.
[0042] In order to develop a lightweight model with high diagnostic accuracy, the present invention proposes a lightweight capsule endoscopy image lesion recognition algorithm based on multi-teacher distillation. The capsule endoscopy image is input into multiple complex teacher networks, and each network outputs a diagnosis result, a feature map, and a saliency map. Multiple teacher networks are integrated (the feature maps of different networks at all levels can be merged, the saliency maps are averaged, and the prediction results are weighted averaged according to the accuracy of each network) to obtain a more powerful lesion recognition network. Afterwards, the diagnosis results of the integrated teacher network, the feature map and the saliency map are used as supervision signals to provide additional supervision for the student network, thereby transferring the knowledge of the teacher network to the student network.
[0043] The model trained by the present invention can also be applied to the segmentation of lesion areas. For disease segmentation tasks, most existing neural networks can only segment one disease. If multiple types of diseases and their segmentation labels are used for the training of a single model at the same time, the segmentation and diagnosis accuracy of the model will decrease. Each teacher network in the present invention can learn the diagnosis of a type of disease, so that a more accurate diagnosis effect can be achieved on this type of disease (which can be compared to each teacher in a school being responsible for a subject). By learning from multiple teacher networks that are proficient in the diagnosis of different diseases, the student network can achieve high-precision real-time diagnosis of multiple types of diseases.
[0044] See also Figure 1 , some embodiments include the following typical but non-limiting training and testing processes.
[0045] The training process includes the following steps:
[0046] Step 1: Input the wireless capsule endoscopy images into large-scale teacher networks 1 to N. These teacher networks have relatively complex and diverse network structures (such as DenseNet121, ResNet50, NASNet, etc.).
[0047] Step 2: For each capsule endoscopy image, each teacher network generates a feature map of different sizes (h*w*c, where h, w, c are the height, width and number of channels of the feature map respectively). By averaging the different channels of these feature maps, the saliency map of the image can be obtained. The areas with high saliency are disease-related areas, and the areas with low saliency are background areas. For classification tasks, the last layer of feature maps of the model can be used to obtain the class prediction probability (p 1 ,p 2 ,…,p C ), where p iis the probability that the image is judged as category i, and C is the total number of categories. The predicted probabilities of all categories add up to 1, and the category with the largest predicted probability is the category that the image is finally judged to be. For segmentation tasks, the model generates a prediction result for each pixel.
[0048] Step 3: Calculate the cross entropy loss for the model’s predictions and true labels.
[0049] Step 4: Minimize the cross entropy loss based on the gradient descent algorithm and continuously update the model parameters until the cross entropy no longer decreases and the model converges.
[0050] Step 5: The feature maps, saliency maps, and prediction results generated by all teacher networks are merged through the multi-teacher integration module as guidance information for the integrated teacher network.
[0051] Step 6: Input the wireless capsule endoscope image into the lightweight student network, and pass the feature map, saliency map, and prediction results of the integrated teacher network to the student network through the knowledge distillation algorithm (Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in aneural network." arXiv preprint arXiv:1503.02531(2015).), so that the performance of the lightweight student network is close to that of the powerful model of multi-teacher network integration.
[0052] Step 7: The loss function of the student network is the sum of the cross entropy function and the knowledge distillation loss function at each level. The student network is trained until convergence by optimizing this loss function.
[0053] The testing process includes the following steps:
[0054] Step 1: Only keep the lightweight student network. Input the wireless capsule endoscopy image into the trained student network to obtain the prediction probability of each category of the image. The category with the largest probability is the predicted category to which the test image belongs.
[0055] Step 2: Generate a saliency map for each test image. The lesion area of the image can be roughly marked according to the saliency value to provide explainability for the model's prediction.
[0056] like Figure 1As shown in the figure, capsule endoscopy images are used to train N complex teacher networks (such as DenseNet121, ResNet50, NASNet, etc.). These teacher networks have complex parameters and models and have strong expressive power. After the training, the outputs of the N teacher networks are integrated to obtain an integrated teacher network with stronger expressive power. Specifically, the feature maps of different teacher networks at all levels are merged, the saliency maps are averaged, and the prediction results are weighted averaged according to the accuracy of each network. Since different networks will learn different knowledge, different errors will occur in the prediction process. Integrating them can effectively correct each other's errors, thereby obtaining an integrated teacher network with stronger expressive power and more stable. Afterwards, the diagnostic results, feature maps and saliency maps of the trained integrated teacher network are used as supervisory signals to provide additional supervision to the student network, thereby transferring the knowledge of the teacher network to the student network.
[0057] The following further describes the training and testing process of a specific embodiment of the present invention.
[0058] During the testing phase, the test images are only diagnosed through a lightweight student network (such as ResNet18, Efficient Net, etc.), which can greatly reduce the time required for diagnosis and meet the needs of real-time diagnosis in clinical practice.
[0059] Training process
[0060] Step 1: Input the wireless capsule endoscopy images into large-scale teacher networks 1 to N. These teacher networks have relatively complex and diverse network structures (such as DenseNet121, ResNet50, NASNet, etc.).
[0061] Step 2: For each capsule endoscopy image, each teacher network generates a feature map of different sizes (h*w*c, where h, w, c are the height, width and number of channels of the feature map respectively). By averaging the different channels of these feature maps, the saliency map of the image can be obtained. The areas with high saliency are disease-related areas, and the areas with low saliency are background areas. For classification tasks, the last layer of feature maps of the model can be used to obtain the class prediction probability (p 1 ,p 2 ,…,p C ), where p i is the probability that the image is judged as category i, and C is the total number of categories. The predicted probabilities of all categories add up to 1, and the category with the largest predicted probability is the category that the image is finally judged to be. For segmentation tasks, the model generates a prediction result for each pixel.
[0062] Step 3: Calculate the cross entropy loss for the model’s predictions and true labels.
[0063]
[0064] Step 4: Minimize the cross entropy loss based on the gradient descent algorithm and continuously update the model parameters until the cross entropy no longer decreases and the model converges.
[0065] Step 5: The feature maps, saliency maps, and prediction results generated by all teacher networks are combined through the multi-teacher integration module as guidance information for the integrated teacher network. Specifically, the different channels of the feature maps generated by multiple teacher networks are averaged in turn to obtain the integrated feature map. The saliency maps of multiple teacher networks are averaged to obtain the integrated saliency map. After averaging the prediction results, the integrated training set image prediction results are obtained.
[0066] Step 6: Input the wireless capsule endoscope image into the lightweight student network, and pass the feature map, saliency map, and prediction results of the integrated teacher network to the student network through the knowledge distillation algorithm, so that the performance of the lightweight student network is close to the powerful model of multi-teacher network integration. The loss function of the feature map distillation is L FCE =1 / 2||F t -F s || 2 .F in the formula t and F s are the feature maps of the integrated teacher network and the student network respectively. The loss function of the saliency map distillation is L SCE =1 / 2||S t -S s || 2 .S in the formula t and S s They are the saliency maps of the integrated teacher network and student network respectively. The loss function of the prediction result distillation is L KL =KL(p t ,p s ). p in the formula t and p s are the class prediction probabilities of the teacher and student networks, respectively, and the KL divergence is used to measure the distance between them.
[0067] Step 7: The loss function of the student network is the sum of the cross entropy function and the knowledge distillation loss function at all levels (L = L CE +L FCE +L SCE +L KL ). The student network is trained by optimizing this loss function until convergence.
[0068] Testing Process
[0069] Step 1: Only keep the lightweight student network. Input the wireless capsule endoscopy image into the trained student network to obtain the prediction probability of each category of the image. The category with the largest probability is the predicted category to which the test image belongs.
[0070] Step 2: Generate a saliency map for each test image. The lesion area of the image can be roughly marked according to the saliency value to provide explainability for the model's prediction.
[0071] The background section of the present invention may contain background information about the problem or environment of the present invention, but does not necessarily describe the prior art. Therefore, the content included in the background section is not an admission by the applicant that the prior art is available.
[0072] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of the reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.
Claims
1. A capsule endoscope image lesion recognition device, characterized in that: It includes a computer program storage medium and a processing device, and the processing device implements a training process and a testing process when executing the computer program; The training process includes the following steps: S1, input capsule endoscopy images to multiple teacher networks 1 to N of the model; S2. For each capsule endoscopy image, each teacher network generates its own feature map h*w*c, where h, w, and c are the height, width, and number of channels of the feature map, respectively. The saliency map of the image is obtained by averaging different channels of the feature map. The model performs classification or segmentation tasks. S3, calculate the cross entropy loss for the model's prediction results and the true labels; S4, minimize the cross entropy loss and continuously update the model parameters until the cross entropy loss no longer decreases and the model converges; S5, merge the feature maps, saliency maps and prediction results generated by all teacher networks as guidance information for the integrated teacher network; S6, inputting the capsule endoscopy image into the student network, and passing the feature map, saliency map and prediction result of the integrated teacher network to the student network through the knowledge distillation algorithm; S7, obtaining the loss function of the student network according to the cross entropy function and the knowledge distillation loss functions at all levels, and training the student network by optimizing the loss function until convergence; The testing process includes the following steps: T1. Input the capsule endoscopy image into the trained student network to obtain the predicted probability of each category of the image. The category with the largest probability is the predicted category to which the image belongs.
2. The capsule endoscope image lesion recognition device according to claim 1, characterized in that: In step S1, the teacher network is selected from any one of DenseNet121, ResNet50, and NASNet.
3. The capsule endoscope image lesion recognition device according to claim 1 or 2, characterized in that: In step S2, the disease-related area and the background area are determined according to the saliency of different areas in the saliency map; for the classification task, the last layer feature map of the model is subjected to global average pooling and full connection layer mapping to obtain the category prediction probability of the image (p1, p2, …, p C ), where p i is the probability that the image is judged as category i, C is the total number of categories, the predicted probabilities of all categories add up to 1, and the category with the largest predicted probability is the category that the image is finally judged to be, as the prediction result; for segmentation tasks, the model generates a prediction result for each pixel.
4. The capsule endoscope image lesion recognition device according to claim 1 or 2, characterized in that: In step S4, the cross entropy loss is minimized based on a gradient descent algorithm.
5. The capsule endoscope image lesion recognition device according to claim 1 or 2, characterized in that: In step S7, the loss function of the student network is obtained by adding the cross entropy function and the knowledge distillation loss functions at all levels.
6. The capsule endoscope image lesion recognition device according to claim 1 or 2, characterized in that: The testing process also includes the following steps: T2. Generate a saliency map for each image.
7. A training method for capsule endoscopy image lesion recognition network based on multi-teacher distillation, characterized in that: The steps include: S1, input capsule endoscopy images to multiple teacher networks 1 to N of the model; S2. For each capsule endoscopy image, each teacher network generates its own feature map h*w*c, where h, w, and c are the height, width, and number of channels of the feature map, respectively. The saliency map of the image is obtained by averaging different channels of the feature map. The model performs classification or segmentation tasks. S3, calculate the cross entropy loss for the model's prediction results and the true labels; S4, minimize the cross entropy loss and continuously update the model parameters until the cross entropy loss no longer decreases and the model converges; S5, merge the feature maps, saliency maps and prediction results generated by all teacher networks as guidance information for the integrated teacher network; S6, inputting the capsule endoscopy image into the student network, and passing the feature map, saliency map and prediction result of the integrated teacher network to the student network through the knowledge distillation algorithm; S7. According to the cross entropy function and the knowledge distillation loss functions at each level, the loss function of the student network is obtained, and the student network is trained by optimizing the loss function until convergence.
8. A capsule endoscope image lesion recognition device, characterized in that: The method comprises a computer program storage medium and a processing device, wherein the processing device implements a test process when executing the computer program, and the test process comprises the following steps: T1. Input the capsule endoscopy image into the student network trained by the method as claimed in claim 7 to obtain the prediction probability of each category of the image, and the category with the largest probability is the predicted category to which the image belongs.
9. The capsule endoscope image lesion recognition device according to claim 8, characterized in that: The testing process also includes the following steps: T2. Generate a saliency map for each image.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps performed by the device according to any one of claims 1 to 6, 8-9 are implemented, or the method according to claim 7 is implemented.