Quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation
By introducing a channel attention self-derived distillation mechanism in endoscopic image classification, the problem of forgetting old knowledge in incremental learning is solved, better feature representation and resource saving are achieved, and classification performance is improved.
Patent Information
- Application Number
- CN202411505232.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-10-28
Smart Images
Figure CN119027746B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation. Background Art
[0002] At present, smart healthcare has been identified as an important application field in the planning of the new generation of artificial intelligence. How to build artificial intelligence medical imaging products for endoscopic images has become a hot topic for medical and research institutions. Quasi-incremental learning can continuously learn new knowledge and integrate it into the model to form a general model suitable for all tasks. It can better cope with new needs that emerge at any time in reality and is widely used in various fields. However, the disease dispersion, individual diversity, and quality uncertainty presented by medical images make the quasi-incremental lesion classification task for endoscopic images face multiple challenges. When performing incremental learning, with the introduction of new categories, the loss of old category knowledge may lead to catastrophic forgetting. Therefore, it is of great application value to construct a robust quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation.
[0003] Existing quasi-incremental learning methods are mainly divided into three categories: regularization-based methods, knowledge replay-based methods, and dynamic network-based methods. Regularization-based methods can be subdivided into weight regularization and data regularization. Specifically, weight regularization methods aim to prevent weight drift related to previous tasks, and data regularization focuses on preventing activation drift with the help of knowledge distillation. Knowledge replay-based methods alleviate the model's forgetting of old knowledge by retaining a certain amount of old task data during the incremental learning phase. Dynamic network-based methods dynamically expand the model structure during the incremental learning phase to increase the model capacity. However, this continuously expanding network structure will face the problems of large model size and high computational cost, which is not convenient for deployment and use in actual systems.
[0004] In summary, it is crucial to develop a quasi-incremental lesion classification learning paradigm that can learn new knowledge without forgetting old knowledge and that takes into account both plasticity and robustness. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation, which retains more information of old tasks in the incremental learning stage, thereby more effectively alleviating the problem of catastrophic forgetting. The present invention introduces channel attention self-derivation distillation on the basis of using response distillation, so as to further retain old knowledge. Channel attention self-derivation distillation not only helps to stimulate the discriminative expression of new knowledge, but also can constrain the new classification model and the old classification model to maintain a relatively consistent degree of attention to the feature channels, so as to prompt the classification model to still pay attention to the feature channels that help distinguish the old classes when new tasks arrive, thereby preventing catastrophic forgetting. With the help of channel attention self-derivation distillation, the quasi-incremental lesion classification model proposed by the present invention takes into account both plasticity and robustness, and shows better performance in incremental learning tasks.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0007] A quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation, inputting an endoscopic image belonging to a category already seen by a classification model into a trained classification model to obtain a classification result of the endoscopic image; the training process of the classification model includes:
[0008] Step 1, initialization of the classification model: the classification model includes a feature extractor and a classifier; the pre-trained convolutional neural network is removed from the fully connected layer and the channel attention mechanism is introduced as the feature extractor; the classification model includes an old classification model and a new classification model; the training process of the old classification model includes the old task stage, and the training process of the new classification model includes multiple incremental learning stages; the fully connected layer corresponding to the old task stage and each incremental learning stage is constructed as the classifier; the base class endoscopic image is input into the old classification model, and the cross entropy loss function is used The parameters of the old classification model are trained. The total number of categories output by the classifier of the old classification model is ; Freeze the parameters of the old classification model and use the parameters of the old classification model to initialize the new classification model; The number of categories output by the classifier in the kth incremental learning stage of the new classification model , is the total number of all categories seen by the new classification model; let the sequence number of the incremental learning stage k = 1;
[0009] Step 2: Training of new classification model:
[0010] The training data of the kth incremental learning stage is input into both the new classification model and the old classification model. Let the logical value of the classifier output of the old classification model be , the logical value of the classifier output of the new classification model is ; Only the logical values of the newly added categories in the kth incremental learning stage are softmax transformed, and the cross entropy loss is calculated with the one-hot vector of the true label :
[0011] Logical values based on the output of the old classification model , the logical value output by the new classification model Calculate logistic distillation loss ;
[0012] Channel attention vector output by the channel attention mechanism of the old classification model , the channel attention vector output by the channel attention mechanism of the new classification model Perform knowledge distillation and calculate attention distillation loss ;
[0013] Attention-based distillation loss , Logical value distillation loss and cross entropy loss Constructing the total loss , and through the total loss Train a new classification model: ;in, and For trade-off parameters;
[0014] Update the sequence number of the incremental learning phase k = k + 1;
[0015] Step three, iteratively execute step two until the new classification model converges or reaches the set number of training times.
[0016] Furthermore, in step 1, the cross entropy loss function is used The parameters for training the old classification model include:
[0017] ;
[0018] Logical value representing the output of the old classification model The elements, represents the Softmax function, is the one-hot vector of the true label elements.
[0019] Furthermore, in step 1, the number of categories output by the classifier in the kth incremental learning stage of the new classification model is , is the total number of all categories seen by the new classification model, including;
[0020] ;
[0021] represents the number of new categories added in the i-th incremental learning stage; The total number of categories output by the classifier for the old classification model.
[0022] Furthermore, in step 2, the one-hot vector with the true label calculates the cross entropy loss :
[0023] ;
[0024] in, Logical value output by the new classification model for the newly added categories in the current incremental learning phase The elements, represents the Softmax function, is the one-hot vector of the true label elements, Indicates The number of new categories added in the incremental learning phase.
[0025] Furthermore, in step 2, the logic value outputted by the old classification model , the logical value output by the new classification model Calculate logistic distillation loss , specifically including:
[0026] The probability that the endoscopic images in the training data belong to each category is used as a soft target:
[0027] ;
[0028] ;
[0029] in, Indicates that the old classification model predicts that the image belongs to the category The probability of Indicates that the new classification model predicts that the image belongs to The probability of the categories, Logical value representing the output of the old classification model The elements, Logical value representing the output of the new classification model The elements, Logical value representing the output of the old classification model No. elements, Logical value representing the output of the new classification model for the old categories that have been introduced in the previous incremental learning phase No. elements and introduces the temperature factor To control the importance of each soft target;
[0030] Logistic Distillation Loss The calculation method is:
[0031] .
[0032] Furthermore, in step 2, the channel attention vector output by the channel attention mechanism of the old classification model , the channel attention vector output by the channel attention mechanism of the new classification model Perform knowledge distillation and calculate attention distillation loss , specifically including:
[0033] Adopting a squeeze-excitation module to implement the channel attention mechanism;
[0034] The channel attention vector calculated by the squeeze-excitation module of the old classification model , the channel attention vector calculated by the squeeze-excitation module of the new classification model Perform knowledge distillation to achieve channel attention self-derivation; attention distillation loss The calculation method is:
[0035] ;
[0036] in, represents the norm of a vector, the superscript Indicates transpose.
[0037] Compared with the prior art, the beneficial technical effects of the present invention are:
[0038] Through the self-derivative distillation method of channel attention, more old knowledge is retained in the incremental learning stage, which more effectively alleviates catastrophic forgetting.
[0039] The addition of the channel attention mechanism can dynamically adjust the weights of different feature channels, so that the classification model can focus more on important feature information, which can enhance the classification model's ability to represent key features and improve the classification model's representation ability and classification performance. On the other hand, by distilling the channel attention vector of the old classification model to the new classification model, the classification model's attention to the features that are more important for distinguishing the old classes can be retained, and the decision logic adopted by the old classification model during the learning process can be transferred. Finally, by analyzing the working principle and output results of the channel attention mechanism, we can better understand the classification model's attention and importance to different features, thereby improving the interpretability of the classification model.
[0040] Compared with the existing incremental learning methods, the present invention does not require any old data, saving expensive computing resources and storage resources. By introducing channel attention self-derivation, the old knowledge is more comprehensively retained, showing better classification performance in class incremental tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A schematic diagram of the training framework of a quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation;
[0042] Figure 2 This is a confusion matrix effect diagram in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.
[0044] If the classification model is trained with endoscopic images of a particular class, then the particular class is referred to as the seen class of the classification model.
[0045] The overall training framework of the present invention is shown in Figure 1 , specifically including the following steps:
[0046] Step 1: Model initialization:
[0047] (a) Selection of backbone network: Decoupling the deep neural network according to feature extraction and linear classification, the classification model can be divided into feature extractor and linear classifier Two parts. A convolutional neural network (such as ResNet50) pre-trained on the large image classification dataset ImageNet is used as the backbone network, the fully connected layer of ResNet50 is removed, and the channel attention mechanism is introduced as the feature extractor of the classification model. The fully connected layer corresponding to each incremental learning stage is constructed as the classifier.
[0048] (b) Training of the old classification model:
[0049] Input the base class data into the old classification model and use the cross entropy loss function The parameters of the old classification model are trained. The number of categories of the old classification model classifier is the number of categories of the base class data. Freeze the parameters of the old classification model so that it can be used as a teacher model to guide the training of the new classification model in the next incremental learning phase. Cross entropy loss function The calculation method is as follows:
[0050] ;
[0051] in, Logits are the logical values output by the old classification model. represents the Softmax function, is the true label.
[0052] (c) Initialization of the new classification model:
[0053] Initialize the new classification model using the parameters of the old classification model. The number of categories of the classifier in the incremental learning stage is the total number of all categories seen by the new classification model .
[0054] Step 2: Training of new classification model:
[0055] (a) Cross Entropy Loss:
[0056] The training data of the kth incremental learning stage is input into the new classification model and the old classification model at the same time. Let the logical value of the classifier output of the old classification model be , the logical value of the classifier output of the new classification model is Only the logical values (Logits) of the newly added categories are transformed by softmax, and the cross entropy is calculated with the one-hot vector of the true label. :
[0057] ;
[0058] in, Logical value output by the new classification model for the newly added categories in the current incremental learning phase The elements, represents the Softmax function, is the one-hot vector of the true label elements.
[0059] (b) Response-based distillation loss:
[0060] The probability that the input endoscopic image data belongs to each category is taken as a soft target, which can be estimated by the softmax function, and its expression is:
[0061] ;
[0062] ;
[0063] in, Indicates that the old classification model predicts that the image belongs to the category The probability of Indicates that the new classification model predicts that the image belongs to The probability of the categories, Logical value representing the output of the old classification model The elements, Logical value representing the output of the new classification model The elements, Logical value representing the output of the old classification model No. elements, Logical value representing the output of the new classification model for the old categories that have been introduced in the previous incremental learning phase No. elements and introduces the temperature factor To control the importance of each soft target.
[0064] Generally speaking, the distillation loss is usually calculated using the Kullback-Leibler (KL) divergence loss. Logits distillation loss The calculation method is as follows:
[0065] ;
[0066] (c) Channel attention self-derivation loss:
[0067] The present invention adopts a squeeze-and-excitation module (SE) as a channel attention unit to improve its feature representation capability while measuring the importance of feature channels. Specifically, the squeeze-and-excitation module consists of a global average pooling layer, two fully connected layers, and an activation function. Considering that the features extracted by the shallow network are usually irrelevant to the specific category in the image, as the number of network layers increases, the subsequent layers will gradually learn more specific and category-related features. Therefore, in the present invention, the squeeze-and-excitation module is used to perform attention calculation on the feature map extracted by the last residual block. On the one hand, this process can help the network pay more attention to those features that can help distinguish different categories, and on the other hand, it can also reflect the contribution of feature channels to the correct classification of old knowledge from the side. Therefore, maintaining the original feature channel attention when learning new knowledge helps to maintain the memory of old knowledge.
[0068] Based on the above analysis, knowledge distillation is performed on the channel attention vectors calculated by the squeeze-excitation module in the new classification model and the old classification model, forcing the two to maintain similar channel attention. This can not only activate the discriminative representation of new knowledge through the attention mechanism, but also help the classification model to still pay attention to the feature channels that help distinguish old categories when learning new knowledge, thus realizing the self-derivation of channel attention. Attention distillation loss The calculation method is as follows:
[0069] ;
[0070] in, is the channel attention vector output by the channel attention mechanism of the old classification model, It is the channel attention vector output by the channel attention mechanism of the new classification model.
[0071] (d) Attention Distillation Loss , Logical value distillation loss and cross entropy loss Combined, the total loss of the new classification model training is constructed , this method encourages the channel attention vectors of the teacher model and the student model to be more similar. The total loss The calculation method is as follows:
[0072] ;
[0073] in, and It is a trade-off parameter used to adjust the influence of each loss function on the total loss.
[0074] Step 3: Iterate repeatedly.
[0075] In the incremental learning phase, when a new task comes, step 2 is executed iteratively multiple times to continuously learn new knowledge and update the new classification model. The final new classification model can show good classification performance on all known classes.
[0076] Example
[0077] We collected 2,434 gastrointestinal endoscopic images of patients with digestive tract diseases from hospitals, and obtained an endoscopic image dataset containing five types of lesions: cancer, colitis, ulcers, polyps, and proctitis, which was used for training and testing the classification model. We selected 80% of the dataset as the training set and the remaining 20% as the test set to evaluate the classification model.
[0078] The settings for incremental learning are as follows: the training set of the first incremental learning stage includes three categories: cancer, ulcers, and polyps, and the test set also includes these three categories. The training set of the second incremental learning stage includes two categories: proctitis and colitis, and the test set covers all categories. This setting can comprehensively evaluate the model's ability to retain and apply previous knowledge when learning new categories. At the same time, it also better simulates the incremental learning process in actual application scenarios, ensuring that the classification model maintains effective use of old knowledge while continuously learning new knowledge, thereby achieving more robust and reliable classification performance.
[0079] This example selected ResNet50 as the backbone network and pre-trained it on the ImageNet dataset. Before the experiment, the original endoscopic images were pre-processed: the size of the endoscopic images was uniformly adjusted to 224×224 pixels, and the endoscopic images were pre-processed by random flipping and random rotation to increase the diversity of the data. Finally, the endoscopic images were normalized.
[0080] The experiment was implemented in Python, and the deep learning library PyTorch was used to build and train the component models. The CUDA version of the experimental platform was 11.4, and the graphics card model was NVIDIA GeForce RTX 3090. During the training process, the batch size was set to 32, and SGD was used as the optimizer with a momentum value of 0.9 and an initial learning rate of 0.001. The cosine annealing learning rate scheduler was used to dynamically adjust the learning rate during model training to improve the performance and stability of the model. In terms of hyperparameter settings, and They are set to 2 and 1 respectively. A total of 60 epochs are set, that is, the entire training set is traversed 60 times to ensure that the loss value converges.
[0081] Figure 2 The confusion matrix diagram of the classification results after the incremental learning phase is given by using the method of the present invention to classify the endoscopic image dataset into incremental lesions. In the confusion matrix, the rows represent the actual categories and the columns represent the predicted categories. The elements on the diagonal represent the number of samples correctly predicted by the classification model, while the elements on the off-diagonal represent the number of samples incorrectly predicted by the classification model. Figure 2 It can be seen that the method of the present invention can still distinguish samples belonging to old categories well after learning new knowledge, and effectively alleviates catastrophic forgetting.
[0082] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.
[0083] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.
Claims
1. A quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation, characterized in that: Inputting the endoscopic image belonging to the category seen by the classification model into the trained classification model to obtain the classification result of the endoscopic image; The training process of the classification model includes: Step 1, initialization of the classification model: the classification model includes a feature extractor and a classifier; the pre-trained convolutional neural network is removed from the fully connected layer and the channel attention mechanism is introduced as the feature extractor; the classification model includes an old classification model and a new classification model; the training process of the old classification model includes the old task stage, and the training process of the new classification model includes multiple incremental learning stages; the fully connected layer corresponding to the old task stage and each incremental learning stage is constructed as the classifier; the base class endoscopic image is input into the old classification model, and the cross entropy loss function is used The parameters of the old classification model are trained. The total number of categories output by the classifier of the old classification model is ; Freeze the parameters of the old classification model and use the parameters of the old classification model to initialize the new classification model; The number of categories output by the classifier in the kth incremental learning stage of the new classification model , is the total number of all categories seen by the new classification model; let the sequence number of the incremental learning stage k = 1; Step 2: Training of new classification model: The training data of the kth incremental learning stage is input into the new classification model and the old classification model at the same time. Let the logical value of the classifier output of the old classification model be , the logical value of the classifier output of the new classification model is ; Only the logical values of the newly added categories in the kth incremental learning stage are softmax transformed, and the cross entropy loss is calculated with the one-hot vector of the true label : Logical values based on the output of the old classification model , the logical value output by the new classification model Calculate logistic distillation loss ; Channel attention vector output by the channel attention mechanism of the old classification model , the channel attention vector output by the channel attention mechanism of the new classification model Perform knowledge distillation and calculate attention distillation loss ; Attention-based distillation loss , Logical value distillation loss and cross entropy loss Constructing the total loss , and through the total loss Train a new classification model: ;in, and For trade-off parameters; Update the sequence number of the incremental learning phase k = k + 1; Step three, iteratively execute step two until the new classification model converges or reaches the set number of training times.
2. The method for classifying lesions in quasi-incremental endoscopic images based on channel attention self-derivation according to claim 1 is characterized in that: In step 1, the cross entropy loss function is used Parameters for training the old classification model, including: ； Logical value representing the output of the old classification model The elements, represents the Softmax function, is the one-hot vector of the true label elements.
3. The quasi-incremental endoscopic image lesion classification method based on channel attention self-derivation according to claim 1 is characterized in that: In step 1, the number of categories output by the classifier in the kth incremental learning stage of the new classification model , is the total number of all categories seen by the new classification model, including; ； represents the number of new categories added in the i-th incremental learning stage; The total number of categories output by the classifier for the old classification model.
4. The method for classifying lesions in quasi-incremental endoscopic images based on channel attention self-derivation according to claim 1, characterized in that: In step 2, the cross entropy loss is calculated with the one-hot vector of the true label : ； in, Logical value output by the new classification model for the newly added categories in the current incremental learning phase The elements, represents the Softmax function, is the one-hot vector of the true label elements, Indicates The number of new categories added in the incremental learning phase.
5. The method for classifying lesions in quasi-incremental endoscopic images based on channel attention self-derivation according to claim 1, characterized in that: In step 2, the logical value outputted by the old classification model , the logical value output by the new classification model Calculate logistic distillation loss , specifically including: The probability that the endoscopic images in the training data belong to each category is used as a soft target: ； ； in, Indicates that the old classification model predicts that the image belongs to the category The probability of Indicates that the new classification model predicts that the image belongs to The probability of the categories, Logical value representing the output of the old classification model The elements, Logical value representing the output of the new classification model The elements, Logical value representing the output of the old classification model No. elements, Logical value representing the output of the new classification model for the old categories that have been introduced in the previous incremental learning phase No. elements and introduces the temperature factor To control the importance of each soft target; Logistic Distillation Loss The calculation method is: 。 6. The method for classifying lesions in quasi-incremental endoscopic images based on channel attention self-derivation according to claim 1, characterized in that: In step 2, the channel attention vector output by the channel attention mechanism of the old classification model is , the channel attention vector output by the channel attention mechanism of the new classification model Perform knowledge distillation and calculate attention distillation loss , specifically including: Adopting a squeeze-excitation module to implement the channel attention mechanism; The channel attention vector calculated by the squeeze-excitation module of the old classification model , the channel attention vector calculated by the squeeze-excitation module of the new classification model Perform knowledge distillation to achieve channel attention self-derivation; attention distillation loss The calculation method is: ； in, represents the norm of a vector, the superscript Indicates transpose.