Gastrointestinal endoscopy image recognition method, device and medium based on IGCNN

Abstract

The present invention discloses a method, device, and medium for digestive tract endoscopy image recognition based on IGCNN. The method comprises: inputting a digestive tract endoscopy image into a convolutional neural network, wherein the last convolutional layer outputs a multi-channel feature map, and the last fully connected layer outputs a class score for image recognition; backpropagation is used to calculate the gradient of the class score with respect to each channel feature map, and the gradient is globally averaged and pooled to obtain the importance weight of each channel feature map, and then weighted combination is performed to obtain an interpretable attention map; the attention map is processed and calculated to obtain a predicted box, and a first loss function is constructed based on the predicted box and the true box; an image recognition classification loss function is constructed based on the class scores and the true box of each image recognition category; the two loss functions are combined to train the convolutional neural network, and the trained model is used to recognize and output the digestive tract endoscopy image to be recognized. The present invention has a high recognition accuracy rate for digestive tract endoscopy images.

Description

Technical Field

[0001] The present invention belongs to the field of medical image recognition, and specifically relates to a gastrointestinal endoscopy image recognition method, device and medium based on IGCNN. Background Art

[0002] Gastrointestinal endoscopy is a diagnostic and therapeutic procedure that allows for the imaging, evaluation, and treatment of gastrointestinal diseases. Each gastrointestinal endoscopy generates a large number of images. Before clinicians can make a diagnosis, they must expend considerable effort and time to identify lesion images within this vast dataset. Therefore, to improve detection efficiency and accuracy, it is necessary and valuable to develop a computer-aided diagnosis method that can automatically identify lesion images within this vast dataset of gastrointestinal endoscopy images.

[0003] Currently, existing methods for identifying gastrointestinal endoscopic lesion images can be roughly divided into two categories: one is based on the combination of traditional image processing technology and machine learning, and the other is based on deep learning.

[0004] The first type of method has strong interpretability, but it requires manual design and feature extraction when using traditional image processing technology, which is time-consuming and labor-intensive. For example, Ansari et al. [1] A system for automatic detection and identification of polyps was developed. First, local fractal dimension (LFD) features were extracted on the detected SIFT keypoints. These features were then connected with uniform local binary patterns (LBPu) or FulpteLBP (CLBP) to integrate texture information. Finally, support vector machines (SVMs), multi-layered layers (MLPs), and random forest classifiers were used for classification, with an accuracy of 97.98%.

[0005] Another type of deep learning-based gastrointestinal endoscopic lesion image recognition method can achieve end-to-end training and automatically learn image features from a large amount of image data. However, this type of method generally lacks interpretability. For example, in order to identify ulcer images from wireless capsule endoscopy videos, Alaskar H et al. [2] GoogLeNet and AlexNet were used in combination with transfer learning to classify ulcer images and normal images captured by wireless capsule endoscopy videos, and the accuracy on the test set can reach 100%. This type of method directly uses deep learning models to identify gastrointestinal endoscopic lesion images, but due to the black box nature of deep learning, these methods lack interpretability. Therefore, some studies use interpretability methods to explain the prediction results of deep learning models, which makes the models more credible. For example, in order to improve the reliability of deep learning model decisions, NadimiES et al. [3] Using pre-trained ZF network [4]to detect colon polyps in wireless capsule endoscopy videos, but this method first uses CNN to train the model and then explains the prediction results. It does not achieve end-to-end training, and the CNN model itself is not interpretable.

[0006] Currently, in the field of medical image analysis, most studies first train deep learning models and then use deep learning interpretability methods to explain the prediction results of the models. Few studies integrate interpretability methods into deep learning models to build interpretable deep learning models. However, there are similar studies in other image analysis fields. For example, LiK [5] et al. combined interpretability methods with convolutional neural networks to design an end-to-end, interpretable weakly supervised image segmentation model. This model achieved a 5.3% improvement in mean Interference over Union (MIoU) on the VOC2012 test set. However, this model requires pixel-level labels, which is a significant challenge in the medical imaging field, where data annotation is difficult. Furthermore, this model is suitable for weakly supervised segmentation tasks, but not for gastrointestinal endoscopy image recognition. Summary of the Invention

[0007] The present invention provides a gastrointestinal endoscopy image recognition method, device and medium based on IGCNN, which uses an interpretability method to guide a convolutional neural network (denoted as IGCNN), forcing the network to focus on the lesion area in the image and improve the classification accuracy.

[0008] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0009] A digestive tract endoscopy image recognition method based on IGCNN, wherein the IGCNN is an interpretable method-guided convolutional neural network, and the digestive tract endoscopy image recognition method based on IGCNN includes:

[0010] The digestive tract endoscopy image x is input into the convolutional neural network. The last convolutional layer of the convolutional neural network outputs multiple channel feature maps. The last fully connected layer of the convolutional neural network outputs the class score s of the image recognition. c ;

[0011] Back propagation calculates the class score s c About each channel feature map F k Gradient The gradient Get F through the global average pooling layer k The corresponding importance weight and will As the weight of each channel feature map F k Weighted combination to obtain an interpretable attention map A corresponding to the input image x c ;

[0012] Attention map A c Perform image processing and calculate the predicted frame, and construct the first loss function based on the predicted frame and the true frame;

[0013] Class scores s for each category based on image recognition c And the real box, construct the image recognition classification loss function, recorded as the second loss function;

[0014] The convolutional neural network is trained by combining the first and second loss functions to obtain a digestive tract endoscopy image recognition model;

[0015] The trained digestive tract endoscopy image recognition model is used to identify and output the digestive tract endoscopy image to be identified.

[0016] Furthermore, the convolutional neural network uses ResNet18 as its backbone network.

[0017] Furthermore, the importance weight The calculation expression is:

[0018]

[0019] Among them, GAP(.) represents the global average pooling operation, Z is the k-th channel feature map F k The number of elements in , Represents the kth channel feature map F k The element in row i and column j.

[0020] Further, As the weight of each channel feature map F k The expression of the attention map obtained by weighted combination is:

[0021]

[0022] Among them, F k represents the k-th channel feature map, and ReLU represents the activation function.

[0023] Furthermore, the attention map A c Perform image processing and calculate the prediction box, specifically: c Upsample to the scale of the input image x, and then binarize to obtain M c , and then calculate the binary image M c The circumscribed matrix of each connected region in :

[0024] Pb=Extract_Box(M c )

[0025] Among them, Pb is the calculated circumscribed matrix, that is, the calculated prediction box; Extract_Box(.) is the operator for calculating the circumscribed rectangle.

[0026] Furthermore, the first loss function is constructed based on the predicted frame and the true frame:

[0027]

[0028] Among them, Loss mul_GIoU Represents the first loss function value, N is the number of images in a batch_size, x l For the lth image, loss xl For image x l The loss function value of , specifically:

[0029]

[0030]

[0031]

[0032]

[0033] in, For image x l The i-th prediction box Pb i The loss function value of m is the image x l The number of prediction boxes; n is the image x l The number of true boxes, Pt j For image x l The j-th ground-truth box in GIoU(Pb i ,Pt j ) represents the prediction box Pb i With real box Pt j The overlap is greater than More consideration of Pb i and Pt j Disjoint situations; Represents the prediction box Pb i With real box Pt j The minimum rectangle of Represents the prediction box Pb i Relative to the real frame Pt j degree of overlap.

[0034] Furthermore, Replaced with the following improved distance loss And the calculation formula is:

[0035]

[0036] Furthermore, the cross entropy is used to construct the image recognition classification loss function, and the weighted sum is used to combine the first and second loss functions to obtain the overall loss function for training the convolutional neural network:

[0037] Loss total =(1-α)Loss class +αLoss mul_GIoU

[0038] Among them, Loss mul_GIoU Represents the first loss function value, Loss class Represents the second loss function value, Loss total Represents the overall loss function value.

[0039] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor implements any one of the above-mentioned gastrointestinal endoscopic image recognition methods based on IGCNN.

[0040] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-mentioned methods for gastrointestinal endoscopy image recognition based on IGCNN.

[0041] This paper proposes an interpretable model, interpretability methods guided convolutional neural networks (IGCNN), for the classification of gastrointestinal endoscopic lesion images, which has the following beneficial effects:

[0042] (1) Design an end-to-end, interpretable deep learning model for the classification of gastrointestinal endoscopic lesion images;

[0043] (2) Using IGCNN to force the network to focus on the lesion area in the image, improving the classification accuracy;

[0044] (3) Innovatively improve the loss function of the IGCNN model training to further improve the accuracy of the trained model in recognizing gastrointestinal endoscopic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the IGCNN model framework of the embodiment of this application.

[0046] Figure 2 There are two cases where the prediction box Pb and GroundTruthGt do not intersect (sub-image (a)) and intersect (sub-image (b)).

[0047] Figure 3 is the GroundTruth and predicted box of the polyp image.

[0048] Figure 4 This is a comparison of the attention maps of the ResNet and IGCNN models. The first sub-image (a) is the original polyp image, the second sub-image (b) is the prediction result of ResNet18, and the third sub-image (c) is the prediction result of IGCNN.

[0049] Figure 5 This is a comparison of the attention maps of the ResNet and IGCNN models. The first sub-image (a) is the original image of ulcerative colitis, the second sub-image (b) is the prediction result of ResNet18, and the third sub-image (c) is the prediction result of IGCNN. DETAILED DESCRIPTION

[0050] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, provides a detailed implementation method and a specific operation process, and further explains the technical solution of the present invention.

[0051] A digestive tract endoscopy image recognition method based on IGCNN includes the following steps:

[0052] Step 1: Input the digestive tract endoscopy image x into the convolutional neural network. The last convolutional layer of the convolutional neural network outputs multiple channel feature maps. The last fully connected layer of the convolutional neural network outputs the class score s of the image recognition. c .

[0053] This example uses a pre-trained ResNet series model as the convolutional neural network backbone. Other models, such as VGG, GoogLeNet, and AlexNet, can also be used as backbones. This example uses ResNet18, which introduces a residual structure to address the degradation problem of deep networks, allowing for deeper network training and better image feature extraction. Furthermore, ResNet has fewer parameters than models like VGG, allowing for faster training.

[0054] Step 2: Back propagation to calculate the class score s c About each channel feature map F k Gradient The gradient Get F through the global average pooling layer k The corresponding importance weight and will As the weight of each channel feature map F kWeighted combination to obtain an interpretable attention map A corresponding to the input image x c .

[0055] Currently, the interpretability methods used in the field of medical image analysis can be roughly divided into two categories: attribute methods and non-attribute methods.

[0056] The attribute method assigns an attribution value or contribution value to each input feature of the network. The attribution values ​​of all input features are arranged in the shape of the input sample, forming a heat map called an attention map or attribute map. It is a model-independent method. Non-attribute methods, on the other hand, develop and validate an interpretability method for a given specific problem. Most research on interpreting deep learning in medical image diagnosis uses attribution-based methods. Based on this fact, this embodiment chooses an attribute-based interpretability method to solve the attention map.

[0057] To describe the mathematical framework of attribution-based interpretability methods, consider the following setting: a collection F of deep learning models, a deep learning model f∈F, where f: c×w×h → d , that is, the input space of the deep learning model is The output space is The attribute-based interpretability method is some function Φ:F× c×w×h → d , that is, given a deep learning model f∈F, and a test sample x∈X, predict the output f(x)∈Y, and provide an explanation (or attention map) Φ(f,x) for the output f(x). Attribute methods include perturbation-based methods and back-propagation-based methods. The perturbation-based method analyzes the impact of changes in input features on the output of the deep learning model, such as the LIME method, SHAP based on cooperative game theory, and so on. The back-propagation-based method calculates the gradients of all input features through a forward and backward network propagation process, and then calculates the attention map based on the gradient, such as guided back-propagation, class activation mapping (CAM), Grad-CAM, Grad-CAM++, and so on. In the field of image analysis, back-propagation-based methods are generally used to explain models. This embodiment builds an interpretability model based on the Grad-CAM interpretability method - an interpretability methods guided convolutional neural networks (IGCNN) model for gastrointestinal endoscopic image recognition.

[0058] The IGCNN model built in this embodiment is an end-to-end, interpretable network. The main idea is to use interpretability methods to guide convolutional neural networks to learn image features, thereby improving the accuracy of network classification and making the network interpretable.

[0059] like Figure 1 As shown in [1], consider an image x from the training set with the image-level label "polyp." First, x is fed into the backbone network to extract features. Given the small experimental dataset used in this example, a shallower ResNet18 network is used as the backbone to prevent overfitting. Next, an attention map for image x is calculated using Grad-CAM. The resulting attention map has the same dimensions as image x. The process for calculating the attention map is described below.

[0060] First, calculate logits, which is the output of the last fully connected layer of the backbone. Let the logit of class c (that is, the output before the softmax layer) be s c , F is the feature map output by the last convolutional layer of the model, let F k is the feature of the kth channel of the feature map F, and s is calculated by back propagation c About F k Gradient The gradient Get F through the global average pooling layer k The corresponding importance weight Right now:

[0061]

[0062] Among them, GAP(.) represents the global average pooling operation, Z is F k The number of elements in . Obtained by back propagation There is no need to update network parameters. Represents the activation map F k The importance of as the weight-to-activation map F k Weighted, and linearly combined according to k, and finally through the ReLU activation function, we can get the attention map A of x c ,Right now:

[0063]

[0064] Step 3: Attention map A c Perform image processing and calculate the predicted frame, and construct the first loss function based on the predicted frame and the true frame.

[0065] Get attention map A cAfter that, we need to upsample to the size of image x and then c Normalized to the interval [0,1].

[0066] Since the data set is small and the image features are not obvious, the convolutional neural network cannot correctly focus on the features of the image, resulting in a low classification accuracy of the convolutional network. c To guide the convolutional network to focus on the correct features. Specifically, a rectangular frame can be used to enclose the areas in the image where the network pays more attention (A c The predicted box is framed by the pixels with larger median values, and then the loss function is used to correct the distance between the predicted box and GroundTruth. The parameters are adjusted through backpropagation, gradually making the convolutional neural network focus on the part of the image that should be paid attention to. The specific steps are as follows:

[0067] First, use the attention map A c Get the binary image M c , the calculation formula is:

[0068]

[0069] Where σ∈[0,1] is the threshold. Then calculate the binary image M c The bounding rectangle of the connected area with a median value of 1 is:

[0070] Pb=Extract_Box(M c ) (4)

[0071] Extract_Box(.) is an operator for calculating the bounding rectangle. In practice, we can use the contour extraction algorithm findContours in the OpenCV package included with Python to first detect the boundaries of the connected region and then calculate the coordinates of the upper left and lower right corners of the bounding rectangle based on these boundaries. The Pb obtained by Extract_Box(.) is the model's predicted box. Assuming there are m predicted boxes in Pb, then:

[0072] Pb=[[(x 11 ,y 11 ),(x 12 ,y 12 )],[(x 21 ,y 21 ),(x 22 ,y 22 )],...,[(x m1 ,y m1 ),(x m2 ,y m2 )]](5)

[0073] Among them, (xi1 ,y i1 ) is the coordinate of the upper left corner of the i-th prediction box, (x i2 ,y i2 ) is the coordinate of the lower right corner of the i-th prediction box. The prediction box obtained is also the area where the model pays more attention. This area may contain multiple bounding rectangles. Therefore, this embodiment defines a loss function Loss mul_GIoU To minimize the distance between the predicted box Pb and the ground truth box. The ground truth box of the image needs to be manually annotated, but unlike object detection, the ground truth box annotated in this embodiment does not need to be particularly precise; as long as the object's location is roughly marked, the purpose of guiding the network can be achieved.

[0074] The first loss function proposed in this embodiment can measure the distance between the predicted box Pb and the true box. It is defined based on the IoU loss function and the GIoU loss function. Therefore, before introducing the Multi-GIoU loss function, it is necessary to briefly describe IoU and GIoU.

[0075] IoU (Intersection over Union) is the most commonly used evaluation metric in target detection and segmentation tasks. However, existing technologies believe that there is a significant gap between minimizing the distance loss of the target bounding box regression parameters (such as the L1, L2, and Smooth L losses commonly used in target detection) and maximizing the IoU metric. Moreover, losses such as L1, L2, and SmoothL1 all calculate the losses of the four position coordinates of the target bounding box separately and then add them together, without considering the correlation between the coordinates. Therefore, existing technologies have proposed losses based on IoU construction.

[0076] Assume that Pb is the predicted box obtained by the model and Gt is the ground truth. The calculation formula for the IoU between Pb and Gt is as follows:

[0077]

[0078] The value range of IoU is [0,1]. The loss function based on IoU is:

[0079] Loss IoU =1-IoU,Loss IoU ∈[0,1], (7)

[0080] The closer the predicted box is to the Ground Truth, the closer the IoU is to 1, and the Loss IoU The closer it is to 0. But when the two boxes do not intersect, IOU = 0, Loss IoU=1, pass the gradient back, when the two boxes are far apart, adjust the position of the prediction box, and its IOU is also equal to 0, which will result in the prediction box Loss after multiple adjustments. IoU If the loss is still equal to 1, the network needs many iterations (random attempts) to achieve the intersection of the two boxes, and then continuously reduce the loss. Therefore, the existing technology further proposes the GIoU (Generalized Intersection over Union) loss based on the IoU loss. The calculation formula of GIoU is:

[0081]

[0082] Among them, C is the smallest rectangle that contains both Pb and Gt. Figure 2 As shown, the shaded part is C-(Pb∪Gt). Note that the value range of GIoU is [-1,1], and the loss function based on GIoU is defined as:

[0083] Loss GIoU =1-GIoU,Loss GIoU ∈[0,2]. (9)

[0084] When two boxes do not intersect and are far apart, GIoU is close to -1 and Loss GIoU When the gradient is close to 2, the loss can still be continuously reduced when the gradient is passed back. Therefore, this embodiment constructs the first loss function between the predicted box and the true box based on GIoU.

[0085] The Ground Truth in image x may have multiple rectangular boxes, denoted as Gt1, Gt2, ..., Gt n , the image x may also have multiple prediction boxes obtained by the IGCNN model, recorded as Pb1, Pb2, ..., Pb m Considering that the optimization goal is to make the prediction box as contained as possible in the Ground Truth, the focus of the IGCNN model will also be in the Ground Truth. i ,i∈{1,2,...,m} defines its loss function as:

[0086]

[0087] in

[0088]

[0089] Then the loss function of image x is:

[0090]

[0091] If there are N images in a batch_size, the final formula of the first loss function is:

[0092]

[0093] Among them, x l is the lth image in a batch_size.

[0094] Considering that the model's attention should be directed to the ground truth box of the image, IoU measures the overlap between two boxes. Therefore, in a more optimal embodiment, the IoU in formula (6) is further modified by replacing the denominator of IoU with the predicted box, that is:

[0095]

[0096] Substituting IM_IoU into formula (13) yields the improved loss function IM_Loss mul_GIoU .like Figure 3 As shown in , when the prediction box is relatively small and included in the Ground Truth, it can be considered that the model has correctly learned the features of the image and there is no need to back propagate to update the parameters. However, The loss function is large and back propagation is still required to update the parameters. The loss function is close to 0, and there is no need to update the parameters. From the experimental results, we can also see that using IM_Loss mul_GIoU The model of loss function uses Loss mul_GIoU The accuracy of the model with the loss function should be high.

[0097] Step 4: Logistic regression value s of each category based on image recognition c and the true frame, construct the image recognition classification loss function, which is recorded as the second loss function.

[0098] Step 5: Combine the first and second loss functions to train the convolutional neural network to obtain a gastrointestinal endoscopy image recognition model.

[0099] As shown in Formula 14, the final overall loss function of the IGCNN model is Loss total is the classification loss function Loss class and multiple GIoU loss functions Loss mul_GIoU The weighted sum of

[0100] Loss total =(1-α)Loss class +αLoss mul_GIoU (14)

[0101] Among them, Loss classIt is the second loss function constructed in step 4, which uses the cross entropy loss function of multi-classification; α∈[0,1], generally α is close to 1, because the IGCNN model mainly wants to correct the model's attention to the GroundTruth frame, so it is necessary to use Loss mul_GIoU The loss function is the main one.

[0102] Step 6: Use the trained digestive tract endoscopy image recognition model to identify and output the digestive tract endoscopy image to be identified.

[0103] Experimental verification:

[0104] (1) Dataset

[0105] The public dataset Kvasir was used to validate the above IGCNN model. The Kvasir dataset contains images of 8 categories, including anatomical landmarks such as Z-line, pylorus, and cecum, and physiological lesions such as esophagitis, polyps, and ulcerative colitis, as well as stained polyps and stained polyp resection margins.

[0106] In research [6-11] In this paper, we use existing deep learning models or our own deep learning models to classify images in the Kvasir dataset. From the confusion matrices of each model, we can see that the classification results of polyps and ulcerative colitis are not very ideal. The main reason is that the two types of images are somewhat similar, and the model cannot learn the features of the images well.

[0107] Since the IGCNN model of the present invention uses interpretability methods to guide the model to learn the correct features of the image, this experiment selects two types of pictures, polyps and ulcerative colitis, to verify the IGCNN model proposed by the present invention. The experimental data consists of 2,000 images, including 1,000 polyps and 1,000 ulcerative colitis. 800 images of each class are randomly selected as the training set and 200 as the test set. In addition, the IGCNN model requires a target detection label (Ground Truth), so these two types of images are labeled with the help of professional doctors.

[0108] (2) Evaluation indicators

[0109] The six performance indicators are Recall (REC), Precision (PREC), Specificity (SPEC), Accuracy (ACC), Matthews correlation coefficient (MCC), and F1 score (F1). If the confusion matrix of the two-class classification is as shown in Table 1:

[0110] Table 1 Confusion matrix for binary classification

[0111]

[0112] The calculation formulas for recall, precision, specificity, accuracy, Matthews correlation coefficient, and F1 score are:

[0113] (1) Recall rate:

[0114] (2) Accuracy:

[0115] (3) Specificity:

[0116] (4) Accuracy:

[0117] (5) Matthews correlation coefficient:

[0118] (6) F1 score:

[0119] Recall is often also called sensitivity, which is the proportion of samples correctly identified as positive among all positive samples. Precision represents the percentage of samples correctly identified as positive among samples predicted to be positive. Specificity represents the proportion of samples correctly identified as negative among all negative samples. Accuracy is the ratio of correctly identified samples among all samples. The Matthews correlation coefficient is a balanced measure that considers all samples, even if the class sizes are very different. The F1 score measures the accuracy of the test by calculating the harmonic mean of precision and recall. This paper uses these six performance metrics to evaluate the IGCNN model.

[0120] (3) Experimental setup and results

[0121] The IGCNN model was implemented using ResNet18 as the backbone. Experiments were conducted using a single Nvidia GeForce RTX 3050 GPU and Torch version 1.13.0. During data preprocessing, images were resized to 400x400 and normalized to a mean and variance of 0.5. During training, the model was trained for 40 epochs with a batch size of 4. Adam was used as the optimization algorithm, with an initial learning rate of lr = 0.001 and a cosine annealing strategy for adjusting the learning rate.

[0122] Under the same experimental settings, ResNet-18 was used to perform binary classification of polyps and ulcerative colitis to compare with the IGCNN model. The experiment found that using the pre-trained ResNet-18 as the backbone network can significantly improve the classification accuracy.

[0123] Quantitative evaluation of IGCNN:

[0124] When evaluating the IGCNN model, σ=0.4 and α=0.8 were taken. Experiments were conducted under the same data set, data processing, optimization algorithm and other conditions. The ResNet18 and IGCNN models with and without pre-training were compared respectively. The IGCNN model using the Multi-GIoU loss function and the IGCNN model using the Improved Multi-GIoU loss function were also compared to verify the effectiveness of the Improved Multi-GIoU loss function proposed in this invention. The experimental results were evaluated based on six performance indicators: recall rate, precision rate, specificity, accuracy rate, Matthews correlation coefficient, and F1 score. The results are shown in Table 2 below. It can be seen that the IGCNN using the Improved Multi-GIoU loss function performed the best. The closer the MCC is to 1, the better the performance of the model. Without pre-training, the MCC of IGCNN (using the Improved Multi-GIoU loss function) is 13.53% higher than that of ResNet18. With pre-training, the MCC of IGCNN (using the Improved Multi-GIoU loss function) is also 4.5% higher than that of ResNet18.

[0125] Table 2 Performance comparison of binary classification of polyp images and ulcerative colitis images in the Kvasir dataset

[0126]

[0127] Qualitative evaluation of IGCNN: We compare the attention maps of ResNet18 and IGCNN models to verify the advantage of IGCNN in correcting model attention.

[0128] Select some images that were misclassified by ResNet18, such as Figure 4 (a) Column and Figure 5 Column (a) is the original image of polyps and ulcerative colitis. In order to analyze which pixels in the image ResNet18 relies on for classification, we use the Grad-CAM++ interpretability method to explain the prediction results of ResNet18 for these images and output the attention maps corresponding to these images, as shown in the figure. Figure 4 and Figure 5In column (b), we can see that the ResNet18 classification error is because it focuses on the wrong features in the image. The IGCNN model itself is interpretable, so the output is the attention map of the image corresponding to the IGCNN model. Figure 4 and Figure 5 Column (c) shows that, with the guidance of interpretable methods, IGCNN correctly learns the image features, thus improving the classification accuracy. Through qualitative analysis of IGCNN, we can find that the IGCNN model can correct the backbone attention, thereby learning the correct image features and improving the classification accuracy.

[0129] In summary, the present invention provides a method for gastrointestinal endoscopy image recognition based on a convolutional neural network guided by an interpretable method. By connecting the interpretable method to the last fully connected layer in the convolutional neural network, the purpose of using the interpretable method to guide the convolutional neural network is achieved. Furthermore, a Multi-GIoU loss function and an Improved Multi-GIoU loss function are proposed to measure the gap between the prediction box output from the IGCNN and GroundTruth. Therefore, the present invention not only corrects the attention of the convolutional neural network, but also the model itself has end-to-end and interpretable characteristics, making the recognition of gastrointestinal endoscopy images highly accurate and interpretable.

[0130] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

[0131] References:

[0132] [1]MEAnsari and S.Charfi, “Computer-aided system for polyp detection in wireless capsuleendoscopy images,” in 2017 International Conference on Wireless Networks and Mobile Communications (WINCOM), 2017, pp.1–6.

[0133] [2]H.Alaskar,A.J.Hussain,N.Alaseem,P.Liatsis,and D.Al-Jumeily,“Application of convolutional neural networks for automated ulcer detectionin wireless capsule endoscopy images,”Sensors(Basel,Switzerland),vol.19,2019.

[0134] [3]E.S.Nadimi,M.Buijs,J.Herp,R.Kroijer,M.Kobaek-Larsen,E.Nielsen,C.D.Pedersen,V.Blanes-Vidal,and G.Baatrup,“Application of deep learning forautonomous detection andlocalization of colorectal polyps in wireless coloncapsule endoscopy,”Comput.Electr.Eng.,vol.81,p.106531,2020.

[0135] [4]M.Zeiler and R.Fergus,“Visualizing and understanding convolutionalnetworks,”in ECCV2014,2014.

[0136] [5]K.Li,Z.Wu,K.-C.Peng,J.Ernst,and Y.Fu,“Tell me where to look:Guidedattention inference network,”in2018IEEE / CVF Conference on Computer Vision andPattern Recognition,2018,pp.9215–9223.

[0137] [6]K.Pogorelov,K.R.Randel,C.Griwodz,S.L.Eskeland,T.de Lange,D.Johansen,C.Spampinato,D.-T.Dang-Nguyen,M.Lux,P.T.Schmidt,M.Riegler,andP.Halvorsen,“Kvasir:A multi-class image dataset for computer aidedgastrointestinal disease detection,”in Proceedings of the 8th ACM onMultimedia Systems Conference,ser.MMSys’17.NewYork,NY,USA:Association forComputing Machinery,2017,p.164–169.[Online].Available:https: / / doi.org / 10.1145 / 3083187.3083212

[0138] [7]T.J.D.Berstad,M.Riegler,H.Espeland,T.deLange,P.H.Smedsrud,K.Pogorelov,H.Kvale Stensland,and P.Halvorsen,“Tradeoffs using binary andmulticlass neural networkclassification for medical multidisease detection,”in 2018 IEEE International Symposium onMultimedia(ISM),2018,pp.1–8.

[0139] [8]Y.Afriyie,B.A.Weyori,and A.A.Opoku,“Gastrointestinal tract diseaserecognition basedon denoising capsule network,”Cogent Engineering,vol.9,no.1,p.2142072,2022.[Online].Available:https: / / doi.org / 10.1080 / 23311916.2022.2142072

[0140] [9]I.Iqbal,K.Walayat,M.U.Kakar,and J.Ma,“Automated identification ofhumangastrointestinal tract abnormalities based on deep convolutional neuralnetwork withendoscopic images,”Intelligent Systems with Applications,vol.16,p.200149,2022.[Online].Available:https: / / www.sciencedirect.com / science / article / pii / S2667305322000862

[0141]

[10] J.P.Escobar,N.Gomez,K.Sanchez,and H.Arguello,“Transfer learningwith convolutionalneural network for gastrointestinal diseases detectionusing endoscopic images,”in 2020 IEEEColombian Conference on Applications ofComputational Intelligence(IEEE ColCACI 2020),Aug 2020,pp.1–6.

[0142]

[11] D.Ezzat,H.M.Afify,M.H.N.Taha,and A.E.Hassanien,ConvolutionalNeuralNetwork with Batch Normalization for Classification of EndoscopicGastrointestinalDiseases.Cham:Springer International Publishing,2021,pp.113–128.[Online].Available:https: / / doi.org / 10.1007 / 978-3-030-59338-4 7。

Claims

1. A digestive tract endoscopy image recognition method based on IGCNN, characterized in that: The IGCNN is an interpretable method-guided convolutional neural network, and the digestive tract endoscopy image recognition method based on the IGCNN includes: The digestive tract endoscopy image x is input into the convolutional neural network. The last convolutional layer of the convolutional neural network outputs multiple channel feature maps. The last fully connected layer of the convolutional neural network outputs the class score s of the image recognition. c ; Back propagation calculates the class score s c About each channel feature map F k Gradient The gradient Get F through the global average pooling layer k The corresponding importance weight and will As the weight of each channel feature map F k Weighted combination to obtain an interpretable attention map A corresponding to the input image x c ; Attention map A c Perform image processing and calculate the predicted frame, and construct the first loss function based on the predicted frame and the true frame; Class scores s for each category based on image recognition c And the real box, construct the image recognition classification loss function, recorded as the second loss function; The convolutional neural network is trained by combining the first and second loss functions to obtain a digestive tract endoscopy image recognition model; Using the trained digestive tract endoscopy image recognition model, the digestive tract endoscopy image to be recognized is recognized and output; Among them, the first loss function constructed based on the predicted box and the true box is: Among them, Loss mul_GIoU Represents the first loss function value, N is the number of images in a batch_size, x l is the lth image, For image x l The loss function value, specifically: in, For image x l The i-th prediction box Pb i The loss function value of m is the image x l The number of prediction boxes; n is the image x l The number of true boxes, Pt j For image x l The j-th ground-truth box in GIoU(Pb i ,Pt j ) represents the prediction box Pb i With real box Pt j The overlap is greater than More consideration of Pb i and Pt j Disjoint situations; Represents the prediction box Pb i With real box Pt j The minimum rectangle of Represents the prediction box Pb i Relative to the real frame Pt j degree of overlap.

2. The digestive tract endoscopy image recognition method according to claim 1, characterized in that: The convolutional neural network uses the ResNet series model as its backbone network.

3. The digestive tract endoscopy image recognition method according to claim 1, characterized in that: Importance Weight The calculation expression is: Among them, GAP(.) represents the global average pooling operation, Z is the k-th channel feature map F k The number of elements in , Represents the kth channel feature map F k The element in row i and column j.

4. The digestive tract endoscopy image recognition method according to claim 1, characterized in that: Will As the weight of each channel feature map F k The expression of the attention map obtained by weighted combination is: Among them, F k represents the k-th channel feature map, and ReLU represents the activation function.

5. The digestive tract endoscopy image recognition method according to claim 1, characterized in that: The attention map A c Perform image processing and calculate the prediction box, specifically: c Upsample to the scale of the input image x, and then binarize to obtain M c , and then calculate the binary image M c The circumscribed matrix of each connected region in : Pb=Extract_Box(M c ) Among them, Pb is the calculated circumscribed matrix, that is, the calculated prediction box; Extract_Box(.) is the operator for calculating the circumscribed rectangle.

6. The digestive tract endoscopy image recognition method according to claim 1, characterized in that: Replaced with the following improved distance loss And the calculation formula is:

7. The digestive tract endoscopy image recognition method according to claim 1, characterized in that: Cross entropy is used to construct the image recognition classification loss function, and the weighted sum is used to combine the first and second loss functions to obtain the overall loss function for training the convolutional neural network: Loss total =(1-α)Loss class +αLoss mul_GIoU Among them, Loss mul_GIoU Represents the first loss function value, Loss class Represents the second loss function value, Loss total Represents the overall loss function value.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images