A method and system for interpretable lesion-aware fundus image classification

Abstract

The present invention discloses an interpretable lesion-aware fundus image classification method and system. First, a Transformer with multiple lesion labels is used to realize feature extraction of lesion perception, and then the lesion TC is enhanced by using the significant local information in the image block Token to obtain the lesion classification result. Then, the lesion knowledge embedding with medical field knowledge is encoded based on the fundus image basic large model, and the lesion concept learning is guided based on the idea of ​​contrast learning. Finally, an interpretable classifier is designed, which uses the preset learnable disease Token as the query sequence and the lesion TC as the key and value, calculates the cross attention, and uses the global average pooling result of its output sequence as the disease diagnosis result. The attention weight is used as an indicator of the contribution of each lesion to the specific disease diagnosis result, thereby generating an explanation of the disease diagnosis result. The present invention greatly improves the reliability of artificial intelligence-assisted medical diagnosis.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing and pattern recognition, and relates to a deep learning-based image classification method and system, and specifically to an interpretable deep learning-based lesion-aware fundus image classification method and system. Background Art

[0002] Taking fundus images is a method of examining the fundus. Doctors can observe the structure of each layer of the fundus and the microscopic morphology of the retinal tissue through fundus images, and obtain fundus information of patients relatively quickly. Automatic classification of fundus images can assist doctors in analyzing the condition to a certain extent and improve the speed of diagnosis.

[0003] At present, there are many intelligent diagnosis methods for fundus images, which can achieve fast and accurate classification of fundus images. For example, Gulshan et al. (Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs [J]. JAMA, 2016, 316 (22): 2402-2410.) proposed to implement DR classification based on the Inception network structure, and Li et al. (Li Z, He Y, Keel S, et al. Efficacy of adeep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs [J]. Ophthalmology, 2018, 125 (8): 1199-1206.) proposed a transfer learning method to transfer the model pre-trained on the natural image classification task to the fundus image classification, and both achieved good performance. In order to achieve a more fine-grained fundus disease classification task, some researchers proposed to introduce fundus lesion information to enhance the classification performance of the model. Dai et al. (Dai L, Wu L, Li H, et al. A deep learning system for detecting diabetic retinopathy across the disease spectrum [J]. Nature communications, 2021, 12 (1): 3242.) built a system that combines fundus image quality assessment, lesion segmentation and disease grading, making full use of lesion information to improve the performance of disease grading. Wang et al. (Wang X, Xu M, Zhang J, et al. Joint learning of multi-level tasks for diabetic retinopathy grading on low-resolution fundus images [J]. IEEE Journal of Biomedical and Health Informatics, 2021, 26 (5): 2216-2227.) developed a multi-task framework to jointly train three tasks: fundus image super-resolution, lesion segmentation and disease grading, and utilize the fine-grained information of lesions in multi-task collaboration.Although these methods have shown excellent results, they all require pixel-level or image block-level lesion annotations, which are very difficult to obtain and therefore difficult to implement in medical environments. In order to alleviate the limitations of annotations, some methods try to use weakly supervised methods or attention mechanisms to capture suspicious lesion areas, such as CabNet proposed by He et al. (He A, Li T, Li N, et al. CABNet: Category attention block for imbalanced diabetic retinopathy grading [J]. IEEE Transactions on Medical Imaging, 2020, 40 (1): 143-153.) and LANet by Hou et al. (Hou J, Xiao F, Xu J, et al. Diabetic Retinopathy Grading with Weakly-Supervised Lesion Priors [C] / / ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023: 1-5.). However, these methods still use convolutional neural networks as the main feature extractor, which makes it difficult to capture fundus lesions that vary greatly and are sparsely distributed.

[0004] Moreover, most of these methods are based on deep learning and have poor interpretability. Although they can quickly give diagnostic results, they cannot give corresponding explanations that can be understood by ophthalmologists. In some scenarios, they can easily mislead doctors to make wrong diagnoses, so the auxiliary effect is limited. At present, interpretable fundus image classification methods have yet to be explored. There are some methods for making decisions interpretable in other medical image analysis fields. For example, Patrício et al. (Patrício C, Neves JC, Teixeira L F. Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosis [C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2023: 3798-3807.) proposed a method based on Concept Bottleneck Model (CBM), forcing the convolutional network to first focus on the concept of predefined skin lesions, and then diagnose skin diseases based on the presence or absence of these concepts. Yan et al. (Yan S, Yu Z, Zhang X, et al. Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decision [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 11568-11577.) also proposed a framework based on CBM that allows doctors to intervene in the decision-making of the skin cancer diagnosis model, thereby improving the credibility of the model-assisted diagnosis results. However, the above methods are not applicable to fundus image classification tasks, because the diagnosis of fundus images depends on fundus lesions, and the relationship between fundus lesions and diseases is very complex, and the relationship between lesions and diseases cannot be simply modeled by the presence or absence of lesions. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention proposes an interpretable lesion-aware fundus image classification method and system. By constructing an interpretable lesion-aware fundus image classification model, the lesion-specific features are extracted based on the lesion-aware Transformer, and the lesion concept is enhanced using the fundus basic model. Finally, an interpretable classifier constructed based on the cross-attention mechanism can achieve accurate and rapid fundus image classification while providing corresponding diagnostic basis, greatly improving the reliability of deep learning network-assisted fundus disease diagnosis.

[0006] The technical solution adopted by the method of the present invention is: a method for classifying fundus images with interpretable lesion perception, using a model for classifying fundus images with interpretable lesion perception to classify and interpret fundus images;

[0007] The interpretable lesion-aware fundus image classification model includes an image preprocessing layer, a lesion feature extraction layer, a local information enhancement layer, a knowledge embedding layer, and a cross-attention based classifier;

[0008] The image to be processed passes through the image preprocessing layer, outputs an image block Token of a preset size, and creates a C l A learnable lesion TC; after the image block Token is spliced ​​with the lesion TC, it is input into the feature extraction layer to obtain the image block information. and expression of lesion characteristics Said and The classification result of the lesion is obtained after the local information enhancement layer Said After passing through the knowledge embedding layer and the cross-attention based classifier, the disease classification results are output.

[0009] Preferably, the image to be processed passes through the image preprocessing layer, and is equally divided into a series of N×N image blocks, and then these image blocks are adjusted to T p ∈R M×D The image block Token of the shape, where N is the preset value, M = N 2 , D represents the dimension of the image block Token, and R represents the real number field.

[0010] Preferably, the creation of C l A learnable lesion TC, that is, randomly initialize C before model training l A tensor of dimension D, with Represents, R represents the real number field.

[0011] Preferably, the local information enhancement layer first Resize to N×N size and then project through a 1×1 convolutional layer to R represents the domain of real numbers; finally, Channel adjustment and global maximum pooling (GMP) are implemented to emphasize the most significant features within the block. Implement Global Average Pooling (GAP) to obtain the results of lesion classification for:

[0012]

[0013] Among them, GMP() represents the global maximum pooling operation, and GAP() represents the global average pooling operation.

[0014] Preferably, the knowledge embedding layer uses a dictionary π based on medical knowledge. EK , mapping the lesion name to a text description set with medical knowledge where n * is the name of the lesion, and P is the number of text descriptions of the lesion name. The text encoder is used to obtain the knowledge embedding of the corresponding lesion name. The formula is as follows:

[0015]

[0016] E={E i |i=0,1,…C l}

[0017] Among them, E i represents the knowledge embedding of the ith lesion, E is the set of lesion knowledge embeddings, n i represents the name of the ith lesion, Φ k refers to the text encoder, D k represents the dimension of knowledge embedding, and R represents the real number domain;

[0018] Using adapter Φ a To project the dimension of the lesion TC to be consistent with the dimension of knowledge embedding, and calculate the similarity between the two, we get the similarity matrix S:

[0019]

[0020] Among them, C l is the number of lesions.

[0021] Preferably, in the cross-attention based classifier, for each disease type, there is a learnable disease token corresponding to the disease-specific information, denoted as Among them C dis the number of disease types, D is the dimension of Token, and R is the real number domain. Then, using the cross-attention mechanism, the disease Token is used as the query input Q, and the lesion TC learned in the knowledge embedding layer is used as the key K and value V. The formula is as follows:

[0022]

[0023] Among them, W q ,W k ,W v ∈R D×D are all learnable parameters in the classifier, serving as weights for Q, K, and V, respectively. A is the calculated cross-attention weight matrix, and C l is the number of lesions; GAP(AV) represents the global average pooling of the output sequence AV and outputs the classification result of the disease

[0024] Preferably, the interpretable lesion-aware fundus image classification model, and After the two are processed by Softmax, the confidence of the model about the diagnosis results of each lesion and each disease can be obtained; according to the attention weight A between the disease and the lesion obtained during the cross-attention calculation in the classifier, the contribution of different lesions to the diagnosis results of a specific disease can be obtained after calculating Softmax, and a reliable explanation of the current sample disease diagnosis result can be generated based on the contribution;

[0025] Generate the corresponding text explanation according to the following template:

[0026] The sample is diagnosed as [disease diagnosis result] (disease diagnosis confidence); because [lesion 1] (lesion 1 contribution)... and [lesion n] (lesion n contribution) are observed to exist, and [lesion n+1] (lesion n+1 contribution)... [lesion m] (lesion m contribution) is observed to not exist;

[0027] Among them, n and m refer to the codes of the lesions.

[0028] Preferably, the interpretable lesion-aware fundus image classification model is a trained model;

[0029] During the training process, the adapter Φ is used a Project the dimension of the lesion TC to be consistent with the dimension of the knowledge embedding E, and calculate the similarity between the two to obtain the similarity matrix S:

[0030]

[0031] Among them, C l is the number of lesions, R represents the domain of real numbers;

[0032] Compare S with an identity matrix The cross entropy loss L KG Lesion concept guided learning as optimization target;

[0033]

[0034] Where ∈ is a very small constant, y l It is the lesion marking. represents the label of the i-th lesion under onehot encoding, I i represents the i-th row of the identity matrix, S i , S j Respectively represent the i-th and j-th rows of the similarity matrix S;

[0035] For lesion classification, we use the image-level lesion annotation y of the sample l , and the lesion classification results are used to calculate the multi-label SoftMarginLoss, and the supervised training loss L for lesion classification is obtained. L ;

[0036]

[0037] in, is the lesion outcome predicted by the model, represents the prediction result of the i-th lesion under onehot encoding, and σ refers to the sigmoid function;

[0038] According to the image-level disease annotation y of the sample d , calculate the cross entropy loss with the disease classification results, and obtain the supervised training loss L for disease diagnosis D ;

[0039]

[0040] Among them, C d is the number of disease types, y d It is a disease marker. Represents the label of the i-th disease under onehot encoding; is the disease outcome predicted by the model, and Represent the prediction results of the i-th and j-th diseases respectively;

[0041] Get the final loss function of the model:

[0042] L=L D +αL KG +βL L ;

[0043] Among them, α and β are preset hyperparameters, which control the proportion of lesion concept learning and lesion supervised learning respectively.

[0044] The present invention also provides an interpretable lesion-aware fundus image classification system, comprising:

[0045] one or more processors;

[0046] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the interpretable lesion-aware fundus image classification method.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] (1) The present invention designs a method for extracting fundus lesion features, which creates independent lesion TCs for each lesion based on the Transformer model, thereby learning more discriminative representations during the training process and realizing feature extraction of lesion perception.

[0049] (2) The present invention designs a method for enhancing the concept of fundus lesions, which encodes medical knowledge based on a cross-modal fundus basic large model and uses the idea of ​​comparative learning to narrow the similarity between lesion features and knowledge encoding, thereby guiding the learning of fundus lesion concepts.

[0050] (3) The present invention designs an interpretable classifier, which establishes the relationship between the concept of fundus lesions and diseases based on the cross-attention mechanism. While obtaining the classification results, the attention weight between the disease and the lesion is used to represent the contribution of each lesion to the diagnosis result decision, thereby generating a reliable diagnostic explanation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0052] Figure 1 This is a diagram of the architecture of an explainable lesion-aware fundus image classification model according to an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the principle of the lesion concept learning method according to an embodiment of the present invention.

[0054] Figure 3 A schematic diagram of a decision explanation case according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0056] Please see Figure 1 , this embodiment provides an interpretable lesion-aware fundus image classification method, which uses an interpretable lesion-aware fundus image classification model to classify and interpret fundus images;

[0057] The interpretable lesion-aware fundus image classification model includes an image preprocessing layer, a lesion feature extraction layer, a local information enhancement layer, a knowledge embedding layer, and a cross-attention based classifier;

[0058] The image to be processed passes through the image preprocessing layer, outputs an image block Token of a preset size, and creates a C l A learnable lesion TC; after the image block Token is spliced ​​with the lesion TC, it is input into the feature extraction layer to obtain the image block information. and expression of lesion characteristics Said and The classification result of the lesion is obtained after the local information enhancement layer Said After passing through the knowledge embedding layer and the cross-attention based classifier, the disease classification results are output.

[0059] In one embodiment, the image to be processed passes through the image preprocessing layer, and is divided into a series of N×N image blocks, and then these image blocks are adjusted to T p ∈R M×D The image block Token of the shape, where N is the preset value, M = N 2 , D represents the dimension of the image block Token, and R represents the real number field.

[0060] In one embodiment, the creation of C l A learnable lesion TC, that is, randomly initialize C before model training l A tensor of dimension D, C l is the number of lesions, express.

[0061] The lesion TC and the image block Token are concatenated and input into the Transformer feature extraction layer for feature extraction. and Respectively represent the processed lesion TC and image block Token. At this time, these lesion TCs are the feature expressions of each lesion, thereby realizing lesion perception. Considering that the Transformer feature extraction layer makes more use of the correlation information between blocks and ignores the local information of small-scale lesions, this embodiment enhances the local information of the lesion TC to perform lesion classification. Specifically, first The image is resized to an N×N image block, and then projected to the same dimension as the lesion TC through a 1×1 convolutional layer to obtain the aggregated image block Token, which can be expressed as Finally, the aggregated image block Token is channel-adjusted and global maximum pooling (GMP) is implemented to emphasize the most significant features within the block, while global average pooling (GAP) is implemented on the lesion TC. This can not only utilize the Transformer's ability to capture long-range dependencies but also enhance the locality of the lesion, and output the final lesion classification result. for:

[0062]

[0063] In one embodiment, considering that supervised learning based only on lesion labels cannot capture the semantic information of lesions well, and thus it is difficult to learn the appropriate expression of lesion concepts, this embodiment uses the domain knowledge of the fundus image basic large model to guide and enhance the learning of lesion concepts. Figure 2 The knowledge embedding layer first uses a dictionary π based on medical knowledge EK , mapping the lesion name to a text description set with medical knowledge where n * Refers to the specific lesion name, and P is the number of text descriptions of the lesion name. P may be different for different lesions. Then, the text encoder of the existing cross-modal fundus image basic model FLAIR is directly used to obtain the knowledge embedding of the corresponding lesion name. The formula is as follows:

[0064]

[0065] E={E i |i=0,1,…C l}

[0066] Among them, e i represents the knowledge embedding of the ith lesion, E is the set of lesion knowledge embeddings, n i represents the name of the ith lesion, Φ k Refers to FLAIR's text encoder, D k represents the dimension of knowledge embedding. Then, this embodiment uses an adapter Φa To project the dimension of the lesion TC to be consistent with the dimension of knowledge embedding, and calculate the similarity between the two, we get the similarity matrix S:

[0067]

[0068] Among them, C l is the number of lesions.

[0069] In one embodiment, in clinical diagnosis, ophthalmologists usually judge whether there is a lesion in the fundus image based on professional knowledge and clinical experience, and make a disease diagnosis based on the lesion information. The cross-attention based classifier proposed in this embodiment imitates this process. Specifically, in the classifier, for each disease type, this embodiment creates a learnable disease token to represent disease-specific information, denoted as Among them C d is the number of disease types. Then, using the cross-attention mechanism, the disease token is used as the query input (Q), and the lesion TC learned in step 2 is used as the key (K) and value (V). The formula is as follows:

[0070]

[0071] Among them, W q ,W k ,W v ∈R D×D are all learnable parameters in the classifier, serving as weights for Q, K, and V, respectively, and A is the calculated cross-attention weight matrix. By performing global average pooling on the output sequence, this embodiment can obtain the result of disease diagnosis.

[0072] In one embodiment, after inputting the fundus image to be diagnosed into a trained model, the model's classification results of lesions and diseases in the image can be obtained. and After the two are processed by Softmax, the confidence of the model about the diagnosis results of each lesion and each disease can be obtained. According to the attention weight A between the disease and the lesion obtained during the cross-attention calculation in the classifier, the contribution of different lesions to the diagnosis results of a specific disease can be obtained after calculating Softmax. According to the contribution, a reliable explanation of the current sample disease diagnosis result can be generated.

[0073] Based on the above classification results, the confidence level of the results, and the contribution of the lesion to the disease diagnosis, the model will generate corresponding text explanations based on the following template:

[0074] The sample is diagnosed as [Disease Diagnosis Result] (Disease Diagnosis Confidence). Because [Lesion 1] (Lesion 1 Contribution)... and [Lesion n] (Lesion n Contribution) are observed to exist, and [Lesion n+1] (Lesion n+1 Contribution)... [Lesion m] (Lesion m Contribution) is observed to not exist.

[0075] Among them, n and m refer to the codes of the lesions, which will be replaced by the lesion names in the actual generated explanations, as shown in the attached Figure 3 As shown in the text explanation.

[0076] In one embodiment, the interpretable lesion-aware fundus image classification model is a trained model;

[0077] During the training process, the adapter Φ is used a Project the dimension of the lesion TC to be consistent with the dimension of the knowledge embedding E, and calculate the similarity between the two to obtain the similarity matrix S:

[0078]

[0079] Among them, C l is the number of lesions;

[0080] Compare S with an identity matrix The cross entropy loss L KG Lesion concept guided learning as optimization target;

[0081]

[0082] Where ∈ is a very small constant, y l It is the lesion marking. represents the label of the i-th lesion under onehot encoding, I i represents the i-th row of the identity matrix, S i , S j Respectively represent the i-th and j-th rows of the similarity matrix S;

[0083] For lesion classification, we use the image-level lesion annotation y of the sample l , and the lesion classification results are used to calculate the multi-label SoftMarginLoss, and the supervised training loss L for lesion classification is obtained. L ;

[0084]

[0085] in, is the lesion outcome predicted by the model, represents the prediction result of the i-th lesion under onehot encoding, and σ refers to the sigmoid function;

[0086] According to the image-level disease annotation y of the sample d , calculate the cross entropy loss with the disease classification results, and obtain the supervised training loss L for disease diagnosis D ;

[0087]

[0088] Among them, C d is the number of disease types, y d It is a disease marker. Represents the label of the i-th disease under onehot encoding; is the disease outcome predicted by the model, and Represent the prediction results of the i-th and j-th diseases respectively;

[0089] Get the final loss function of the model:

[0090] L=L D +αL KG +βL L ;

[0091] Among them, α and β are preset hyperparameters, which control the proportion of lesion concept learning and lesion supervised learning respectively.

[0092] The present invention is further described below through experiments.

[0093] The deep learning framework used in this experiment is Pytorch. The hardware environment of the experiment is NVIDIA GeForce RTX4090 graphics card and the processor is Intel(R) Xeon(R) Gold 6240 CPU.

[0094] The specific experimental process is as follows:

[0095] Step 1: Embedding and acquiring lesion knowledge;

[0096] For the names of fundus lesions involved in the dataset, a mapping between lesion names and lesion description text sets is constructed based on medical knowledge. Multiple description texts corresponding to each lesion name are input into the text encoder of the fundus image basic model FLAIR to obtain the knowledge embedding of the lesions.

[0097] Step 2: Building an interpretable lesion-sensing network;

[0098] In the experiment, the Class-Attention in Image Transformers (CaiT) model is used as a feature extractor. On its basis, a single CLSToken is replaced with multiple lesion TCs to independently characterize the corresponding lesions. Then, the image block Token that has passed through the Transformer is projected, aggregated, and globally max-pooled, and the lesion TC is globally average-pooled. The average of the two is used as the result of lesion classification. Then, the domain knowledge of the fundus basic model is used to guide lesion concept learning based on the idea of ​​contrastive learning. Finally, an interpretable classifier is constructed based on the cross-attention mechanism to achieve self-explanatory disease diagnosis.

[0099] Step 3: Model training;

[0100] The data set was divided into 10-fold cross-validation, with 1 fold taken as the test set each time and the rest as the training set until all folds were tested. The fundus images in each round of training set were sent to the model for training, and the multi-label SoftMarginLoss was used to optimize the lesion classification, the cross entropy loss was used to optimize the lesion concept learning and disease diagnosis, and the back propagation was used to optimize the model parameters.

[0101] Step 4: Model testing

[0102] In the 10-fold cross validation, after each round of training, the trained model is inferred using samples in the test set to obtain disease classification results, lesion classification results, the contribution of lesions to the corresponding disease diagnosis results, and the text explanations generated accordingly. The disease classification results in the RAO dataset are evaluated using the F1, Sensitivity, Specificity, and AUC indicators, the disease classification results in the DDR dataset are evaluated using the Kappa, Sensitivity, Specificity, and AUC indicators, and the lesion classification results are evaluated using the Accuracy and F1 indicators. The above indicators can correctly reflect the performance of the corresponding tasks.

[0103] The datasets used in this experiment include retinal fundus images of patients with retinal artery occlusion (RAO) disease collected from a certain hospital and a lesion segmentation subset of the public DDR dataset of patients with diabetic retinopathy (DR).

[0104] The RAO dataset contains a total of 397 images, 3 disease labels, and 6 lesion labels. Among them, there are 253 disease labels for non-RAO, 30 for chronic RAO, and 114 for acute RAO. Image-level lesion labels include 139 for retinal arteriovenous crossing sign (RACS), 88 for cotton wool spots (CWS), 50 for retinal hemorrhage (HE), 45 for segmental changes in retinal vessels (SCRV), 140 for cherry red spots in the macular area (CRS), and 117 for retinal ischemic edema (RW).

[0105] The lesion segmentation subset of the DDR dataset has a total of 757 images, 2 of which do not have disease labels and are therefore removed in the experiment. Considering that there are no images with non-DR disease labels in the segmentation subset, we randomly selected 745 non-DR images from the disease grading dataset of DDR to form a new DDR subset containing a total of 1,500 images for comparative experiments. Among them, the disease labels are: 745 non-DR, 99 mild DR, 548 moderate DR, 34 severe DR, and 74 proliferative DR. Image-level lesion labels: 238 retinal soft exudates (SE), 569 microaneurysms (MA), 599 retinal hemorrhages (HE), and 485 retinal hard exudates (EX).

[0106] In this experiment, the image size is uniformly adjusted to 224*224. The training data is horizontally flipped with a probability of 0.5, randomly rotated 90°, and Gaussian blurred. The image color attributes are changed with a probability of 0.8 by setting the brightness to 0.4, the contrast to 0.4, the saturation to 0.2, and the hue to 0.1. Finally, the mean and standard deviation of each channel are standardized to [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225]. The parameters of the backbone network are initialized with the ImageNet-1K pre-trained parameters. When dividing the image blocks, N is set to 14 and the dimension of the Token is 384. In the learning of lesion concepts, the dimension D of knowledge embedding k is set to 512, ∈ is 1e-8. The number of cross attention heads in the classifier is set to 8. In the final training loss function weights, α and β are set to 0.4 and 0.6 respectively. During training, the AdamW optimizer is used, the initial learning rate is 1e-4, the weight decay is 1e-4, and the CosineAnnealingWarmRestarts is used to dynamically adjust the learning rate. The minimum learning rate is limited to 1e-5. The number of training times is 200, and the batch size is set to 16. The implementation of the entire experiment is based on PyTorch.

[0107] In order to verify the effectiveness of the present invention, this experiment compared the classification results of the present invention with the existing fundus image classification method and the concept-based interpretability model. The main comparison methods are:

[0108] (1)CABNet: He A, Li T, Li N, et al. CABNet: Category attention block forimbalanced diabetic retinopathy grading[J]. IEEE Transactions on MedicalImaging, 2020, 40(1):143-153.

[0109] (2) LANet: Hou J, Xiao F, Xu J, et al. Diabetic Retinopathy Grading with Weakly-Supervised Lesion Priors[C] / / ICASSP 2023-2023IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2023:1-5.

[0110] (3)CCBM: Patrício C,Neves JC,Teixeira L F.Coherent Concept-basedExplanations in Medical Image and Its Application to Skin Lesion Diagnosis[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2023:3798-3807.

[0111] (4)PCBM&PCBM-h: Yuksekgonul M, Wang M, Zou J.Post-hoc Concept BottleneckModels[C] / / The Eleventh International Conference on LearningRepresentations.2022.

[0112] The test was conducted on the RAO dataset collected by a certain hospital. The results are shown in Table 1 below:

[0113] Table 1 Test results on the RAO dataset

[0114]

[0115]

[0116] The test was conducted on a subset of the public DDR dataset, and the results are shown in Table 2 below:

[0117] Table 2 Test results on DDR subset

[0118]

[0119] Among them, CABNet and LANet are the latest fundus image classification methods in recent years, which are not interpretable and cannot classify lesions at the same time. CCBM, PCBM and PCBM-h are the latest concept-based interpretability models in recent years. It can be seen from Tables 1 and 2 that compared with fundus image classification methods in recent years, the method proposed in the present invention has achieved the best performance in disease classification; compared with concept-based interpretability models in recent years, the method of the present invention is significantly better than other methods in both disease classification and lesion classification. In addition, as shown in the attached Figure 3 As shown in the figure, the method of the present invention can generate textual explanations that are faithful to the disease diagnosis results. By visualizing the lesion area located by the model and the confidence of the disease classification lesion classification, the model of the present invention can provide ophthalmologists with sufficient disease-related information and effectively and reliably assist doctors in diagnosing fundus diseases. The experimental results on the above two datasets prove the effectiveness and generalization of the method proposed in the present invention.

[0120] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0121] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A method for interpretable lesion-aware fundus image classification, characterized by: An interpretable lesion-aware fundus image classification model is used to classify and interpret fundus images. The interpretable lesion-aware fundus image classification model includes an image preprocessing layer, a lesion feature extraction layer, a local information enhancement layer, a knowledge embedding layer, and a cross-attention based classifier; The image to be processed passes through the image preprocessing layer, outputs an image block Token of a preset size, and creates a C l A learnable lesion TC; after the image block Token is spliced ​​with the lesion TC, it is input into the feature extraction layer to obtain the expression of the image block information and expression of lesion characteristics Said and The classification result of the lesion is obtained after the local information enhancement layer Said After passing through the knowledge embedding layer and the cross-attention based classifier, the disease classification results are output. The local information enhancement layer first Resize to N×N size and then project through a 1×1 convolutional layer to R represents the domain of real numbers; finally, Channel adjustment and global maximum pooling are performed to emphasize the most salient features within the block. Implement global average pooling to obtain the result of lesion classification for: Among them, GMP() represents the global maximum pooling operation, and GMP() represents the global average pooling operation; The knowledge embedding layer uses a dictionary π based on medical knowledge EK , mapping the lesion name to a text description set with medical knowledge where n * is the name of the lesion, and P is the number of text descriptions of the lesion name. The text encoder is used to obtain the knowledge embedding of the corresponding lesion name. The formula is as follows: And={And i ∣i=0,1,…C l }; Among them, E i represents the knowledge embedding of the ith lesion, E is the set of lesion knowledge embeddings, n i represents the name of the ith lesion, Φ k refers to the text encoder, D k The dimension representing knowledge embedding; Using adapter Φ a To project the dimension of the lesion token to be consistent with the dimension of the knowledge embedding, and calculate the similarity between the two, we get the similarity matrix S: Among them, C l is the number of lesions, and R represents the domain of real numbers.

2. The interpretable lesion-aware fundus image classification method according to claim 1, characterized in that: The image to be processed passes through the image preprocessing layer and is divided into a series of N×N image blocks, and then these image blocks are adjusted to T p ∈R M×D The image block Token of the shape, where N is the preset value, M = N 2 , D represents the dimension of the image block Token, and R represents the real number field.

3. The interpretable lesion-aware fundus image classification method according to claim 1, characterized in that: The creation of C l A learnable lesion TC is randomly initialized C l A tensor of dimension D, with Represents; R represents the real number field.

4. The interpretable lesion-aware fundus image classification method according to claim 1, characterized in that: In the cross-attention based classifier, for each disease type, there is a learnable disease token corresponding to the disease-specific information, denoted as Among them C d is the number of disease types, D is the dimension of Token, and R is the real number domain. Then, using the cross-attention mechanism, the disease Token is used as the query input Q, and the lesion TC learned in the knowledge embedding layer is used as the key K and value V. The formula is as follows: Among them, W q ,W k ,W v ∈R D×D are all learnable parameters in the classifier, serving as weights for Q, K, and V, respectively. A is the calculated cross-attention weight matrix, and C l is the number of lesions; GAP(AV) represents the global average pooling of the output sequence AV and outputs the classification result of the disease 5. The interpretable lesion-aware fundus image classification method according to claim 1, characterized in that: The interpretable lesion-aware fundus image classification model, and After the two are processed by Softmax, the confidence of the model about the diagnosis results of each lesion and each disease can be obtained; according to the attention weight A between the disease and the lesion obtained during the cross-attention calculation in the classifier, the contribution of different lesions to the diagnosis results of a specific disease can be obtained after calculating Softmax, and a reliable explanation of the current sample disease diagnosis result can be generated based on the contribution; Generate the corresponding text explanation according to the following template: The sample is diagnosed as [disease diagnosis result] (disease diagnosis confidence); because [lesion 1] (lesion 1 contribution)... and [lesion n] (lesion n contribution) are observed to exist, and [lesion n+1] (lesion n+1 contribution)... [lesion m] (lesion m contribution) is observed to not exist; Among them, n and m refer to the codes of the lesions.

6. The interpretable lesion-aware fundus image classification method according to any one of claims 1 to 5, characterized in that: The interpretable lesion-aware fundus image classification model is a trained model; During the training process, the adapter Φ is used a Project the dimension of the lesion TC to be consistent with the dimension of the knowledge embedding E, and calculate the similarity between the two to obtain the similarity matrix S: Among them, C l is the number of lesions, R represents the domain of real numbers; Compare S with an identity matrix The cross entropy loss L KG Lesion concept guided learning as optimization target; Where ∈ is a very small constant, y l It is the lesion marking. represents the label of the i-th lesion under onehot encoding, I i represents the i-th row of the identity matrix, S i , S j Respectively represent the i-th and j-th rows of the similarity matrix S; For lesion classification, we use the image-level lesion annotation y of the sample l , and the lesion classification results are used to calculate the multi-label SoftMarginLoss, and the supervised training loss L for lesion classification is obtained. L ; in, is the lesion outcome predicted by the model, represents the prediction result of the i-th lesion under onehot encoding, and σ refers to the sigmoid function; According to the image-level disease annotation y of the sample d , calculate the cross entropy loss with the disease classification results, and obtain the supervised training loss L for disease diagnosis D ; Among them, C d is the number of disease types, y d It is a disease marker. Represents the label of the i-th disease under onehot encoding; is the disease outcome predicted by the model, and Represent the prediction results of the i-th and j-th diseases respectively; Get the final loss function of the model: L=L D +αL KG +βL L ; Among them, α and β are preset hyperparameters, which control the proportion of lesion concept learning and lesion supervised learning respectively.

7. A lesion-aware fundus image classification system with interpretability, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the interpretable lesion-aware fundus image classification method as described in any one of claims 1 to 6.

Images