A training method of a classification model and an eye fundus image classification method

Abstract

The application provides a kind of training method of classification model and fundus image classification method, belong to the field of incremental learning.A kind of training method of classification model, the classification model is used for fundus image classification, the method comprises: obtaining pre-trained classification model as teacher model, it includes feature extraction network and classifier, the fundus class that the teacher model can identify is classified as old class;Student model is obtained, the feature extraction network of which is initialized with the teacher model, and its classifier is set to be able to identify the fundus class corresponding to the old class and new class, the new class is the fundus class outside the old class;For each old class used, obtain the pseudo sample corresponding to the old class;The student model is incrementally trained using the generated pseudo sample of old class and the fundus image belonging to new class, and the parameters of the student model are updated based on the total loss determined by the preset total loss function during training, to obtain the student model after incremental training.The present application can alleviate catastrophic forgetting.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of incremental learning, in particular to the field of medical imaging applying incremental learning, and more particularly to a training method of a classification model and an eye fundus image classification method. BACKGROUND

[0002] Deep neural networks (DNNs) have shown great performance in many machine learning classification tasks, for example, the performance of deep neural networks in medical imaging applications has reached the level of human beings, and recent studies have shown that deep learning-based models can be successfully applied to retinal disease (such as diabetic retinopathy DR and glaucoma) screening. Among them, the model applied in retinal disease screening is assumed to be implemented under the condition that all classes are known before training. However, this assumption is often violated in the medical field. In the actual training process, it is extremely difficult to prepare data of all classes in advance, for example, due to the different stages of some diseases, the degree of disease change will also change slightly, so it is difficult to prepare data of the degree of disease change in advance. In addition, if a model only learns to identify the generalization class to which the input sample (such as the sample corresponding to the disease) belongs, but cannot accurately identify the specific class (such as the class corresponding to the different stages and degrees of disease) to which it belongs, this will have extremely serious consequences. Because the output result of the model after training for the input sample will affect the subsequent application effect. For example, different treatment strategies are used for the same disease at different stages of the disease, and starting the correct treatment plan is the key to achieving good treatment results. Therefore, the class incremental learning method in the machine learning classification task can identify the specific class (such as the class corresponding to the degree of disease change and the stage of disease) to which it belongs, so that in subsequent applications, it can provide great help in providing continuous and effective, timely and accurate health detection and disease warning for patients. The main challenge to be solved in the incremental learning method is catastrophic forgetting. Intuitively, catastrophic forgetting is caused by the overlap or confusion between the representations of new and old classes in the feature space. When learning new classes, the decision boundary of the previous classes may change greatly, and the unified classifier will have a serious bias.

[0003] To address this challenge (catastrophic forgetting), two different breakthrough directions are included in existing methods: the first is to bias the parameters towards the direction of learning on the old classes; the second is to keep a small data buffer from previous tasks (this is also known as experience replay). The regularization strategy adopted for the first is effective in the scenario of using multi-head classifiers and leveraging the available task identifiers at inference time. For the second, the most common approach is to implement experience replay by saving a small number of real data for subsequent model training. In addition, some methods also employ an additional generative model, such as a generative adversarial network (GNN), to generate data to implement experience replay.

[0004] However, as some literature has noted, methods that address catastrophic forgetting perform poorly in the class-incremental learning (CIL) scenario. The most effective regularization strategy currently is based on knowledge distillation, which forces the student model to completely mimic the teacher model. Specifically, the distilled student model aims to mimic the logits of the teacher model's model fully connected layer output on the training samples to obtain similar generalization performance as the teacher model. However, completely mimicking the output of the teacher model can not be optimal, as the teacher model can confidently mispredict some classes, which increases the risk of transferring incorrect information during the incremental process. For the experience replay method, a large amount of memory is often needed to replay previously seen or modeled data to avoid the problem of catastrophic forgetting. However, in some practical scenarios (e.g., on devices for Internet of Things applications or privacy issues), data storage can be limited due to memory constraints. This focuses people on incrementally incorporating new information without storing old knowledge, which is called non-exemplar-based incremental learning.

[0005] In non-save sample-based incremental learning, in order to further learn more useful knowledge from the teacher model to determine which knowledge of the teacher model helps to establish a better student model, in addition to the traditional logit-based distillation method, feature-based distillation method has also attracted much attention. This is because the features of the teacher model have more information than the logit-based model, and using feature distillation can make the student model learn more abundant information. However, most feature-based distillation studies manually link the features of the teacher model and the student model, and distill through separate links, which has the risk of imposing incorrect intermediate processes on the student. In addition, selecting a small number of links among all possible links (in other words, in the existing method, the step of distilling the student model is selected by using a manual method to select some links that are considered to be representative), which will also limit the teacher model to fully utilize all its knowledge; Moreover, in most knowledge distillation cases, the features of the student model and the teacher model have different widths, heights and channels, and convolutional layers or fully connected layers are usually used to match their sizes. This makes it possible to lose some useful information of the teacher model in the feature adjustment process.

[0006] Based on the above problems, incremental learning is applied to the field of medical imaging, especially in retinal abnormality screening, the patent CN106022368A has a catastrophic forgetting problem due to category difference. SUMMARY

[0007] Therefore, the purpose of the present application is to overcome the defects of the prior art, and provide a training method of a classification model and an eye fundus image classification method.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] According to a first aspect of the present application, a training method of a classification model for fundus image classification is provided, the method comprising: obtaining a pre-trained classification model as a teacher model, the teacher model comprising a feature extraction network for extracting image features from an input fundus image and a classifier for identifying a fundus category to which the image features correspond based on the image features, the classifier comprising a fully connected layer and a Softmax layer, the fundus categories identifiable by the teacher model being classified as old categories; obtaining a student model, the trainable parameters of the feature extraction network of the student model being initialized by the trainable parameters of the feature extraction network of the teacher model, and the classifier of the student model being configured to identify fundus categories corresponding to old categories and new categories, the new categories being fundus categories other than the old categories; for each old category used in the pre-training, obtaining a pseudo sample corresponding to the old category, the pseudo sample being generated by extracting image features of multiple fundus images belonging to the old category using the feature extraction network of the teacher model; incrementally training the student model using the generated pseudo samples of the old categories and fundus images belonging to the new categories, the parameters of the student model being updated based on a total loss determined according to a preset total loss function during the training, to obtain an incrementally trained student model, the total loss being determined according to a weighted sum of an old category classification loss, a new category classification loss, a first feature distillation loss and a second feature distillation loss; wherein the first feature distillation loss is determined according to a difference between outputs of the pseudo sample of the old category on the following two aspects: an output of the fully connected layer of the teacher model, and an output of the fully connected layer of the student model on the old category; and the second feature distillation loss is determined according to a total difference between all teacher features extracted by the feature extraction network of the teacher model and all student features extracted by the feature extraction network of the incrementally trained student model, wherein each teacher feature is an image feature extracted by the feature extraction network of the teacher model from a pseudo sample, and each student feature is an image feature extracted by the feature extraction network of the student model from the pseudo sample and a fundus image belonging to the new categories.

[0010] In some embodiments of the present application, the total difference is calculated by the following steps: determining an attention value of each teacher feature to each student feature based on an attention mechanism, and constructing an attention vector of the teacher feature to all student features using the attention value of the teacher feature to each student feature and normalizing the attention vector to obtain a normalized attention vector; determining a spatial distance between each teacher feature and each student feature; calculating a product of the spatial distance and a corresponding element in the normalized attention vector, and summing all the products to obtain the total difference.

[0011] In some embodiments of the present application, based on the attention mechanism, the attention value of each teacher feature to each student feature is calculated by the following steps: data conversion is performed on the student feature to obtain a key of the student feature in the attention mechanism; data conversion is performed on the teacher feature to obtain a query of the teacher feature in the attention mechanism; and the attention value of each query to each key is calculated.

[0012] In some embodiments of the present application, data conversion is performed by the following rules:

[0013]

[0014]

[0015] wherein q t represents the query in the attention mechanism, represents the tth teacher feature, P HW (·) represents global average pooling, represents the linear transformation parameter of P represents the linear transformation parameter space matrix of P Q (·) represents the first activation function, k s represents the key in the attention mechanism, represents the sth student feature, represents the linear transformation parameter of k represents the linear transformation parameter space matrix of k represents the linear transformation parameter space matrix of k K (·) represents the second activation function, d represents the dimension of the linear transformation parameter space matrix.

[0016] In some embodiments of the present application, the attention vector of each query to all keys is calculated and normalized by the following rules:

[0017]

[0018] wherein softmax(.) represents a normalization function, represents the transpose of q t represents a bilinear weight, k t，1 represents the corresponding key value of q ​​position encoding representing the t-th teacher feature, position encoding representing the s-th student feature, k t，S position encoding representing the corresponding key value, position encoding representing the product of and position encoding representing the transpose of

[0019] In some embodiments of the present application, the second feature distillation loss is calculated by the following method:

[0020]

[0021] wherein, α t，s normalized attention value of the t-th teacher feature to the s-th student feature, represents the spatial distance, ||.||2 represents the L2 norm, represents the combination function of channel average pooling layer and L2 normalization v / ||v||2, v represents the vector obtained by averaging pooling, represents the feature obtained by using down-sampling or up-sampling.

[0022] In some embodiments of the present application, the pseudo sample of the old class is obtained according to the following steps:

[0023] T1, calculating the class mean vector corresponding to the old class based on the plurality of image features corresponding to the old class used in the pre-training; T2, randomly sampling at least one noise vector from a Gaussian distribution, the noise vector and the class mean vector are of the same dimension; T3, based on the pre-defined enhancement scale, the noise vector randomly sampled from the Gaussian distribution and the class mean vector corresponding to the old class, the pseudo sample of the old class is calculated according to the preset enhancement rule.

[0024] In some embodiments of the present application, the method comprises multiple incremental training of the student model, wherein the teacher model is the student model after the last incremental training of the current incremental training, and the class mean vector corresponding to the old class is calculated by the following rule:

[0025]

[0026] wherein, represents the class mean vector corresponding to the old class k old , represents the number of samples of the old class k old , F(X b-1，n ; θ​​b-1 represents the sample feature of the nth sample in the old class k old b-1 represents the parameters of the feature extraction network of the teacher model in the bth incremental training.

[0027] In some embodiments of the present application, the preset enhancement rule is:

[0028]

[0029] wherein, represents the old class k used in the bth incremental training old corresponds to the pseudo sample, r b represents the predefined enhancement scale in the bth incremental training, e represents a random noise vector sampled from a Gaussian distribution, e and have the same dimension.

[0030] In some embodiments of the present application, the predefined enhancement scale is set by the following rule:

[0031]

[0032] wherein, K old and K mew respectively represent the number of old classes and new classes in the bth incremental training, r b-1 represents the predefined enhancement scale in the bth-1 incremental training, ∑ b，k represents the covariance matrix of the kth new class in the bth incremental training, Tr(∑ b，k ) represents the rank of ∑ b，k , and M represents the dimension of the deep feature space, when b = 1, K1 represents the number of classes in the first pre-training.

[0033] In some embodiments of the present application, the total loss is calculated according to the following rule:

[0034]

[0035] wherein, represents the total loss, represents the new class classification loss, represents the old class classification loss, and λ represents the first hyperparameter, represents the first feature distillation loss, and β represents the second hyperparameter, represents the second feature distillation loss, and γ represents the third hyperparameter.

[0036] In some embodiments of the present application, the first feature distillation loss is calculated by the following rule:

[0037] ​

[0038] wherein, represents the pseudo sample data used in the incremental training, represents the sample features extracted from the pseudo sample used in the incremental training by the feature extraction network in the student model, represents the parameters of the full connection layer in the classifier of the student model in the incremental training, G b represents the output of the corresponding full connection layer on the old class in the processing of the classifier of the student model, represents the sample features extracted from the pseudo sample data used in the incremental training by the feature extraction network in the teacher model, represents the parameters of the full connection layer in the classifier of the teacher model in the incremental training, represents the output of the full connection layer in the processing of the teacher model, represents the sum of the difference values between the corresponding elements between and, wherein, ||.||2 represents the L2 norm.

[0039] In some embodiments of the present application, the old class includes any two of the following categories, and the remaining categories are new classes: no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy, proliferative diabetic retinopathy.

[0040] According to the second aspect of the present application, there is provided an eye fundus image classification method, which comprises:

[0041] S1, obtaining an eye fundus image to be classified; S2, using the student model subjected to the incremental training obtained by the method provided in the first aspect of the present application to perform image classification on the eye fundus image to be classified obtained in step S1, wherein the feature extraction network of the student model is used to extract image features of the eye fundus image to be classified, and the classifier of the student model is used to identify the eye fundus class to which the eye fundus image to be classified belongs according to the image features.

[0042] In some embodiments of the present application, the result of image classification includes: no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy, proliferative diabetic retinopathy.

[0043] Compared with the prior art, the application has the advantages that the student model is incrementally trained by using the generated pseudo samples of the old class and the fundus images belonging to the new class, and the introduction of the old class classification loss, the second feature distillation loss and the new class classification loss can well guide the classification model to perform incremental learning, so that the overall classification ability (accuracy) of the classification model after incremental training on the fundus images or fine-grained fundus images of all categories is obviously improved, and the forgetting problem in the incremental learning is alleviated. BRIEF DESCRIPTION OF DRAWINGS

[0044] The embodiments of the application will be further described below with reference to the drawings, in which:

[0045] Figure 1 FIG. 1 is a structure and incremental training process diagram of a classification model according to an embodiment of the application;

[0046] Figure 2 FIG. 3 is a diagram of experimental results according to another embodiment of the application;

[0047] Figure 3 FIG. 4 is a diagram of experimental results according to another embodiment of the application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0049] As pointed out in the background art, incremental learning is applied to the field of medical imaging, especially when screening for retinal abnormalities, the problem of catastrophic forgetting caused by category difference in the process of incremental acquisition of fundus images. To solve this problem, the inventors of the present application first analyzed the existing technology of fundus image disease recognition scheme for data increment, such as the patent document with the patent publication number CN113066025A, which proposed an image defogging method based on incremental learning and feature, attention transfer. This method does not solve the problem of catastrophic forgetting in the increment well; the patent document with the patent publication number CN112990280-A adopts a class-increment method on the image classification problem, but does not further focus on the fine-grained classification problem in the class-increment process. Therefore, it is found that most of them do not make full use of the feature relationship and semantic relationship between new and old data in the incremental process (in other words, the existing class-increment process does not make full use of incremental learning based on feature distillation and incremental learning based on relationship distillation), and the problem of catastrophic forgetting is still not well solved. In addition, the method widely used to solve the problem of catastrophic forgetting is to use the distillation loss and cross-entropy loss as the loss function to guide the model update, which has the following shortcomings: 1. It does not explore which features extracted by the deep learning model in the incremental process are key features; 2. It does not make full use of the representation feature knowledge learned by the model in the incremental process to realize the transfer of key features in the model. Therefore, it is a key solution direction to find out which attributes the representation features obtained by the model in the incremental process have to alleviate the catastrophic forgetting in the incremental process, and further to frame the boundary range of the representation feature knowledge learned by the model, which can reduce the risk of error information transmission in the incremental process and thus alleviate the catastrophic forgetting.

[0050] Thus, the present application proposes a training method of a classification model for fundus image classification and a classification method of fundus image based on the classification model aiming at the catastrophic forgetting problem caused by category difference in the process of fundus image increment. During training, firstly, the relative similarity between teacher features and student features is learned by constructing a neural network model based on attention (the neural network model is an existing basic network structure used to simulate the calculation process in the attention mechanism), in other words, the relative similarity between teacher features and student features is learned by using the attention mechanism, and the recognized similarity is applied to control the distillation strength between all feature pairs (a group of teacher features and a group of student features form a feature pair), and the decision boundary of the previous category is maintained by saving the pseudo sample set representing the class in the representation space for each old class, which alleviates the restriction of memory or privacy problem of data in the incremental process (in other words, the model learns the representation feature knowledge by using the pseudo sample to maintain the decision boundary of the previous class, which also reduces the demand of data on memory or privacy in the incremental process, thereby well dealing with the catastrophic forgetting problem caused by category difference). Wherein, each teacher feature is an image feature extracted from a pseudo sample by a feature extraction network in the teacher model, and each student feature is an image feature extracted from a pseudo sample and a fundus image belonging to a new class by a feature extraction network in the student model. In addition, the pseudo sample is used to maintain the decision boundary of the previous class, which alleviates the forgetting effect of the model in the incremental process and improves the accuracy of the model. It should be noted that the forgetting of the model in the incremental process is due to the increase of the category with the increase of the category increment step, and the overall classification ability (accuracy) of the model for all categories also gradually decreases with the increase of the incremental step. The pseudo sample is used to participate in subsequent incremental learning, which can alleviate the forgetting and reduce the restriction of memory or privacy problem on data in the incremental process.

[0051] According to one embodiment of the present application, a training method of a classification model for fundus image classification is provided, the method comprising: obtaining a pre-trained classification model as a teacher model, which comprises a feature extraction network for extracting image features from an input fundus image and a classifier for identifying the fundus category to which the image features correspond according to the image features, the classifier comprising a fully connected layer and a Softmax layer, the fundus categories identifiable by the teacher model being classified as old categories; obtaining a student model, the trainable parameters of the feature extraction network of which are initialized by the trainable parameters of the feature extraction network of the teacher model, and the classifier of which is configured to be able to identify fundus categories corresponding to old categories and new categories, the new categories being fundus categories other than the old categories; for each old category used in the pre-training, obtaining a pseudo sample corresponding to the old category, the pseudo sample being generated by extracting image features of multiple fundus images belonging to the old category using the feature extraction network of the teacher model; incrementally training the student model using the generated pseudo samples of the old categories and fundus images belonging to the new categories, the parameters of the student model being updated based on a total loss determined according to a preset total loss function during the training, to obtain an incrementally trained student model, the total loss being determined according to a weighted sum of an old category classification loss, a new category classification loss, a first feature distillation loss, and a second feature distillation loss; wherein the first feature distillation loss is determined according to the difference between the outputs of the pseudo samples of the old categories on the following two: the output of the fully connected layer of the teacher model, and the output of the fully connected layer of the student model on the old categories; the second feature distillation loss is determined according to the total difference between all teacher features extracted by the feature extraction network of the teacher model and all student features extracted by the feature extraction network of the incrementally trained student model, wherein in the present application, each teacher feature is an image feature extracted by the feature extraction network of the teacher model from a pseudo sample, and each student feature is an image feature extracted by the feature extraction network of the student model from a pseudo sample and a fundus image belonging to the new categories. Wherein the incrementally training of the student model using the generated pseudo samples of the old categories and fundus images belonging to the new categories, and the introduction of the old category classification loss, the second feature distillation loss, and the new category classification loss can well guide the incremental learning of the classification model, so that the overall classification ability (accuracy) of the incrementally trained classification model for all categories of fundus images or fine-grained fundus images is significantly improved, and the forgetting problem in incremental learning is alleviated. In order to better illustrate the embodiments of the present application, first introduce the data set and the structure of the classification model used for incremental training.

[0052] For the data set, the embodiment of the present application adopts fundus images and labels corresponding to the fundus images to constitute the data set, also known as original data. Preferably, the labels corresponding to the fundus images in the embodiment of the present application according to the severity of diabetic retinopathy (DR) include no diabetic retinopathy (no DR), mild diabetic retinopathy (mild DR), moderate diabetic retinopathy (moderate DR), severe diabetic retinopathy (severe DR), and proliferative diabetic retinopathy (proliferative DR). The data stream of the original data is composed of different tasks, and the data stream of the original data can be described as: wherein is the data received by the system in step b (the step b in the embodiment of the present application is also referred to as the bth incremental training, which is not described elsewhere), X b represents all fundus image samples in step b, Y b represents the category labels corresponding to all fundus image samples in step b. The data set D b contains N b labeled data for training, and y b , j∈C b , wherein C b is a category set (also referred to as a category label set) in step b, and the category sets in different steps are mutually exclusive. It should be noted that the mutually exclusive category sets in different steps are the most difficult training scenarios in incremental training. The embodiment of the present application takes the most difficult training scenario as an example to explain the training method of the classification model. Other easier training scenarios, such as the scenario in which the category sets in different steps partially intersect, are also applicable to the present application. The most difficult training scenario refers to the incremental process of category incremental learning, in which the categories in each incremental step are different, for example, the first incremental step makes the model learn categories A and B, the second incremental step adds training data for categories C and D, but the model is expected to classify A, B, C, and D after training. Each category of data has its own most essential key features, and in the incremental process, the model learned in the incremental step may forget the essential key features of the previous categories (cat and dog categories), thereby affecting the classification accuracy. In order to evaluate the effectiveness of the method provided by the present application, two data sets can be used in the embodiment of the present application. They include the public medical image data set EyePACS and another private data set from the internal data set provided by the Aier Hospital, EyePACS contains 35125 fundus images, and is graded according to the severity of diabetic retinopathy (DR), and the fundus images are divided into 5 categories: no DR, mild DR, moderate DR, severe DR, and proliferative DR. Another private data set from the internal data set provided by the Aier Hospital also includes the above 5 categories, and contains a total of 5479 fundus images.

[0053] To illustrate the goal achieved by the embodiment of the present application with step b as an example, in step b, the goal is to minimize the loss function of the new data set D b on the old data set D and accurately classify the new class set C b without affecting the knowledge learned in the previous steps, and can improve the knowledge learned before to a certain extent on the premise that the current step b data stream (for example, training data) can be accurately classified, thereby alleviating the forgetting problem in the incremental process.

[0054] For a classification model, in class-incremental learning (CIL), the goal of CIL is to learn a unified classification model in sequence, and classify the test samples of all classes learned at the end of training. The classification model includes a feature extractor F θ (also known as a feature extraction network) and a unified classifier G φ . For convenience of description, in the process of incremental training, the classification model in the current incremental training step is referred to as the student model, and the classification model in the previous incremental training of the current incremental training is referred to as the teacher model. As Figure 1 shown, in step b-1, the teacher model M b-1 includes a feature extraction network and a classifier, and the classifier includes a fully connected layer and a Softmax layer, at this time the fundus classes that the classifier of the teacher model can recognize are the fundus classes that the teacher model can recognize at this time are classified as old classes; in step b, the student model M b includes a feature extraction network and a classifier, and the classifier includes a fully connected layer and a Softmax layer, at this time the student model's classifier is set to be able to recognize the fundus classes corresponding to the old classes and the new classes, which are the fundus classes other than the old classes. The feature extraction network of the student model is used to extract image features from the fundus image, and the classifier of the student model is used to identify the fundus class to which the image features correspond. The trainable parameters of the feature extraction network in the student model are initialized with the trainable parameters of the feature extraction network in the teacher model. When the student model is incrementally trained in step b, the parameters of the teacher model are frozen and will not be updated. The new classes that the student model's classifier can recognize can also be understood as: for step b-1, the newly added classes in step b. According to an embodiment of the present application, the old classes include any two of the following classes, and the remaining classes are new classes: no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy, proliferative diabetic retinopathy. Among them, the number of classes in the new class and the old class can be set to other numbers according to demand, which will not be described here.

[0055] In order to better explain the understanding of the embodiments of the present application, the following mainly explains from the aspects of the generation of pseudo samples, the incremental training process, and experimental verification.

[0056] I. Generation of pseudo samples

[0057] As mentioned earlier, without fully utilizing the knowledge of the learned representation features in the incremental process to realize the migration of key features in the model, the problem of catastrophic forgetting will occur. In other words, when training a new task with an old model (such as a teacher model), knowledge forgetting occurs, which in turn leads to a significant shift in the decision boundary of the old class and causes the feature extraction network of sequential learning to easily bias towards the new class. To alleviate the problem of catastrophic forgetting in incremental learning, unlike existing memory-based methods, the present application focuses on the latent feature space and generates pseudo samples corresponding to the original data to participate in incremental training to alleviate catastrophic forgetting. In the embodiments of the present application, it is also called pseudo sample enhancement (PSA). PSA only enhances the feature vector from the output of the old feature extractor (the feature extraction network in the teacher model as the old feature extractor relative to the student model), and uses these enhanced feature vectors (also called pseudo samples) to represent the distribution of the previous task data. The joint training of new data (which includes pseudo samples and eye fundus image data corresponding to new classes in the incremental step) in the latent space helps to alleviate catastrophic forgetting. According to an embodiment of the present application, the pseudo samples of the old class are obtained according to the following steps: T1, calculating the class mean vector corresponding to the old class based on the image features of the old class used in the pre-training; T2, randomly sampling at least one noise vector from a Gaussian distribution, the noise vector and the class mean vector have the same dimension; T3, based on the pre-defined enhancement scale, the noise vector randomly sampled from the Gaussian distribution, and the class mean vector corresponding to the old class, the pseudo sample of the old class is calculated according to the preset enhancement rule.

[0058] It should be noted that the training method of the classification model provided by the embodiments of the present application can perform multiple incremental training on the student model, wherein the pre-trained classification model is the student model after the last incremental training of the current incremental training. Here, the incremental training of the student model can be understood as a kind of pre-training of the classification model. According to an embodiment of the present application, the class mean vector corresponding to the old class is calculated by the following rule:

[0059]

[0060] wherein, represents the class mean vector of the old class k old , represents the number of samples of the old class k old , represents the number of samples of the old class k oldsample feature of the nth sample, θ b-1 denotes the parameters of the feature extraction network of the teacher model in the bth incremental training. It should be noted that the multiple image features corresponding to the old class can be all image features corresponding to the old class, or image features that can represent the old class, i.e., the old class k old The number of samples of the old class is less than or equal to the total number of all samples corresponding to the old class. In addition, in formula (1), a first correction coefficient and other mathematical transformations can be added to increase the ability of the class mean vector to represent the old class, for example wherein is the first correction coefficient.

[0061] When learning a new task (the bth incremental training is referred to as a new task for the b-1th incremental training), the prototype (wherein the prototype can be understood as the class mean feature vector of the old class) of each old class is enhanced. According to an embodiment of the present application, the preset enhancement rule is:

[0062]

[0063] wherein, denotes the old class k old corresponding to the pseudo sample, r b denotes the predefined enhancement scale in the bth incremental training, e denotes a random noise vector sampled from a Gaussian distribution, and e and have the same dimension.

[0064] It should be noted that PSA assumes that the feature embedding obtained from the old feature extractor is extracted from a multivariate Gaussian distribution. In step b, only the data D b can be trained, in order to alleviate the deformation of the feature space when learning a new task, the embodiments of the present application preferably calculate and memorize the class embedding by outputting the mean value of each class embedding, for each class in the previous task (the b-1th incremental training is referred to as a previous task for the bth incremental training), before training the model on the new task (the bth incremental training is referred to as a new task for the b-1th incremental training), a random feature embedding of each old class is generated to perform pseudo enhancement, which obeys a Gaussian distribution. Specifically, given a random noise e from a standard normal distribution , which has the same dimension as each class embedding, the embodiments of the present application apply a linear transformation (formula 2) to enhance the feature embedding of each old class, and the class label of the old class is used as the label of the corresponding pseudo sample.

[0065] wherein, r(r bThe pre-defined scale of enhancement r (b) in the b-th incremental training can control the scale of enhancement of the class mean feature vector of the old classes. Specifically, the scale of enhancement r can be pre-defined or calculated as the average variance of the class representation. According to an embodiment of the present application, the pre-defined scale of enhancement is set by the following rule:

[0066]

[0067] where K old and K mew represent the number of old classes and new classes in the b-th incremental training, respectively, r b-1 represents the pre-defined scale of enhancement in the b-1-th incremental training, ∑ b，k represents the covariance matrix of the k-th new class in the b-th incremental training, Tr(∑ b，k ) represents the rank of ∑ b，k , and M represents the dimension of the deep feature space. When b = 1, K1 represents the number of classes in the first pre-training, is the average variance of the class representation. In the embodiments of the present application, the PSA component aims to generate more pseudo samples using the statistical information of each class in the embedding space, and considers the covariance matrix ∑ b，k represents the k classes in step b, and the static representation is the variance between any pair of elements in the feature embedding; the Tr operation is to calculate the trace of the covariance matrix. In the process of the class incremental learning CIL experiment, r changes slightly in different steps. Therefore, in the first incremental training or pre-training, the average variance of the class representation is: In addition, in formula (3), a second correction coefficient or other mathematical transformations can also be added to better control the scale of enhancement of the class mean feature vector of the old classes, for example, represents the second correction coefficient.

[0068] II. Incremental training process

[0069] When starting to train the new task, the student model is incrementally trained using the generated pseudo samples of the old class and the fundus images belonging to the new class. The parameters of the student model are updated based on the total loss determined by the preset total loss function during training, and an incrementally trained student model is obtained. The total loss is determined by the weighted sum of the following losses: old class classification loss, new class classification loss, first feature distillation loss, and second feature distillation loss. Wherein, training the new task, that is, starting a new incremental training, uses joint training from the fundus images corresponding to the new class and the enhanced pseudo samples (from the old class), extracts features through the feature extraction network in the student model, and then inputs the extracted features into the classifier (a unified classifier for all classes, including the old class and the new class) of the student model. The classifier of the student model obtains the corresponding classification probability (which includes the classification probability of the pseudo sample on the old class and the classification probability of the fundus image corresponding to the new class on the new class) and the output logit of the pseudo sample of the old class on the old class on the full connection layer of the student model. In addition, the pseudo sample is input into the teacher model to obtain the image features after feature extraction of the pseudo sample of the old class in the teacher model and the output logit of the pseudo sample of the old class on the full connection layer of the student model. The first feature distillation loss is determined according to the difference between the outputs of the pseudo sample of the old class on the full connection layer of the teacher model and the full connection layer of the student model on the old class. In order to explore which features extracted by the deep learning model in the incremental process are the key features, the second feature distillation loss is determined based on the total difference between all teacher features extracted by the feature extraction network in the teacher model and all student features extracted by the feature extraction network in the incrementally trained student model to guide the update of the classification model (student model).

[0070] According to one embodiment of the present application, the total loss is calculated according to the following rules:

[0071]

[0072] wherein, denotes the total loss, denotes the new class classification loss, denotes the old class classification loss, and λ denotes the first hyperparameter, preferably λ = 0.03, denotes the first feature distillation loss, β denotes the second hyperparameter, preferably β = 0.6, denotes the second feature distillation loss, and γ denotes the third hyperparameter, preferably γ = 0.5. It should be noted that b in formula (4) denotes the bth incremental training, and the total loss of the bth incremental training is used to represent the calculation process of the total loss. In addition, in formula (4), the values of the hyperparameters after mathematical transformation can also be used to better guide the training of the classification model. For example, wherein, λ', β' and γ' represent values after mathematical transformation of the hyperparameters.

[0073] To better illustrate the embodiments of the present application, the following will continue to combine the attached Figure 1 Taking the b-th incremental training as an example, the specific obtaining process of the new class classification loss, the old class classification loss, the first feature distillation loss and the second feature distillation loss is explained respectively.

[0074] As Figure 1 shown, in step b-1, that is, the b-1-th incremental training, the data used for training the teacher model includes the corresponding fundus image samples obtained from the original data and the fundus categories corresponding to the fundus image samples, and the fundus categories used for training the teacher model are called old classes, for example Figure 1 old class 1 and old class 2 shown in the above table; pseudo sample 1 represents data after enhancement of the class mean feature vector corresponding to the old class 1; pseudo sample 2 represents data after enhancement of the class mean feature vector corresponding to the old class 2. In step b, that is, the b-th incremental training, the data used for training the student model includes the corresponding fundus image samples obtained from the original data and the fundus categories corresponding to the fundus image samples, and the fundus categories used for training the student model are called new classes, for example Figure 1 new class 1 and new class 2 shown in the above table, preferably, the new class 1 and the new class 2 are different from the old class 1 and the old class 2. In the embodiments of the present application, in the b-th incremental training, the data used for training the student model also uses the pseudo samples of the old classes, for example, the pseudo sample 1 and the pseudo sample 2 to alleviate the problem of catastrophic forgetting in the incremental process. Wherein, the second feature distillation is determined according to the total difference between all the teacher features extracted by the feature extraction network in the teacher model and all the student features extracted by the feature extraction network in the student model after incremental training; the first feature distillation loss is determined according to the difference between the outputs of the pseudo samples of the old classes on the following two: the output of the fully connected layer of the teacher model, the output of the fully connected layer of the student model on the old classes; the classification loss is generated by using the classification probability of the generated pseudo samples of the old classes and the fundus images belonging to the new classes after processing by the student model, and the classification loss includes the old class classification loss and the new class classification loss.

[0075] According to an embodiment of the present application, the old class classification loss is calculated by the following rule:

[0076]

[0077] wherein, F i represents the features extracted by the feature extraction network of the student model from the old class set C i (that is, the set composed of the pseudo samples of the i-th old class), represents the cross-entropy loss, and φ bparameters of the classifier of the student model, Y i represents the label corresponding to the pseudo sample of the i-th old class. Intuitively, in the feature space, the class mean feature vector of the old class is soft variance enhanced, which represents the confidence of the authenticity of the generated features. For the training of the current task (the b-th incremental training), the features extracted from the pseudo samples are fed into the classifier to maintain the distinction and balance between all the classes that have been learned so far.

[0078] According to an embodiment of the present application, the new class classification loss is calculated by the following rule:

[0079]

[0080] wherein F(X b ; θ b ) represents the features extracted by the feature extraction network of the student model for the new class set C b (that is, the set composed of the fundus image samples of the new classes), represents the cross-entropy loss, φ b represents the parameters of the classifier of the student model, Y b represents the label corresponding to the fundus image sample of the new class. For the training of the current task (the b-th incremental training), the features extracted from the new class corresponding fundus image samples are fed into the classifier to obtain the distinction and balance between all the new classes.

[0081] According to an embodiment of the present application, the first feature distillation loss is calculated by the following rule:

[0082]

[0083] wherein, represents the pseudo sample data used in the incremental training, represents the sample features extracted by the feature extraction network in the student model for the pseudo sample used in the incremental training, represents the parameters of the full connection layer in the classifier of the student model in the incremental training, G b represents the output of the corresponding full connection layer on the old class in the processing of the classifier of the student model, represents the sample features extracted by the feature extraction network in the teacher model for the pseudo sample data used in the incremental training, represents the parameters of the full connection layer in the classifier of the teacher model in the incremental training, represents the output of the full connection layer in the processing of the teacher model, represents the sum of the difference values between the corresponding elements between the two, It is to be noted that the feature extraction network in the student model is updated gradually when learning a new class (new class). In order to alleviate the mismatch between the saved pseudo samples and the feature extraction network, a logic-based knowledge distillation (first feature distillation loss) is also used to regularize the feature extraction network in the student model; the logit output by the fully connected layer is used to calculate the first distillation loss, which can make full use of the logit to guide the learning of the classification model, so that the learned classification model has more information.

[0084] The second feature distillation loss in the embodiment of the present application is determined by attention-based feature distillation. According to an embodiment of the present application, the total difference is calculated by the following steps: based on the attention mechanism, the attention value of each teacher feature to each student feature is determined, and the attention value of the teacher feature to each student feature is used to form an attention vector of the teacher feature to all student features and is normalized to obtain a normalized attention vector; based on the determined spatial distance of each teacher feature and each student feature; calculate the product of the corresponding elements in the spatial distance and the normalized attention vector, and sum all the products to obtain the total difference.

[0085] According to an embodiment of the present application, based on the attention mechanism, the attention value of each teacher feature to each student feature is calculated by the following steps: data conversion is performed on the student feature to obtain a key of the student feature in the attention mechanism; data conversion is performed on the teacher feature to obtain a query of the teacher feature in the attention mechanism; and the attention value of each query to each key is calculated.

[0086] The following explains the calculation process of the second feature distillation loss in the form of a formula.

[0087] For the training of the current step b (bth incremental training), the embodiment of the present application uses the current model as the student model and uses the step b-1 model as the teacher model. Let be a candidate feature set from the teacher model (the candidate feature set from the teacher model is a set composed of image features extracted by the feature extraction network in the teacher model from all pseudo samples), is a set of candidate features from students (the set of candidate features from students is a set of image features extracted by the feature extraction network in the student model for all pseudo samples and image features extracted by the feature extraction network in the student model for the fundus images belonging to the new class), where T and S represent the number of image features extracted from the teacher model and the student model, respectively. Each candidate feature (candidate feature is also referred to as image feature) is mapped to a size of and a channel size of where H, W and d represent height, width and channel size, respectively. When given two sets of candidates, AFD (Attention Meta Network) aims to identify the similarity of all possible combinations (T x S pairs) and transfer the knowledge of the teacher model candidates to the student model with the identified similarity. To identify the similarity between and , AFD uses the query-key concept in the attention mechanism. As shown in Figure 1 , each teacher feature generates a Query (as shown in Figure 1 , multiple Queries are represented as Queries), q t , and each student feature is a key (as shown in Figure 1 , multiple keys are represented as keys), k s . According to an embodiment of the present application, data conversion is performed by the following rules:

[0088]

[0089]

[0090] where q t represents query in the attention mechanism, represents the tth teacher feature, P HW (·) represents global average pooling, represents linear transformation parameters of, represents linear transformation parameter space matrix of, Q (·) represents the first activation function, k s represents key in the attention mechanism, represents the s student feature, represents linear transformation parameters of, represents linear transformation parameter space matrix of, represents linear transformation parameter space matrix of, f K(·) represents the second activation function d represents the dimension of the linear transformation parameter space matrix. It is noted that the transition weights of these features (e.g., W t Q , W s K ) are different because they have different properties through different levels, i.e., low-level visual features can represent a line and high-level visual features can represent an object. Therefore, the embodiment of the present application uses different transformation weights for each feature ( or ).

[0091] By using queries and keys, the attention values representing the relationship between the teacher and student representations can be normalized using the "softmax" function. According to an embodiment of the present application, the attention vector of each query to all keys is calculated and normalized by the following rules:

[0092]

[0093] where softmax(.) represents the normalization function, represents the transpose of q t , represents the bilinear weight, k t，1 represents the corresponding key value for , represents the position encoding of the tth teacher feature, represents the position encoding of the st student feature, k t，S represents the corresponding key value for , represents the product of and , represents the transpose of . In the attention mechanism, since the query and the key are identified from different dimensional features, the bilinear weight is used to summarize the attention values from different source levels (in other words, since q and k dimensions are not the same, they cannot be directly multiplied, and the two dimensions of the bilinear weight are the same as q and k, so q and k can be multiplied). Position encoding is used to share common information across different instances. The attention vector a t captures the relationship between the tth teacher feature and the entire student feature. By utilizing a t , the teacher feature can selectively transfer its knowledge to the student feature.

[0094] According to one embodiment of the present application, the second feature distillation loss is calculated by the following method:

[0095]

[0096] wherein, α t，s denotes the normalized attention value of the t-th teacher feature to the s-th student feature, denotes the spatial distance, ||.||2 denotes the L2 norm, denotes the combination function of the channel average pooling layer and the L2 normalization v / ||v||2, v denotes the vector obtained by performing the average pooling on , denotes the vector obtained by performing the average pooling on , the features obtained by using upsampling or downsampling, wherein the upsampling or downsampling is used to match the feature mapping size and the size of the teacher feature.

[0097] In summary, the total loss determined by the old class classification loss, the new class classification loss, the first feature distillation loss and the second feature distillation loss can guide the training of the classification model. On the one hand, the pseudo sample set is constructed by using the pseudo sample enhancement method to preserve the key features of the old class, and on the other hand, the key feature knowledge (it should be noted that the key feature knowledge is a more critical part of the feature knowledge) obtained by the student model from the teacher model in the incremental process is strengthened by using the feature distillation based on the attention mechanism, so that the forgetting effect of the classification model in the incremental learning process is smaller, and the accuracy is obviously improved. In other words, by alleviating the forgetting effect of the model in the incremental process, the accuracy of the model can be improved. The classification model (i.e. the student model trained by the incremental training) trained by the embodiment of the present application is a high-precision and strong-robustness classification model which can learn new knowledge while alleviating the catastrophic forgetting, and has a good classification effect on the new knowledge. In addition, it is of great significance and application value to make the deep learning model extract features with key attributes and control the distillation strength between feature pairs by using similar features between the teacher and student models. The embodiment of the present application also provides an eye fundus image classification method, which comprises the following steps: S1, obtaining an eye fundus image to be classified; S2, using the student model trained by the incremental training obtained by the training method of the classification model provided by the embodiment of the present application to classify the eye fundus image obtained in step S1, wherein the feature extraction network of the student model is used to extract image features of the eye fundus image to be classified according to the eye fundus image to be classified, and the classifier of the student model is used to identify the eye fundus category to which the eye fundus image to be classified belongs according to the image features. The image classification result includes: no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy, proliferative diabetic retinopathy. The eye fundus image classification method provided by the embodiment of the present application can identify the specific category and the generalization category to which the eye fundus image to be classified belongs, and has good accuracy.

[0098] III. Experimental verification

[0099] In order to better illustrate the technical effects of the embodiments of the present application, the following comparative experiments are used for verification. Among them, the experimental data set uses the data set provided by the Aier Hospital and the EyePACS data set; the classification model uses CNN+classifier (full connection layer+Softmax layer); the training methods for comparison include LUCIR, LwF, WA, MAS, DER, MUC, CwD, Our (representing the method provided by the present application), CODA, UpperBound; the index for evaluating the classification model is Accuracy (accuracy). Among them, LUCIR, LwF, WA, MAS, DER, MUC, CwD, CODA are the abbreviations of the corresponding training methods for comparison in the paper method. The specific full name of the paper method and the corresponding relationship of the abbreviation are as follows:

[0100] LwF: Learning without forgetting

[0101] MAS: Memory aware synapses: Learning what (not) to forget

[0102] MUC: More classifiers, less forgetting: A generic multi-classifier paradigm for incremental learning

[0103] CODA: CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

[0104] LUCIR: Learning a unified classifier incrementally via rebalancing

[0105] WA: Maintaining discrimination and fairness in class incremental learning

[0106] DER: Der: Dynamically expandable representation for class incremental learning.

[0107] CwD: Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning.

[0108] In addition, UpperBound represents the upper bound of the incremental learning experiment, which is the classification result of the entire data set of the data set after being divided into training set and test set.

[0109] It should be noted that in this experiment, the data set provided by the Aier Hospital and the EyePACS data set are trained according to the following experimental settings respectively:

[0110] Any 2 classes (e.g., no DR, mild DR) in the data set are combined and a classification model is pre-trained from the 2 classes, and then the remaining 3 classes (e.g., moderate DR, severe DR, proliferative DR) are divided into three incremental training.

[0111] The evaluation results of the EyePACS data set after being set according to the experimental setting and being trained by using the above-mentioned comparative training methods are as shown in the following table. Figure 2 The Accuracy represents the accuracy, the Number of Classes represents the number of classes, and 2_3steps represents the experimental results according to the experimental setting.

[0112] The evaluation results of the EyePACS data set after being set according to the experimental setting and being trained by using the above-mentioned comparative training methods are as shown in the following table. Figure 3 The Accuracy represents the accuracy, the Number of Classes represents the number of classes, and 2_3steps represents the experimental results according to the experimental setting.

[0113] As shown in the following table, Figure 2 and Figure 3 It can be seen that the method provided by the present application (represented by Our) is obviously better than the non-sample-based method, and is greatly better than most sample-based methods in terms of classification accuracy curve trend and average incremental accuracy, which proves that the method provided by the present application can effectively solve the catastrophic forgetting problem in CIL without storing old training samples, and better stability-plasticity balance is achieved. In particular, the experimental results on the EyePACS data set and the EyePACS data set show that the overall performance of the total of 5 classes is improved by 4.27% at the end of incremental learning, and the average accuracy is improved by 3.33%, so the method provided by the present application is always better than the non-sample-based SOTA method CODA.

[0114] The present application provides a training method of a classification model and an eye fundus image classification method aiming at the catastrophic forgetting problem caused by class differences during the incremental acquisition process of eye fundus disease images. The method belongs to a class-incremental fine-grained eye fundus image recognition method based on attention feature distillation. The method first learns the relative similarity between teacher and student features by constructing an attention-based neural network model, applies the recognized similarity to control the distillation strength between all feature pairs, and maintains the decision boundary of the previous class by saving a pseudo-sample set and representing the class in the representation space for each old class, thereby alleviating the memory or privacy problem restrictions on data during the incremental process. The class-incremental fine-grained eye fundus image recognition method based on attention feature distillation has obvious improvement in accuracy and robustness compared with the existing most advanced method during the incremental process.

[0115] It is to be understood that even though various embodiments can be described in conjunction with a specific order, the order is not absolute and the steps can be performed in other orders.

[0116] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0117] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se.

[0118] Embodiments of the application have been described above with the aid of example embodiments described above. The description above is intended to be illustrative, and not exhaustive, of the embodiments disclosed. Many modifications and variations of the described embodiments are possible and are within the scope of the described embodiments, as those skilled in the relevant art will recognize. The embodiments were chosen and described in order to best explain the principles of the application and its practical application, and to thereby enable others skilled in the art to best utilize the application and various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A method for training a classification model for fundus image classification, the method comprising: obtaining a pre-trained classification model as a teacher model, the teacher model comprising a feature extraction network for extracting image features from an input fundus image and a classifier for identifying a fundus class to which the image features correspond based on the image features, the classifier comprising a fully connected layer and a Softmax layer, the fundus classes identifiable by the teacher model being classified as old classes; obtaining a student model, the student model comprising a feature extraction network and a classifier, the feature extraction network of the student model being initialized with the feature extraction network of the teacher model, and the classifier of the student model being configured to identify fundus classes corresponding to old classes and new classes, the new classes being fundus classes other than the old classes; for each old class used in the pre-training, obtaining a pseudo sample corresponding to the old class, the pseudo sample being generated by using the feature extraction network of the teacher model to extract image features from a plurality of fundus images belonging to the old class, the pseudo sample of the old class being obtained according to the following steps: T1, calculating a class mean vector corresponding to the old class based on the plurality of image features corresponding to the old class used in the pre-training, T2, randomly sampling at least one noise vector from a Gaussian distribution, the noise vector having the same dimension as the class mean vector, and T3, calculating the pseudo sample of the old class according to a pre-defined augmentation scale, randomly sampling a noise vector from a Gaussian distribution, and the class mean vector corresponding to the old class according to a pre-set augmentation rule, the class mean vector corresponding to the old class being calculated according to the following rule: incrementally training the student model a plurality of times using the generated pseudo sample of the old class and fundus images belonging to the new classes, the parameters of the student model being updated based on a total loss determined according to a pre-set total loss function during the training, to obtain an incrementally trained student model, the total loss being calculated according to the following rule: the total difference being calculated according to the following steps: determining an attention value of each teacher feature with respect to each student feature based on an attention mechanism, and using the attention value of the teacher feature with respect to each student feature to form an attention vector of the teacher feature with respect to all student features and to obtain a normalized attention vector by normalizing the attention vector; determining a spatial distance between each teacher feature and each student feature; calculating a product of corresponding elements in the spatial distance and the normalized attention vector, and summing all the products to obtain the total difference. The attention value of each teacher feature with respect to each student feature is calculated based on an attention mechanism according to the following steps: data converting the student feature to obtain a key of the student feature in the attention mechanism; data converting the teacher feature to obtain a query of the teacher feature in the attention mechanism; and calculating an attention value of each query with respect to each key. The data conversion is performed according to the following rule: the attention vector of each query with respect to all keys is calculated and normalized according to the following rule: the second feature distillation loss is calculated according to the following method: the pre-set augmentation rule is: the pre-defined augmentation scale is set according to the following rule: the first feature distillation loss is calculated according to the following rule: ​ ​ ​ ​ wherein, represents an old class a corresponding class mean vector, represents an old class a number of samples, represents an old class a sample feature of a sample in the old class represents a parameter of a feature extraction network of the teacher model in the incremental training. ​ wherein, denotes the total loss, denotes the new class classification loss, denotes the old class classification loss, denotes a first hyper-parameter, denotes a first feature distillation loss determined according to a difference between outputs of the teacher model’s fully connected layer and the student model’s fully connected layer on the old class, the teacher model being the student model after the last incremental training of the current incremental training; denotes a second hyper-parameter, denotes a second feature distillation loss determined according to a total difference between all teacher features extracted by the feature extraction network in the teacher model and all student features extracted by the feature extraction network in the incrementally trained student model, wherein each teacher feature is an image feature of the pseudo sample extracted by the feature extraction network in the teacher model, and each student feature is an image feature of the pseudo sample and the fundus image belonging to the new class respectively extracted by the feature extraction network in the student model, denotes a third hyper-parameter.

2. The method of claim 1, wherein, ​ ​ ​ ​ 3. The method of claim 2, wherein, ​ ​ ​ ​ 4. The method of claim 3, wherein, ​ in, express In the attention mechanism, query Indicates the first Individual teacher characteristics Indicates global average pooling. express The linear transformation parameters, express The linear transformation parameter space matrix, ( ) represents the first activation function. express The key in the attention mechanism Indicates the first Student characteristics express The linear transformation parameters, express The linear transformation parameter space matrix, express The space matrix of the linear transformation parameters, This represents the second activation function. This represents the dimension of the linear transformation parameter space matrix.

5. The method of claim 4, wherein, ​ wherein, denotes a normalization function, denotes the transpose of denotes a bilinear weight, denotes to a corresponding key value, denotes the position encoding of the teacher feature, denotes the position encoding of the student feature, , , denotes to a corresponding key value, denotes and the product of denotes the transpose of 6. The method of claim 5, wherein, ​ wherein, denotes the normalized attention value of the th teacher feature to the th student feature, denotes the spatial distance, denotes the L2 norm, denotes the combination function of channel average pooling layer and L2 normalization , denotes the vector obtained by average pooling on , denotes the feature obtained by using up-sampling or down-sampling on .

7. The method of claim 1, wherein, ​ wherein, represents the old classes used in the b-th incremental training corresponding pseudo samples, represents the b-th incremental training predefined augmentation scale in the b-th incremental training, represents a random noise vector sampled from a Gaussian distribution, and have the same dimension.

8. The method of claim 7, wherein, ​ wherein, and denote the number of old and new classes in the th incremental training, respectively, denote the number of old and new classes in the th incremental training, respectively, denote the predefined augmentation scale in the th incremental training, denote the covariance matrix of the th new class in the th incremental training, denote the rank of , denote the number of classes in the th pre-training.​ 9. The method of claim 8, wherein, ​ wherein, denotes the pseudo sample data used in the incremental training, denotes the sample feature extracted by the feature extraction network in the student model from the pseudo sample used in the incremental training, denotes the parameters of the fully connected layer in the classifier of the student model in the incremental training, denotes the output of the corresponding fully connected layer in the classifier of the student model on the old classes, denotes the sample feature extracted by the feature extraction network in the teacher model from the pseudo sample used in the incremental training, denotes the parameters of the fully connected layer in the classifier of the teacher model in the incremental training, denotes the output of the fully connected layer when processed by the teacher model, denotes the sum of the difference values between the corresponding elements, denotes the L2 norm.

10. The method according to any one of claims 1 to 9, characterized in that, The old classes include any two of the following categories, and the remaining categories are new classes: no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy, proliferative diabetic retinopathy.

11. An eye fundus image classification method characterized by, The method comprises: S1, obtaining an eye fundus image to be classified; S2, using the incrementally trained student model obtained by the method of any one of claims 1-10 to classify the eye fundus image to be classified obtained in step S1, wherein the feature extraction network of the student model is used to extract image features of the eye fundus image to be classified according to the eye fundus image to be classified, and the classifier of the student model is used to identify the eye fundus category to which the eye fundus image to be classified belongs according to the image features.

12. The method of claim 11, wherein, The results of image classification include: no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy, proliferative diabetic retinopathy.

13. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method of any one of claims 1 to 12.

14. An electronic device, comprising: Comprise: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the steps of the method of any one of claims 1 to 12.

Images