Fish disease category increment method based on double distillation

Abstract

The invention relates to the technical field of image recognition, in particular to a fish disease category increment method based on double distillation, which comprises the following steps: S1, performing data enhancement on new category data and part of old category data in a category increment learning model; s2, constructing a teacher model and a student model, respectively inputting the data stream after data enhancement into the teacher model and the student model to extract a feature matrix, and obtaining sample information by adopting a characterization deviation distillation module; s3, a similar pair distillation module is adopted to ensure the consistency of the characterization features of the teacher model and the characterization features of the student model, and catastrophic forgetting caused by excessive correction is prevented; s4, reversely training the teacher model by adopting a random momentum updating method and utilizing the parameters trained by the student model, and updating the teacher model; the problem of disastrous forgetting caused by excessive correction is prevented by controlling the consistency of the representation similarity between the teacher model and the student model.

Description

Technical Field

[0001] The invention relates to the technical field of image recognition, and in particular to a fish disease category increment method based on double distillation. Background Art

[0002] Deep neural networks (DNNs) have performed well in many machine learning classification tasks and have shown human-level performance in fish disease diagnosis applications. Studies have shown that deep learning-based models have been successfully applied to fish disease screening. However, this is only possible when all fish disease categories are known. This assumption is often violated in the process of fish disease diagnosis because it is extremely difficult to prepare data for all fish disease categories in advance. For example, the severity of some fish diseases will change slightly at different stages of onset. If a model only learns to diagnose the generalized category of the disease, but cannot accurately identify the specific category of different stages and degrees of onset, it will have extremely serious consequences, because the treatment strategies used for the same disease at different stages of onset are different, and starting the correct treatment plan is the key to achieving good treatment results and improving production capacity. Therefore, it is necessary to construct a fine-grained, high-precision fish disease category increment method to timely and accurately identify the severity and stage of the disease, which can be of great help in improving the production capacity of aquatic products.

[0003] In class incremental learning (CIL), an ideal intelligent system is expected to be able to learn new categories from streaming data without forgetting the old categories they have already mastered. However, when new data is available, all (or most) old data cannot be seen again in CIL. In this case, deep neural networks suffer from severe catastrophic forgetting, which indicates that deep neural networks have difficulty recalling old knowledge after learning new knowledge. To deal with catastrophic forgetting, researchers have proposed different techniques, which can be divided into three categories: structure-based, preview-based, and knowledge distillation regularization-based. Semantic recognition methods based on knowledge distillation (KD) regularization have achieved remarkable performance in semantic recognition and have been widely used. Specifically, knowledge distillation-based methods usually utilize the old model as a teacher model and help the new model mitigate catastrophic forgetting by distilling its feedback. However, the misrepresentation information in the teacher model is also inherently present, resulting in representation bias, which limits the positive effect of knowledge distillation and leads to catastrophic forgetting. Such representation bias is caused by the data imbalance between the old and new classes, because the old class data stored in the buffer is small, while a large amount of new class data is input in the incremental step. This situation causes the weight of the classifier (i.e. the last fully connected layer) to tend to point the old class data to the new class of the current step.

[0004] Among the existing patents, there is no invention that alleviates the problem of catastrophic forgetting by solving the inherent representation bias accumulated by the model during the incremental learning process. For example, patent CN1 13066025A proposes an image dehazing method based on incremental learning and feature and attention transfer, but this method does not solve the problem of catastrophic forgetting in the incremental process very well; patent CN106022368A designs an incremental trajectory anomaly detection method based on incremental kernel principal component analysis, and uses kernel principal component analysis to realize incremental anomaly detection; patent CN1 12990280-A adopts the category incremental method for image classification problems, but does not pay further attention to the generalization problem in the category incremental process; patent US2020302230-A applies category incremental learning to the field of target detection. Summary of the invention

[0005] In order to solve the above limitations, the present invention proposes a double distillation network (D2-Net), which includes a representation deviation distillation module (RBD-Module) and a similarity pair distillation module (PSD-Module). The representation deviation distillation module (RBD-Module) is designed to align those representations with less bias between the teacher model and the student model. Specifically, the teacher model is trained by accumulating the parameters of the student model because the student model at each time step is good at classifying different categories. The similarity pair distillation module (PSD-Module) prevents over-correction by controlling the similarity consistency between the teacher model representation and the student model representation, thereby making the model more robust. Through the combination of the representation deviation distillation module (RBD-Module) and the similarity pair distillation module (PSD-Module), the teacher model can allow the student model to learn richer information to preserve the learned knowledge, and the student model can help the teacher model reduce forgetting while accumulating new knowledge.

[0006] A fish disease category increment method based on double distillation, the method comprising the following steps:

[0007] Step S1: Perform data enhancement on new category data and part of old category data in the category incremental learning model (CIL);

[0008] Step S2: Constructing the teacher model and student models The data stream after data enhancement is input into the teacher model respectively and student models The feature matrix is ​​extracted from the dataset and the representation deviation distillation module (RBD-Module) is used to obtain sample information.

[0009] Step S3: Use the Similarity Pair Distillation Module (PSD-Module) to ensure that the teacher model Characterization features and student models The consistency of representational features prevents catastrophic forgetting caused by overcorrection;

[0010] Step S4: Adopting the random momentum update method and using the student model The trained parameters are used to reversely train the teacher model Model for teachers to update.

[0011] Further, step S1: performing data enhancement on new category data and part of old category data in the category incremental learning model (CIL) includes:

[0012] Step S1.1: The Category Incremental Learning model (CIL) is trained at time step t by using labeled examples To train, introduce a replay buffer Used to store a small amount of old category data and label samples Divided into multiple data streams X b , each data stream X b The batch size after inputting the model is n = |X b | and with the replay buffer The m samples randomly selected from the sum form a superposition batch data And the superimposed batch data As input data for the Category Incremental Learning model (CIL);

[0013] Step S1.2: CIL model superimposes batch data Perform double data enhancement to obtain enhanced batch data x k The samples obtained by double random enhancement are expressed as

[0014] Furthermore, step S2: constructing the teacher model and student models The data stream after data enhancement is input into the teacher model respectively and student models The feature matrix is ​​extracted from the dataset, and the representation deviation distillation module (RBD-Module) is used to obtain the sample information including:

[0015] Step S2.1: Augment the batch data As input data, they are input into the teacher model and student models Characterization features and characterization features Then characterize the features and characterization features The features are respectively input into the softmax layer for feature matrix extraction, and the extracted feature matrix is ​​directly subjected to knowledge distillation to obtain more basic and profound sample information; the student model It is trained by characterizing the bias distillation loss, which is formulated as:

[0016]

[0017] in, represents the characterization bias distillation loss; Indicates that the input data is in the student model The characterization features obtained above; Represents the input data in the teacher model The representation features obtained above; ||·||2 represents l2 regularization.

[0018] Further, step S3: the similarity pair distillation module (PSD-Module) is used to ensure that the teacher model Characterization features and student models The characteristics that characterize consistency include:

[0019] Step S3.1: The enhanced batch data As student models Input, teacher model The input is used to obtain the student representation feature matrix and the teacher representation feature matrix The t-SNE algorithm is used to characterize the student feature matrix and the teacher representation feature matrix For dimensionality reduction, the symmetric SNE similarity captures the global similarity distribution between representation features without using supervision or prior knowledge, and visualizes high-dimensional data by approximating similar probability distribution. The formula is:

[0020]

[0021]

[0022] Where α is the temperature hyperparameter; and α>0; p ij , The student model and the frozen teacher model Mapping characteristics The pairwise similarity of i|j It means to find the similarity on i under the condition of j, p j|i It means to find the similarity on j under the condition of i, and exp is an exponential function with the natural constant e as the base.

[0023] Step S3.2: The distillation loss of similar pairs is obtained by pairwise similarity:

[0024]

[0025] Among them, p ij , The student model and the frozen teacher model Mapping characteristics The pairwise similarity of .

[0026] Step S3.3: In order to form a valid distribution, the sum of the given pairwise similarities is 1, that is, The same rule applies to Right now In the Similarity Pair Distillation Module (PSD-Module), the learned knowledge is stored by pairwise similarities. Therefore, penalizing the KL divergence of pairwise similarity distributions of past models can prevent catastrophic updates.

[0027] Further, step S4: the random momentum update method is used and the student model is used The trained parameters are used to reversely train the teacher model Model for teachers To update, the formula is:

[0028] θ T ←mθ T +(1-m)[(1-X)θ T +Xθ S ];

[0029] Where m is the momentum coefficient, and X follows the Bernoulli distribution, which can be expressed as:

[0030] P(X=k)=p k (1-p) 1-k , k = {0, 1};

[0031] The Bernoulli probability p is in the range of (0, 1). By doing so, we avoid adjusting the momentum coefficient m, which is particularly sensitive. In addition, in order to help the teacher model better accumulate new knowledge, the value of the momentum coefficient m is gradually increased in the early stage:

[0032] m = min (n / (n+1), η);

[0033] Where n is the number of model iterations, and the constant η is set to 0.999.

[0034] Furthermore, the total loss function has two terms: on the one hand, a novel representation bias distillation module (RBD-Module) is used to correct the representation bias in the teacher model during the knowledge distillation process, and on the other hand, a pairwise similarity distillation module (PSD-Module) is used to prevent over-correction:

[0035]

[0036] The classification loss The calculation formula is:

[0037]

[0038] Among them: β and λ are hyperparameters, and λ is used to balance the three losses.

[0039] The structural design of the present invention is reasonable and reliable. On the one hand, it proposes representation bias distillation (RBD) to guide the training process of the teacher model by exploring the advantages of the student model to correct the representation bias of the teacher model; on the other hand, the method proposes pairwise similarity distillation (PSD) to prevent catastrophic forgetting caused by excessive correction by controlling the consistency of the representation similarity between the teacher model and the student model. The fish disease category increment method based on double distillation has a significant improvement in accuracy and robustness in the increment process compared with the existing most advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural schematic diagram of the present invention.

[0041] Figure 2 It is a schematic diagram of the experimental effect on Tiny-ImageNet as the incremental steps change in the embodiment.

[0042] Figure 3 Schematic diagram of the average experimental effect of all incremental steps on CIFAR-100 and Tiny-ImageNet in the embodiment. DETAILED DESCRIPTION

[0043] Aiming at the catastrophic forgetting problem caused by the accumulation of the inherent representation bias of the old model in the new model during the incremental acquisition of fish disease images, the present invention proposes a fish disease category incremental method based on double distillation. On the one hand, the method proposes representation bias distillation (RBD), which guides the training process of the teacher model by exploring the advantages of the student model to correct the representation bias of the teacher model; on the other hand, the method proposes pairwise similarity distillation (PSD), which prevents the catastrophic forgetting problem caused by excessive correction by controlling the consistency of the representation similarity between the teacher model and the student model.

[0044] A fish disease category increment method based on double distillation, the method comprising the following steps:

[0045] Step S1: Performing data enhancement on new category data and part of old category data in the category incremental learning model (CIL), including the following steps:

[0046] Step S1.1: The Category Incremental Learning model (CIL) is trained at time step t by using labeled examples To train, introduce a replay buffer Used to store a small amount of old category data and label samples Divided into multiple data streams X b , each data stream X b The batch size after inputting the model is n = |X b | and with the replay buffer The m samples randomly selected from the sum form a superposition batch data And the superimposed batch data As input data for the Category Incremental Learning model (CIL);

[0047] Step S1.2: CIL model superimposes batch data Perform double data enhancement to obtain enhanced batch data x k The samples obtained by double random enhancement are expressed as

[0048] Step S2: Constructing the teacher model and student models The data stream after data enhancement is input into the teacher model respectively and student models The feature matrix is ​​extracted from the dataset, and the RBD-Module is used to obtain more basic and profound sample information to mine the student model. The advantages of the teacher model are extended to expand the positive role of knowledge distillation and promote the training process of the teacher model. The following steps are included:

[0049] Step S2.1: Augment the batch data As input data, they are input into the teacher model and student models Characterization features and characterization features Then characterize the features and characterization features The features are respectively input into the softmax layer for feature matrix extraction, and the extracted feature matrix is ​​directly subjected to knowledge distillation to obtain more basic and profound sample information; the student model It is trained by characterizing the bias distillation loss, which is formulated as:

[0050]

[0051] in, represents the characterization bias distillation loss; Indicates that the input data is in the student model The characterization features obtained above; Represents the input data in the teacher model The representation features obtained above; ||·||2 represents l2 regularization.

[0052] Step S3: Use the Similarity Pair Distillation Module (PSD-Module) to ensure that the teacher model Characterization features and student models The consistency of the characterization features includes the following steps:

[0053] Step S3.1: The enhanced batch data As student models Input, teacher model The input is used to obtain the student representation feature matrix and the teacher representation feature matrix The t-SNE algorithm is used to characterize the student feature matrix and the teacher representation feature matrix For dimensionality reduction, the symmetric SNE similarity captures the global similarity distribution between representation features without using supervision or prior knowledge, and visualizes high-dimensional data by approximating similar probability distribution. The formula is:

[0054]

[0055]

[0056] where α is the temperature hyperparameter; and α>0; pij, The student model and the frozen teacher model Mapping characteristics The pairwise similarity of i|j It means to find the similarity on i under the condition of j, p j|i It means to find the similarity on j under the condition of i, and exp is an exponential function with the natural constant e as the base.

[0057] Step S3.2: The distillation loss of similar pairs is obtained by pairwise similarity:

[0058]

[0059] Among them, p ij , The student model and the frozen teacher model Mapping characteristics The pairwise similarity of .

[0060] Step S3.3: In order to form a valid distribution, the sum of the given pairwise similarities is 1, that is, The same rule applies to Right now In the Similarity Pair Distillation Module (PSD-Module), the learned knowledge is stored by pairwise similarities. Therefore, penalizing the KL divergence of pairwise similarity distributions of past models can prevent catastrophic updates.

[0061] Step S4: Adopting the random momentum update method and using the student model The trained parameters are used to reversely train the teacher model Model for teachers To update, the formula is:

[0062] θ T ←mθ T +(1-m)[(1-X)θ T +Xθ S ];

[0063] Where m is the momentum coefficient, and X follows the Bernoulli distribution, which can be expressed as:

[0064] P(X=k)=p k (1-p) 1-k , k = {0, 1};

[0065] The Bernoulli probability p is in the range of (0, 1). By doing so, we avoid adjusting the momentum coefficient m, which is particularly sensitive. In addition, in order to help the teacher model better accumulate new knowledge, the value of the momentum coefficient m is gradually increased in the early stage:

[0066] m = min (n / (n+1), η);

[0067] Where n is the number of model iterations, and the constant η is set to 0.999.

[0068] The total loss function of this method has two items: on the one hand, a novel representation bias distillation module (RBD-Module) is used to correct the representation bias in the teacher model during the knowledge distillation process, and on the other hand, a pairwise similarity distillation module (PSD-Module) is used to prevent over-correction:

[0069]

[0070] The classification loss The calculation formula is:

[0071]

[0072] Among them: β and λ are hyperparameters, and the hyperparameters β and λ are used to balance the three losses. Figure 1 The overall training process of our proposed method is elaborated in detail in . When new data appears, we first use the storage sample strategy to update the buffer. Then, the student model is optimized according to the overall goal

[0073] Example

[0074] The effectiveness of this method is extensively evaluated on two commonly used public datasets, CIFAR-100 and Tiny-ImageNet. In the experiments of this invention, ResNet-18 is used and trained from scratch. The batch size is 128, and the Adam optimizer is used for 200 iterations of training optimization, where the learning rate is set to 0.1 and is reduced to 1 / 10 of the previous learning rate after 50, 100, and 150 epochs, respectively. All experiments are repeated three times, and the average results are reported. The training images are randomly flipped and cropped as input. We use a simple public grid search to find the optimal hyperparameters in Formula 7 and find that the best effect is when β and λ are set to 0.4 and 0.2, respectively. We use the public herd method to select and store 20 samples for each old class.

[0075] To verify the experimental effect, we report the average incremental accuracy and average forgetting rate, as Figure 2 and Figure 3 As shown in Figure 2, the average incremental accuracy fairly compares the overall incremental performance of different methods. The average forgetfulness directly measures the ability of different methods to resist catastrophic forgetting.

[0076] In the description of the present invention, it is necessary to understand that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A fish disease classification increment method based on double distillation, characterized in that: The method comprises the following steps: Step S1: Perform data enhancement on new category data and part of old category data in the category incremental learning model (CIL); Step S2: Constructing the teacher model and student models The data stream after data enhancement is input into the teacher model respectively and student models The feature matrix is ​​extracted from the dataset and the representation deviation distillation module (RBD-Module) is used to obtain sample information. Step S3: Use the Similarity Pair Distillation Module (PSD-Module) to ensure that the teacher model Characterization features and student models The consistency of representational features prevents catastrophic forgetting caused by overcorrection; Step S4: Adopting the random momentum update method and using the student model The trained parameters are used to reversely train the teacher model Model for teachers to update.

2. A fish disease classification increment method based on double distillation according to claim 1, characterized in that: The data enhancement of new category data and part of old category data in the category incremental learning model (CIL) includes: Step S1.1: The Category Incremental Learning model (CIL) is trained at time step t by using labeled examples To train, introduce a replay buffer Used to store a small amount of old category data and label samples Divided into multiple data streams X b , each data stream X b The batch size after inputting the model is n = |X b | and with the replay buffer The m samples randomly selected from the sum form a superposition batch data And the superimposed batch data As input data for the Category Incremental Learning model (CIL); Step S1.2: CIL model superimposes batch data Perform double data enhancement to obtain enhanced batch data x k The samples obtained by double random enhancement are expressed as 3. A fish disease classification increment method based on double distillation according to claim 2, characterized in that: The teacher model is constructed and student models The data stream after data enhancement is input into the teacher model respectively and student models The feature matrix is ​​extracted from the dataset, and the representation deviation distillation module (RBD-Module) is used to obtain the sample information including: Step S2.1: Augment the batch data As input data, they are input into the teacher model and student models Characterization features and characterization features Then characterize the features and characterization features The features are respectively input into the softmax layer for feature matrix extraction, and the extracted feature matrix is ​​directly subjected to knowledge distillation to obtain more basic and profound sample information; the student model It is trained by characterizing the bias distillation loss, which is formulated as: in, represents the characterization bias distillation loss; Indicates that the input data is in the student model The characterization features obtained above; Represents the input data in the teacher model The characterization features obtained above; ||·| | 2 indicates l2 regularization.

4. The fish disease classification increment method based on double distillation according to claim 3, characterized in that: The similarity pair distillation module (PSD-Module) is used to ensure that the teacher model Characterization features and student models The characteristics that characterize consistency include: Step S3.1: The enhanced batch data As student models Input, teacher model The input is used to obtain the student representation feature matrix and the teacher representation feature matrix The t-SNE algorithm is used to characterize the student feature matrix and the teacher representation feature matrix For dimensionality reduction, the symmetric SNE similarity captures the global similarity distribution between representation features without using supervision or prior knowledge, and visualizes high-dimensional data by approximating similar probability distribution. The formula is: where α is the temperature hyperparameter; and α>0; pij, The student model and the frozen teacher model Mapping characteristics The pairwise similarity of ; pi|j means the similarity on i under the condition of j, pj|i means the similarity on j under the condition of i, and exp is an exponential function with the natural constant e as the base; Step S3.2: The distillation loss of similar pairs is obtained by pairwise similarity: Among them, pij, The student model and the frozen teacher model Mapping characteristics The pairwise similarity of Step S3.3: In order to form a valid distribution, the sum of the given pairwise similarities is 1, that is, The same rule applies to Right now In the Pair Similarity Distillation Module (PSD-Module), the learned knowledge is stored via pairwise similarities; thus, penalizing the KL divergence of pairwise similarity distributions of past models can prevent catastrophic updates.

5. The fish disease classification increment method based on double distillation according to claim 4, characterized in that: The random momentum update method is used and the student model is used The trained parameters are used to reversely train the teacher model Model for teachers To update, the formula is: i T ←mθ T +(1-m)[(1-X)θ T +Xθ s ]; Where m is the momentum coefficient, and X follows the Bernoulli distribution, which can be expressed as: P(X=k)=p k (1-p) 1-k ,k={0,1}; The Bernoulli probability p is in the range of (0, 1). By doing so, we avoid adjusting the momentum coefficient m, which is particularly sensitive. In addition, in order to help the teacher model better accumulate new knowledge, the value of the momentum coefficient m is gradually increased in the early stage: m = min (n / (n+1), η); Where n is the number of model iterations, and the constant η is set to 0.

999.

6. A fish disease classification increment method based on double distillation according to claim 5, characterized in that: The total loss function has two items: on the one hand, a novel representation bias distillation module (RBD-Module) is used to correct the representation bias in the teacher model during the knowledge distillation process, and on the other hand, a pairwise similarity distillation module (PSD-Module) is used to prevent over-correction: The classification loss The calculation formula is: Among them: β and λ are hyperparameters, and the hyperparameters β and λ are used to balance the three losses.

Images