Class increment image classification method and system based on knowledge distillation

Abstract

The invention relates to a class increment image classification method and system based on knowledge distillation. The method comprises the following steps: generating output representations of old class training samples among old classes through a teacher model, and respectively carrying out Z score standardization to obtain first standard output, generating old class training samples, inter-class output expressions of the new class training samples between new classes and old classes and intra-class output expressions of the old class training samples between the old classes through a student model, and respectively performing Z score standardization to obtain second standard output and third standard output; calculating inter-class distillation loss according to the first standard output and the second standard output, calculating intra-class distillation loss according to the first standard output and the third standard output, and obtaining total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss; and optimizing the total loss of the image classification model to obtain a trained image classification model, and carrying out image classification by adopting the trained image classification model. By adopting the method, the accuracy of image classification can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a class-incremental image classification method and system based on knowledge distillation. Background Art

[0002] In today's big data era, data is growing so fast that many traditional learning methods are unable to adapt. To address this challenge, incremental learning has emerged, which aims to enable models to learn from new data gradually instead of using data for training all at once. This approach is particularly important for application scenarios where data continues to grow or change, such as real-time data stream processing, audio recognition, and activity recognition in smart homes.

[0003] Class-Incremental Learning is a specific scenario of incremental learning, which involves an algorithm gradually learning to distinguish between a growing set of categories. In this scenario, the algorithm must learn to distinguish between classes encountered in different tasks or segments, and needs to identify which task or category a sample belongs to during the testing phase. The core challenge of class-incremental learning is to learn new categories while retaining the memory of old categories and avoiding catastrophic forgetting, that is, the model quickly loses the ability to learn previously when learning new tasks. The setting of class-incremental learning usually involves a series of classification tasks, each of which contains a set of non-overlapping categories. During the learning process, the model can only access the data of the current task, while at the test time it needs to classify the categories of all previously learned tasks. This requires the model to be able to learn and remember new categories without forgetting the old categories.

[0004] Although existing class-incremental learning techniques have shown great potential in various dynamic scenarios, research shows that even the most advanced methods are not effective enough in alleviating the challenge of catastrophic forgetting. In the class-incremental learning (CIL) scenario in image classification tasks, knowledge distillation is widely used to alleviate the phenomenon of the model forgetting old classes when learning new classes. Traditional knowledge distillation regularizes model parameters by optimizing knowledge distillation loss and cross entropy loss simultaneously. However, knowledge distillation requires forced matching of the logits output distribution of the student model and the teacher model on the old class. For example, if the true value label of the second class is 0, but the teacher's predicted probability is 0.9, then the optimization objective of the student model will conflict, making optimization difficult. This makes knowledge distillation unable to play a good role and leads to recency bias, and the model will tend to classify instances into new categories more and more. This problem is particularly prominent in incremental learning of image classification. This phenomenon not only greatly reduces the effect of knowledge distillation, but also aggravates the model's dependence on new categories, further deepens the forgetting of old categories, and ultimately leads to a decline in overall performance in image classification tasks, affecting the accuracy of image classification. Summary of the invention

[0005] Based on this, it is necessary to provide a class-incremental image classification method and system based on knowledge distillation to address the above technical problems.

[0006] A class-incremental image classification method based on knowledge distillation, the method comprising: Obtaining old class training samples, new class training samples and a pre-built image classification model; the image classification model includes a student model and a teacher model; Generate output representations of old class training samples between old classes by the teacher model and perform Z-score standardization respectively to obtain a first standard output; generate inter-class output representations of old class training samples and new class training samples between new classes and old classes and intra-class output representations of old class training samples between old classes by the student model and perform Z-score standardization respectively to obtain a second standard output and a third standard output; Calculate the inter-class distillation loss according to the first standard output and the second standard output, calculate the intra-class distillation loss according to the first standard output and the third standard output, and obtain the total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss; The total loss of the image classification model is optimized to obtain a trained image classification model, and the trained image classification model is used to perform image classification.

[0007] A class-incremental image classification system based on knowledge distillation, the system comprising: A model acquisition module, used to acquire old class training samples, new class training samples and a pre-built image classification model; the image classification model includes a student model and a teacher model; A sample input module is used to generate output representations of old class training samples between old classes through the teacher model and perform Z-score standardization respectively to obtain a first standard output, generate inter-class output representations of old class training samples and new class training samples between new classes and old classes and intra-class output representations of old class training samples between old classes through the student model and perform Z-score standardization respectively to obtain a second standard output and a third standard output; a loss construction module, used to calculate the inter-class distillation loss according to the first standard output and the second standard output, calculate the intra-class distillation loss according to the first standard output and the third standard output, and obtain the total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss; The image classification module is used to optimize the total loss of the image classification model to obtain a trained image classification model, and use the trained image classification model to perform image classification.

[0008] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: Obtaining old class training samples, new class training samples and a pre-built image classification model; the image classification model includes a student model and a teacher model; Generate output representations of old class training samples between old classes by the teacher model and perform Z-score standardization respectively to obtain a first standard output; generate inter-class output representations of old class training samples and new class training samples between new classes and old classes and intra-class output representations of old class training samples between old classes by the student model and perform Z-score standardization respectively to obtain a second standard output and a third standard output; Calculate the inter-class distillation loss according to the first standard output and the second standard output, calculate the intra-class distillation loss according to the first standard output and the third standard output, and obtain the total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss; The total loss of the image classification model is optimized to obtain a trained image classification model, and the trained image classification model is used to perform image classification.

[0009] The above-mentioned class-incremental image classification method and system based on knowledge distillation, by standardizing the Z score of the model output representation, makes the logits output distribution change of the student model unrestricted, retains the semantic relationship of the old class of the teacher model, effectively improves the learning ability of the new class, reduces the forgetting of old knowledge, and alleviates the weight drift caused by scale and shift changes, avoids the optimization difficulties of traditional exact matching, reduces the instantaneous forgetting phenomenon in task switching, calculates the inter-class distillation loss according to the first standard output and the second standard output, can capture the inter-class relationship and ensure fairness between the new and old classes, so that the student model retains the knowledge of the old class when learning the new class, alleviates the forgetting problem, calculates the intra-class distillation loss according to the first standard output and the third standard output, ensures fairness within the old class, strengthens the intra-class similarity, thereby improving the overall performance of the image classification model, and can achieve more accurate image classification. The embodiment of the present invention can improve the accuracy of image classification without additional computing costs in a resource-constrained environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an application scenario diagram of a class-incremental image classification method based on knowledge distillation in one embodiment; Figure 2 A schematic diagram of accuracy comparison in a traditional knowledge distillation method in one embodiment; Figure 3 Schematic diagram of a process of a class-incremental image classification method based on knowledge distillation in one embodiment; Figure 4 A schematic diagram showing a comparison of processing flows of precise matching and loose semantically invariant matching in one embodiment; Figure 5 is a pseudo code diagram of a training algorithm (Algorithm 2: Training Algorithm of Our Method) in one embodiment; Figure 6 is a structural block diagram of a class-incremental image classification system based on knowledge distillation in one embodiment; Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0012] In order to study the limitations of knowledge distillation, this paper conducts an empirical study on the stability of the model in traditional knowledge distillation methods through continuous evaluation of each iteration. Figure 2 The accuracy comparison diagram of the traditional knowledge distillation method is shown in FIG. It is a widely used method in the field of image classification with class increment, which combines knowledge distillation technology to avoid catastrophic forgetting problem. Its core idea is to use the output of the old network to guide the balance of network parameters for training new tasks, and is usually used as a baseline method to evaluate the advancedness of the method. and yes Two variants of The metric used to calculate the loss is cross entropy loss method, The LwF method using KL divergence is used to calculate the loss. It is the method of the present invention, that is, the class increment method based on inter-class and intra-class distillation losses. Figure 2 In , after completing the training of the current stage, it can be observed that there is a significant transient forgetting during the task switching process, followed by a recovery phase of accuracy. This phenomenon is called the stability gap. This paper analyzes this phenomenon from two aspects: temporary forgetting and partial recovery, using gradient-based analysis and empirical analysis. The results reveal two key issues: (1) If the model only focuses on old logits during the knowledge distillation process, it will lead to severe forgetting during the early transition; (2) The exact matching requirement in the knowledge distillation loss hinders the restatement of forgotten knowledge in the later transition. Therefore, it is difficult for knowledge-based methods to effectively learn from new tasks while reducing catastrophic forgetting.

[0013] In one embodiment, Figure 1 As shown, a class-incremental image classification method based on knowledge distillation is provided, comprising the following steps: Step 102, obtaining old class training samples, new class training samples and a pre-built image classification model.

[0014] Image classification models include student models and teacher models. Knowledge Distillation (KD) is a model compression technology that improves the performance of a student model by transferring the knowledge from a large and complex teacher model to a small and simple student model. The core idea of ​​this technology is to use the soft labels of the teacher model to train the student model instead of the traditional hard labels. These soft labels contain the probability distribution information of the teacher model for the input, thereby helping the student model to learn knowledge better.

[0015] In an embodiment of the present invention, an incremental task with O old classes and N new classes is designed, and the goal is to train the model to perform image classification across all O + N classes, and both the old class training samples and the new class training samples include image samples and corresponding target labels. Image classification can be classification of remote sensing images, and the remote sensing image dataset contains pictures of geographical areas taken from satellites or drones, and the task is to classify different geographical categories. The old classes usually reflect existing natural resources and infrastructure types. These categories are already widely present and can be stably identified in remote sensing images, including forests, water bodies, cities, agricultural land, deserts, grasslands, mountains and wetlands. With the changes in human activities and the changes in natural resources, the newly added categories usually reflect urban expansion, environmental changes, and the impact of human activities on natural ecology. New classes include densely built areas, emerging agricultural land, transportation infrastructure, artificial lakes / reservoirs, industrial parks, wetland restoration areas and ecological protection areas.

[0016] Step 104, using the teacher model to generate output representations of old class training samples between old classes and perform Z-score standardization respectively to obtain a first standard output, and using the student model to generate inter-class output representations of old class training samples and new class training samples between new classes and old classes and intra-class output representations of old class training samples between old classes and perform Z-score standardization respectively to obtain a second standard output and a third standard output.

[0017] The inter-class output refers to the output distribution relationship between the new class and the old class, that is, the relative relationship between the new and old class logits. The intra-class output refers to the output distribution relationship between the old class and the old class. In the teacher model and the student model, the inter-class output is and The outputs within the class are and The second standard output is the standardized inter-class output representation of the new and old classes, focusing on the relationship between the new and old classes. The first and second standard outputs are used to ensure that the student model retains the knowledge of the teacher model in distinguishing between the new and old classes. The third standard output is the standardized intra-class output representation, focusing on the sample relationship within the old class. The first and third standard outputs are used to ensure that the student model retains the characteristics of the teacher model in the sample distribution within each old class.

[0018] like Figure 3 FIG. 1 is a flow chart of a class incremental image classification method based on knowledge distillation in a specific embodiment. Figure 3In Figure 1, we can see that the output of the teacher model is usually a probability distribution in a classification task. For example, for a classification problem with 3 categories, the output of the teacher model may be [0.7, 0.2, 0.1], which represents the probability that the teacher model believes that the input belongs to category 1, category 2, and category 3. This distribution usually provides more information than hard labels (such as [1, 0, 0]), especially for ambiguous samples. Figure 3 In , each row represents a different input instance, i.e., an image sample. In incremental learning, these instances are samples used to train the model to learn new classes, and each column represents the logits output of a class. The figure distinguishes between new classes and old classes. New classes are newly added categories learned by the student model, as shown in the "Student" matrix on the left. The logits output of the teacher model does not include new classes. Old classes are categories that the model has learned before incremental learning, and are located on the right side of the output of the student model and the teacher model. Each value represents the model's unnormalized prediction score (logits) that a specific instance belongs to a certain class. For example, in the "Student" section, 7.65 in the first row represents the student model's prediction score that the first instance belongs to a certain old class. Figure 3 It also demonstrates the use of Z-score standardization to process logits and calculate losses based on the processing results. It not only directly matches the old class output logits of the teacher and student models, but also distills the inter-class relationship and intra-class relationship separately to further optimize the performance of the student model on the old class. In this way, the recency bias problem that occurs in traditional knowledge distillation can be effectively alleviated, that is, the student model tends to predict new classes while ignoring the phenomenon of old classes.

[0019] like Figure 4The figure shows a comparison of the processing flow of exact matching and relaxed semantic invariant matching. In this example, the old class logits of student S1 are closer to the teacher's logits in terms of value range and variance, while student S2 retains the same semantic relationship with the teacher. In traditional knowledge distillation methods, this exact matching results in a much lower knowledge distillation loss for student S1 than for student S2. However, it results in a much higher cross entropy loss, which greatly reduces the old knowledge learned by the model. This method introduces Z-score normalization preprocessing to ensure that the knowledge distillation loss and cross entropy loss are no longer in conflict, allowing both to be optimized simultaneously to achieve balanced learning and anti-forgetting. Z-score normalization essentially ensures that the mean of the standardized logits is zero and the standard deviation is 1. The specific implementation method is to subtract the dataset mean from each data point and then divide it by the standard deviation of the dataset. These properties ensure that Z-score normalization is a monotonic positive linear transformation that is invariant under separate changes in scale and shift, thereby retaining equal semantic information.

[0020] Step 106, calculating the inter-class distillation loss according to the first standard output and the second standard output, calculating the intra-class distillation loss according to the first standard output and the third standard output, and obtaining the total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss.

[0021] Step 108, optimizing the total loss of the image classification model to obtain a trained image classification model, and using the trained image classification model to perform image classification.

[0022] In the above-mentioned class-incremental image classification method based on knowledge distillation, by standardizing the Z score of the model output representation, the logits output distribution change of the student model is not restricted, the semantic relationship of the old class of the teacher model is retained, the learning ability of the new class is effectively improved, the forgetting of old knowledge is reduced, and the weight drift caused by scale and shift changes is alleviated, the optimization difficulty of traditional exact matching is avoided, and the instantaneous forgetting phenomenon in task switching is reduced. The inter-class distillation loss is calculated according to the first standard output and the second standard output, which can capture the inter-class relationship and ensure fairness between the new and old classes, so that the student model retains the knowledge of the old class when learning the new class, and the forgetting problem is alleviated. The intra-class distillation loss is calculated according to the first standard output and the third standard output, which ensures fairness within the old class and strengthens the intra-class similarity, thereby improving the overall performance of the image classification model and achieving more accurate image classification. The embodiment of the present invention can improve the accuracy of image classification without additional computing costs in a resource-constrained environment.

[0023] In one embodiment, the output representation of the old class training samples between old classes is generated by the teacher model and Z-score normalized respectively to obtain the first standard output, including: obtaining the logits value of each old class training sample of the current training batch by the teacher model; performing Z-score normalization on the logits values ​​of the same old class training samples output by the teacher model in the current training batch on different old classes to obtain the first standard output.

[0024] In one embodiment, the student model is used to generate inter-class output representations between the new class and the old class for the old class training samples and the new class training samples and the intra-class output representations of the old class training samples between the old classes and perform Z-score normalization respectively to obtain the second standard output and the third standard output, including: obtaining the logits value of the student model for each old class training sample and the new class training sample in the current training batch; obtaining the corresponding inter-class output representation according to the logits value of the same training sample output by the student model in the current training batch on different new classes and old classes; obtaining the corresponding intra-class output representation according to the logits value of the same old class training sample output by the student model in the current training batch on different old classes; performing Z-score normalization on the inter-class output representation and the intra-class output representation corresponding to the student model in the current training batch to obtain the second standard output and the third standard output.

[0025] In this embodiment, it is assumed that represents samples from the new class, where M is the number of samples, and and are input data and target labels respectively. Given an input ,make and Represent the output logit of the teacher model and the student model respectively.

[0026] This paper proposes a plug-and-play preprocessing method that uses loose semantic invariant matching to capture the true intrinsic inter-class relations and ensure fairness between new and old classes. In the actual implementation and reasoning process, the key factor is the inter-class semantic relations between logit values, because these semantic relations ultimately determine the final prediction results. In traditional knowledge distillation, the teacher's logits The distance measure between the logits of the students is , is minimized to zero, thus ensuring an exact match. However, our approach does not adopt such a strict alignment, but instead focuses on preserving the logical relational structure to maintain semantic integrity and accurate reasoning.

[0027] To achieve this, this method introduces Z-score normalization as a monotone positive linear transformation function to provide an order-preserving mapping. The monotone positive linear transformation provides a simple and effective mapping that remains invariant under separate changes in scale and logits shift: .

[0028] The preprocessing method of the present invention solves the weight drift problem faced by traditional incremental learning by normalizing logits, and reduces the impact of weight updates caused by scale and shift changes on the performance of old tasks. This method not only maintains semantic integrity and accurate reasoning, but also avoids the optimization difficulties caused by the precise matching requirements in traditional knowledge distillation, thereby reducing the instantaneous forgetting phenomenon during task switching.

[0029] In one embodiment, calculating the inter-class distillation loss according to the first standard output and the second standard output includes: calculating the inter-class probability distribution of the teacher model according to the first standard output; calculating the inter-class probability distribution of the student model according to the second standard output; calculating the KL loss according to the inter-class probability distribution of the teacher model and the student model to obtain the inter-class distillation loss.

[0030] In this embodiment, inter-class knowledge distillation helps the student model learn the teacher model's understanding of the distinction between categories by capturing the relative relationship between different categories. In each given training batch, there are multiple instances of old and new classes. For the old class set, the teacher model's inter-class prediction output logits for each instance is recorded as , the student model predicts the inter-class output logits of the same instance on each new class and old class as In inter-class knowledge distillation, and First, Z-score normalization is performed to obtain the normalized inter-class output, and then the similarity loss between the inter-class output distributions of the teacher model and the student model is calculated to make the inter-class distribution of the student model match the teacher model as much as possible. This distillation method helps the student model learn the relative discrimination ability of the teacher model between the old classes.

[0031] The plug-and-play preprocessing method proposed in the present invention captures the inherent inter-class relationship and ensures fairness between new and old classes through loose semantic invariant matching. This processing method allows the student model to retain knowledge of old categories while learning new categories, effectively alleviating the challenge of forgetting old task knowledge when learning new tasks. Specifically, the present invention uses Z-score normalization processing to make the changes in the distribution mean and variance of the logits output of the student model unrestricted, while retaining the semantic relationship of the old class logits of the teacher model. This improvement greatly enhances the ability of the student model to learn new classes while reducing the forgetting of old knowledge. Thereby solving the stability-plasticity dilemma faced by traditional class incremental learning.

[0032] In one embodiment, the inter-class distillation loss is:

[0033] in is the inter-class distillation loss, is the loss calculated based on KL divergence, is an image sample, It is the sample obtained by normalizing the logits value of the teacher model after Z score standardization for the old class In the The image classification probability on each old class, Represents the sample obtained by normalizing the logits value of the student model output by Z score for both the new class and the old class In the The image classification probability on each old class, is the distillation temperature parameter, is the number of old categories in the classification task, is the number of new categories.

[0034] In this embodiment, Gejiu class and In the incremental task of new classes, the goal is to train the model to The model uses inter-class distillation loss to balance the weight parameters between new and old categories. In the inter-class distillation loss, , The logits values ​​of the same old class image samples on different old classes output by the teacher model, The logits value of the same image sample output by the student model on different new and old classes, for the teacher model, Representative from arrive (i.e. the number of old categories), for the student model, Representative from arrive The category number (i.e. the total number of categories consisting of old and new categories).

[0035] In one embodiment, calculating the intra-class distillation loss according to the first standard output and the third standard output includes: calculating the intra-class probability distribution of the teacher model according to the first standard output; calculating the intra-class probability distribution of the student model according to the third standard output; calculating the KL loss according to the intra-class probability distributions of the teacher model and the student model to obtain the intra-class distillation loss.

[0036] In this embodiment, after applying Z-score normalization, the implicit information (called dark knowledge) in the teacher's output probability is more effectively transmitted. The present invention proposes an intra-class relationship extraction method to further maintain fairness within the old class.

[0037] By capturing the semantic similarity of different instances on each old class, for each given training batch, there are specific old classes o and k instances, and the teacher model will classify these k instances and output the predicted logits, denoted as , the student model will classify these k instances and output the predicted logits, recorded as , the two pairs of logits are first Z-score standardized and then intra-class knowledge distillation is performed. By introducing intra-class relationship distillation, fairness between old classes is further guaranteed. This method ensures fairness within old classes by extracting the teacher model's predictions for multiple instances of each old class, allowing the student model to learn knowledge from old classes equally. This consideration of intra-class relationships emphasizes the importance of intra-class similarity and further improves the overall performance of the model.

[0038] In one embodiment, the intra-class distillation loss is:

[0039] in, is the intra-class distillation loss, is the loss calculated based on KL divergence, is the number of old categories in the classification task, is an image sample, is the Z-score standardized logits value output by the teacher model. The normalized samples of the old class In the The image classification probability on each old class, The Z-score standardized logits values ​​representing the output of the student model are obtained by The normalized samples of the old class In the The image classification probability on each old class, is the distillation temperature parameter.

[0040] In this embodiment, is the number of old categories in the classification task. This loss does not involve new categories. The model balances the weight parameters between old categories through intra-class distillation loss. Specifically:

[0041] in, The logits values ​​of the same old class training image samples on different old classes output by the teacher model, The logits values ​​of the same old class training image samples on different old classes output by the student model, represents Z-score standardization, Representative from arrive The category number of .

[0042] In one embodiment, the total loss is:

[0043] in, is the classification loss, is the inter-class distillation loss, is the intra-class distillation loss, α and β are coefficients for controlling the weights of inter-class and intra-class losses, respectively. In this embodiment, the classification loss is represented by cross entropy loss:

[0044] in, is the indicator function of the true label, is the predicted probability of the correct category, and the knowledge distillation loss based on KL divergence is given as follows: .

[0045] In a specific embodiment, Figure 5 As shown in the figure, a pseudo code diagram of the training algorithm is provided. First, the Z-score normalization function is defined. Specifically, the input logits are Z-score normalized so that their mean is 0 and their standard deviation is 1. The normalization formula is (logits−mean) / (std+1e−7). Secondly, the input parameters are defined, where O is the number of old classes, N is the number of new classes, and k is the batch size. is the output logits of the student model (Student Output Logits), with a shape of [k, O+N], is the output logits of the teacher model (Teacher Output Logits), with a shape of [k, O], τ is the temperature coefficient (Temperature Scalar), which is used to smooth logits, α, β are hyperparameters (Hyperparameters), which are used to control the weights of inter-class and intra-class losses respectively, and finally, calculate (calculate) the inter-class knowledge distillation loss (Inter-Class Distillation Loss) and the intra-class knowledge distillation loss (Intra-Class Distillation Loss), and define the total loss function (Total Distillation Loss) based on the classification loss, inter-class knowledge distillation loss and intra-class knowledge distillation loss. The method of the present invention improves the performance of the model without increasing the additional training cost. This is particularly important in a resource-constrained environment, so that the model can more effectively handle class incremental learning tasks under limited computing and storage conditions.

[0046] The steps to calculate the inter-class knowledge distillation loss include: Step 1: Perform Z-score normalization on the output logits Zt of the teacher model to obtain the normalized output , perform Z-score standardization on the output logits Zs[:,:O] of the student model on the old class to obtain the standardized output , to eliminate the effects of scale and shift.

[0047] Step 2: Based on the normalization, calculate the distance metric between the logits of the student model and the logits of the teacher model: Calculate and The normalized probability distribution (softmax) of is then used to calculate the Kullback-Leibler divergence between the two as the loss of inter-class knowledge distillation.

[0048] Step 3: To scale the losses, multiply by the temperature coefficient And take the mean.

[0049] The steps to calculate the intra-class knowledge distillation loss include: Step 1: For each given training batch, the output logits of the teacher model and the student model (Zt[o] and Zs[o], respectively) are Z-score normalized for the k instances in each old class o.

[0050] Step 2: Calculate the normalized probability distribution (softmax) of the logits between the teacher model and the student model for different instances within the old class, and then calculate their Kullback-Leibler divergence as the intra-class knowledge distillation loss.

[0051] Step 3: Multiply by the temperature coefficient as well And take the mean.

[0052] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0053] In one embodiment, Figure 6 As shown, a class-incremental image classification system based on knowledge distillation is provided, including: Model acquisition module 602, used to acquire old class training samples, new class training samples and pre-built image classification models; the image classification model includes a student model and a teacher model; The sample input module 604 is used to generate output representations of old class training samples between old classes through the teacher model and perform Z-score standardization respectively to obtain a first standard output, generate inter-class output representations of old class training samples and new class training samples between new classes and old classes and intra-class output representations of old class training samples between old classes through the student model and perform Z-score standardization respectively to obtain a second standard output and a third standard output; A loss construction module 606 is used to calculate the inter-class distillation loss according to the first standard output and the second standard output, calculate the intra-class distillation loss according to the first standard output and the third standard output, and obtain the total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss; The image classification module 608 is used to optimize the total loss of the image classification model to obtain a trained image classification model, and use the trained image classification model to perform image classification.

[0054] In one of the embodiments, it is also used to obtain the logits value of the teacher model for each old class training sample in the current training batch; the logits values ​​of the same old class training samples output by the teacher model in the current training batch on different old classes are Z-score standardized to obtain a first standard output.

[0055] In one of the embodiments, it is also used to obtain the logits value of the student model for each old class training sample and new class training sample in the current training batch; obtain the corresponding inter-class output representation according to the logits value of the same training sample output by the student model in the current training batch on different new classes and old classes; obtain the corresponding intra-class output representation according to the logits value of the same old class training sample output by the student model in the current training batch on different old classes; perform Z-score normalization on the inter-class output representation and intra-class output representation corresponding to the student model in the current training batch, respectively, to obtain the second standard output and the third standard output.

[0056] In one of the embodiments, it is also used to calculate the inter-class probability distribution of the teacher model according to the first standard output; calculate the inter-class probability distribution of the student model according to the second standard output; calculate the KL loss according to the inter-class probability distribution of the teacher model and the student model to obtain the inter-class distillation loss.

[0057] In one embodiment, the inter-class distillation loss is:

[0058] in, is the inter-class distillation loss, is the loss calculated based on KL divergence, is an image sample, It is the sample obtained by normalizing the logits value of the teacher model after Z score standardization for the old class In the The image classification probability on each old class, Represents the sample obtained by normalizing the logits value of the student model output by Z score for both the new class and the old class In the The image classification probability on each old class, is the distillation temperature parameter, is the number of old categories in the classification task, is the number of new categories.

[0059] In one of the embodiments, it is also used to calculate the intra-class probability distribution of the teacher model according to the first standard output; calculate the intra-class probability distribution of the student model according to the third standard output; calculate the KL loss according to the intra-class probability distribution of the teacher model and the student model to obtain the intra-class distillation loss.

[0060] In one embodiment, the intra-class distillation loss is:

[0061] in, is the intra-class distillation loss, is the loss calculated based on KL divergence, is the number of old categories in the classification task, is an image sample, is the Z-score standardized logits value output by the teacher model. The normalized samples of the old class In the The image classification probability on each old class, The Z-score standardized logits values ​​representing the output of the student model are obtained by The normalized samples of the old class In the The image classification probability on each old class, is the distillation temperature parameter.

[0062] In one embodiment, the total loss is:

[0063] in, is the classification loss, is the inter-class distillation loss, is the intra-class distillation loss, and are the coefficients controlling the weights of inter-class and intra-class losses, respectively.

[0064] For the specific limitations of the class-incremental image classification system based on knowledge distillation, please refer to the limitations of the class-incremental image classification method based on knowledge distillation above, which will not be repeated here. Each module in the above-mentioned class-incremental image classification system based on knowledge distillation can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0065] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input system connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a class incremental image classification method based on knowledge distillation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input system of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0066] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0068] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A class-incremental image classification method based on knowledge distillation, characterized in that: The method comprises: Obtaining old class training samples, new class training samples and a pre-built image classification model; the image classification model includes a student model and a teacher model; Generate output representations of old class training samples between old classes by the teacher model and perform Z-score standardization respectively to obtain a first standard output; generate inter-class output representations of old class training samples and new class training samples between new classes and old classes and intra-class output representations of old class training samples between old classes by the student model and perform Z-score standardization respectively to obtain a second standard output and a third standard output; Calculate the inter-class distillation loss according to the first standard output and the second standard output, calculate the intra-class distillation loss according to the first standard output and the third standard output, and obtain the total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss; The total loss of the image classification model is optimized to obtain a trained image classification model, and the trained image classification model is used to perform image classification.

2. The method according to claim 1, characterized in that The output representation of the old class training samples between the old classes is generated by the teacher model and the Z-score normalization is performed respectively to obtain the first standard output, which includes: Get the logits value of the teacher model for each old class training sample in the current training batch; The logits values ​​of the same old class training samples output by the teacher model in the current training batch on different old classes are Z-score standardized to obtain the first standard output.

3. The method according to claim 1, characterized in that The student model is used to generate inter-class output representations of old class training samples and new class training samples between new classes and old classes, and intra-class output representations of old class training samples between old classes, and Z-score standardization is performed respectively to obtain the second standard output and the third standard output, including: Get the logits value of the student model for each old class training sample and new class training sample in the current training batch; According to the logits values ​​of the same training sample output by the student model in the current training batch on different new and old classes, the corresponding inter-class output representation is obtained; According to the logits values ​​of the same old class training samples output by the student model in the current training batch on different old classes, the corresponding in-class output representation is obtained; The inter-class output representation and the intra-class output representation corresponding to the student model in the current training batch are respectively Z-score standardized to obtain the second standard output and the third standard output.

4. The method according to claim 1, characterized in that: Calculating the inter-class distillation loss according to the first standard output and the second standard output includes: Calculate the between-class probability distribution of the teacher model based on the first standard output; Calculate the between-class probability distribution of the student model based on the second standard output; The KL loss is calculated according to the inter-class probability distribution of the teacher model and the student model to obtain the inter-class distillation loss.

5. The method according to claim 1 or 4, characterized in that: The inter-class distillation loss is: in, is the inter-class distillation loss, is the loss calculated based on KL divergence, is an image sample, It is the sample obtained by normalizing the logits value of the teacher model after Z score standardization for the old class In the The image classification probability on each old class, Represents the sample obtained by normalizing the logits value of the student model output by Z score with the new class and the old class together. In the The image classification probability on each old class, is the distillation temperature parameter, is the number of old categories in the classification task, is the number of new categories.

6. The method according to claim 1, characterized in that Calculating the intra-class distillation loss according to the first standard output and the third standard output includes: Calculate the intra-class probability distribution of the teacher model based on the first standard output; Calculate the within-class probability distribution of the student model based on the third standard output; The KL loss is calculated according to the intra-class probability distribution of the teacher model and the student model to obtain the intra-class distillation loss.

7. The method according to claim 1 or 6, characterized in that: The intra-class distillation loss is: in, is the intra-class distillation loss, is the loss calculated based on KL divergence, is the number of old categories in the classification task, is an image sample, is the Z-score standardized logits value output by the teacher model. The normalized samples of the old class In the The image classification probability on each old class, The Z-score standardized logits values ​​representing the output of the student model are obtained by The normalized samples of the old class In the The image classification probability on each old class, is the distillation temperature parameter.

8. The method according to claim 1, characterized in that The total loss is: in, is the classification loss, is the inter-class distillation loss, is the intra-class distillation loss, and are the coefficients controlling the weights of inter-class and intra-class losses, respectively.

9. A class-incremental image classification system based on knowledge distillation, characterized in that: The system comprises: A model acquisition module, used to acquire old class training samples, new class training samples and a pre-built image classification model; the image classification model includes a student model and a teacher model; A sample input module is used to generate output representations of old class training samples between old classes through the teacher model and perform Z-score standardization respectively to obtain a first standard output, generate inter-class output representations of old class training samples and new class training samples between new classes and old classes and intra-class output representations of old class training samples between old classes through the student model and perform Z-score standardization respectively to obtain a second standard output and a third standard output; a loss construction module, used to calculate the inter-class distillation loss according to the first standard output and the second standard output, calculate the intra-class distillation loss according to the first standard output and the third standard output, and obtain the total loss according to the inter-class distillation loss, the intra-class distillation loss and the classification loss; The image classification module is used to optimize the total loss of the image classification model to obtain a trained image classification model, and use the trained image classification model to perform image classification.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images