A continuous learning performance evaluation method based on class activation region offset metric

Abstract

The application discloses a kind of based on class activation area offset metric's continuous learning performance evaluation method, belong to the class incremental continuous learning field of depth neural network model.This application carries out image processing task, and depth neural network model needs to extract and focus on the object to be processed in image, to complete subsequent downstream task;In the continuous learning scene, the catastrophic forgetting of model in the learning of subsequent task can cause its extraction and attention ability to lose target in previous task.This scheme proposes to use Grad-cam class activation mapping spectrum to measure the effectiveness of continuous learning algorithm to curb catastrophic forgetting in different task stages, mainly gives the quantitative attention area stability evaluation index, and forms the deviation and correlation index in combination with classification accuracy, finally, DV and RL are integrated as DR evaluation index to measure the overall performance of continuous learning algorithm.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of class-incremental continual learning of deep neural network model. In the dynamic data flow scenario, the required recognition categories of the model are constantly increasing, but the previous data cannot be obtained due to storage limitations and privacy protection, etc., which leads to the catastrophic forgetting phenomenon of the deep neural network model. BACKGROUND

[0002] Current deep neural networks have achieved good results in many tasks, such as image classification, detection and segmentation, and natural language processing, etc. However, these are based on training and testing on a static complete data set. In reality, data flow is often not static, and deep learning models need to face dynamic data flow, i.e. after the first training of the model is completed, it still needs to be trained and tested on newly emerging data, but the previous old training task data cannot be obtained. At this time, the catastrophic forgetting problem will cause the deep learning model to perform poorly, especially the test accuracy of the previous old task is greatly reduced, so that the model cannot adapt to the dynamic change of data distribution over time. In class-incremental continual learning, the model needs to continuously increase the recognition of new classes in different tasks during inference, and the task identifier is unknown. In this case, the catastrophic forgetting problem limits the model to a static environment and can only recognize a limited number of classes.

[0003] In order to solve the training problem of deep learning model when facing dynamic data flow, a series of improved algorithms have been proposed, such as model regularization, sample replay, dynamic network expansion, etc. The improvement effect evaluation of these algorithms in the class-incremental continual learning scenario is often based on the classification accuracy of the model. The main indicators include the final classification accuracy, the process average classification accuracy, and the forgetting rate which measures the degree of model forgetting. The final classification accuracy of the model is the classification accuracy of the model on the test set of all task categories that have been learned. The process average accuracy is the average value of the final classification accuracy after each task has been learned. The forgetting rate refers to the degree of decrease in classification accuracy of the model on the test set of the previous learned old task. The average forgetting rate refers to the average value of the forgetting rate of the previously learned old task. These indicators based on the classification accuracy of the model cannot directly reflect the specific forgetting situation of the model, and more dimensional discriminant indicators need to be added to better understand the reasons for the catastrophic forgetting of the model.

[0004] Due to the unexplainability of deep neural network model, class activation mapping spectrum is often used to visualize the attention degree of the network to different regions of the input image, so as to understand the processing of the model to the input image. The basic principle is to take the output feature map of the last convolutional layer and apply a weight parameter, so as to obtain a class activation mapping spectrum. Combining the class activation mapping spectrum with the original input image to form a heat map can show which regions of the input image are paid more attention to by the network. In the image classification task, it is embodied whether the classification target corresponding to the label is accurately found. The scheme applies the gradient weighted class activation mapping spectrum Grad-cam to the evaluation scene of class-incremental continuous learning, designs quantifiable indexes to measure the stability of the attention region of the test image by different algorithms, and combines the model classification accuracy to construct an evaluation index for measuring the whole continuous learning process. SUMMARY

[0005] Although the final classification accuracy, average classification accuracy and forgetting rate and other indexes generated based on the classification accuracy of the model to the test sample can reflect the final performance of the model in class-incremental continuous learning, the specific performance and reason of the catastrophic forgetting of the deep learning model lack intuitive explanation and measurement, and the measurement dimension is relatively single. In order to make intuitive explanation and quantifiable evaluation of the catastrophic forgetting phenomenon of the model from more dimensions, the application proposes to use the class activation mapping spectrum for explaining the attention degree of the deep neural network model to different regions of the input image to judge the forgetting degree of the model, and combine the model classification accuracy index to evaluate the overall performance of the algorithm continuous learning process.

[0006] When performing an image processing task, the deep neural network model needs to extract and pay attention to the object to be processed in the image to complete the subsequent downstream tasks such as image classification, target detection and target segmentation; in the continuous learning scene, the catastrophic forgetting of the model in the learning of the subsequent task may cause the loss of extraction and attention ability to the target in the previous task. The specific performance is the change of the attention region in the test image. Measuring the change of the attention region to the target region can reflect the forgetting degree of the deep neural network model to the knowledge. Therefore, the scheme proposes to use the change of the Grad-cam class activation mapping spectrum in different task stages to measure the effectiveness of the continuous learning algorithm in suppressing catastrophic forgetting, mainly gives the quantifiable changes in regions of interest (CRI) evaluation index, and combines the classification accuracy to form the deviation (DV) and relevance (RL) indexes, and finally the DV and RL are combined as the DR evaluation index to measure the overall performance of the continuous learning algorithm. Thus, the technical scheme of the application is: a continuous learning performance evaluation method based on class activation region deviation measurement, the method comprises:

[0007] Step 1: For each image recognition task t learned in successive training, when model M has completed the training set {x} for the current image recognition task... t ,y t After learning from}, for the test set of the current image recognition task The i-th image In its true category Activation map on Save and set the standard class activation map spectrum for this image, and let z t For all The set representation; and simultaneously calculate the final classification accuracy FA for the test set of all learned image recognition task categories after the image recognition task is completed. t The calculation is performed using the following formula:

[0008]

[0009] Where, N m This represents the number of test set samples for the m-th task;

[0010] Step 2: Calculate the test data for the old image recognition task The class activation map z on the model after learning the current image recognition task t, where m < t. m,t This is used to calculate the stability of the region of interest; for the test dataset of the current image recognition task t, we have z t,t =z t For t=0, there is no activation map spectrum z for the old image recognition task test data class. m,t ;

[0011] Step 3: Apply the activation mapping spectrum z to the old image recognition task test data class. m,t and the standard class activation map spectrum generated during the first training (standard map z) m Perform probability normalization preprocessing to obtain the normalized result z′ m,t and z′ m This is done to measure the similarity of the regions of interest; to amplify the representation of the differences in the regions of interest, the probability distribution is normalized using the normalized exponential function shown in Formula 2; for a class activation map spectrum z, the processing procedure is shown in the following formula:

[0012]

[0013] Where z′ represents the class activation map spectrum after probability normalization, p, q represent points in the class activation map spectrum with x and y coordinates p and q respectively, and z(p, q) represents points in the class activation map spectrum z with x and y coordinates p and q respectively; where ρ is the nonlinear mapping amplification parameter, here ρ = 0.1 is taken to amplify the class activation map spectrum with a numerical distribution range of 0-1 to 0-10 to fully reflect the differences in the region of interest of different algorithms;

[0014] Step 4: Normalize the probability z′ m,t and z′ m The SIM matrix similarity measure compares the corresponding elements of two matrices, sums the smaller values, and assigns a score of 1 if the two distributions are identical and a score of 0 if they have no overlap. The formula is as follows:

[0015] SIM(z′ m,t ,z′ m )=∑ p ∑ q MIN(z′ m,t (p,q),z′ m (p,q)) (3)

[0016] z′ m,t (p,q) represents z′ m,t The x-coordinate and y-coordinate of the point z′ are on points p and q respectively. m (p,q) represents z′ m The points in the middle have x-coordinates and y-coordinates on p and q, respectively;

[0017] In each task phase t, test image samples from previous image recognition tasks are tested using class activation mapping spectra and standard maps to obtain similarity scores, and the average score is taken as the CRI score for the stability of the model's region of interest. t ;

[0018]

[0019] in, This represents the class activation mapping spectrum of the i-th image in the m-th task test set on the t-th task model. This represents the standard mapping spectrum of the i-th image in the m-th task test set on the m-th task model;

[0020] Furthermore, when 0 ≤ m < t, t ≥ 1 is required, which means that the degree of change in attention to the test image region of the previous old image recognition task is defined as CRI on each task after completing the initial image recognition task. t Similarly, we can obtain the classification accuracy FA of the old image recognition task under the condition 0≤m<t.t -past;

[0021] Step 5: combined with the model classification accuracy, construct the deviation DV and the correlation degree RL; the specific method is, with the attention region stability CRI t as the abscissa, with the classification accuracy FA t as the ordinate, taking the coordinate (CRI0, FA0) of the first task as the standard point, the Euclidean distance between the subsequent continuous learning tasks (CRI t , FA t ) and the standard point is defined as the deviation DV and the average is taken; in addition, the straight line with a slope of 1 passing through the standard point is set as the standard line, and the distance from the subsequent task point to the standard line is taken as the correlation degree RL and the average is taken, and the specific calculation formula is:

[0022]

[0023]

[0024] Wherein, T represents that the class increment continuous learning has T tasks in common;

[0025] Step 6: combine the deviation and the correlation, use the S-shaped curve to normalize the two indexes respectively, multiply and subtract 1 to form the evaluation index DR, a good continuous learning algorithm DR should approach 1, and the algorithm with poor performance will approach 0;

[0026]

[0027] Wherein, alpha1 and alpha2 are the S curve normalization parameters of DV and RL respectively, alpha1=2alpha2.

[0028] The application can evaluate the effect of the class increment continuous learning algorithm on the attention region stability of the test image, and measures the advantages and disadvantages of the class increment learning algorithm from the perspective of the region attention degree of the model to the input image at different task stages, wherein CRI directly evaluates the attention region stability of the algorithm to the test image; DV shows the distance between the final classification accuracy and the attention region stability after the first training of the continuous learning algorithm and the initial performance, and in the ideal case, DV should be consistent with the initial performance, and the value is 0; in addition, the correlation degree RL shows whether the region attention stability and the classification accuracy in the subsequent tasks of the algorithm model reflect consistency; the greater the RL value, the lower the relevance of the classification accuracy and the region attention stability of the model in the continuous learning subsequent tasks, and the more serious the loss of the mapping relationship between the attention ability of the model to the class objects of the old tasks and the classification accuracy; finally, DR measures the overall performance of the algorithm in the class increment continuous learning process in the attention region stability and the classification accuracy of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 Evaluation results of CRI on different continual learning algorithms.

[0030] Figure 2 Evaluation results of FA on different continual learning algorithms.

[0031] Figure 3 Evaluation results of CRI-past on different continual learning algorithms.

[0032] Figure 4 Evaluation results of CRI-past on different continual learning algorithms.

[0033] Figure 5 CRI score results on the same test sample for several typical continual learning algorithms.

[0034] Figure 6 CRI-past average values for different algorithm classification correct or error samples.

[0035] Figure 7 Different algorithm CRI-FA continuous learning observation points and standard points distance and distance to standard line compare. Embodiments

[0036] Class-incremental continual learning refers to training and testing a model in a dynamic data stream, and the model can only access part of the training data at a time. Let t represent the ID of the current task, and let t = 0 be the first task learned by the model, and so on. The input data of the t-th task is denoted as where x t is the image set of the t-th task, y t is the corresponding label set, {x t , y t} is the training data, is the test data. Let C t represent the number of classes of the t-th task, and N t represent the number of samples in the test data set of the t-th task. T is the total number of tasks. For class-incremental learning, different stages have different classes to learn, and there is no class overlap, i.e. Class-incremental continual learning requires recognition of the increasing classes of the task, and the ability to recognize the classes of the old task cannot be lost. For the test image The class-incremental continual learning model must verify all learned classes , because the ID t of the task will not be given when reasoning.

[0037] Experimental part:

[0038] Dataset: CIFAR100 dataset is used, which contains 100 classes in total. Each class has 500 training images and 100 test images, a total of 60,000 images. The size of each image is 32x32. In CIFAR-100, 100 classes are divided into 10 tasks, each task has 10 classes, and class-incremental continual learning training test is performed.

[0039] The selected class-incremental continual learning algorithms for testing are: (1) Finetune algorithm, which does not use any additional method to prevent network forgetting, and directly fine-tunes the model on different task datasets. (2) Ewc algorithm, which uses parameter matrix to estimate the importance of network parameters, and the model hinders the change of important parameters on old tasks when learning new tasks. (3) Lwf algorithm, which copies and freezes the model on the old task, and uses the distillation strategy to approximate the response of the copied old model while training on the new task. (4) Replay algorithm, which retains a small part of the old task training data and trains the model with the new task data. (5) Icral algorithm, which retains old task data by targeted screening and combines old model response distillation loss for continuous learning training. (6) Wa algorithm, which reduces the imbalance between old and new classes by aligning the average modulus of the weight vectors of new and old classes while retaining part of the old task samples for continuous learning.

[0040] The deep neural network structure used in the experiment is Resnet32, and the Grad-cam algorithm is used to generate class activation mapping. In order to more clearly and accurately show the experimental results, all index tables and figure values are enlarged from 0-1 to 0-100.

[0041] By observing Figure 1 and Figure 2 It can be seen that the CRI curve reflects the scores of different algorithms in the stability of the focus area, and the performance of the algorithm is basically consistent with the performance of the FA curve of Figure 2 , but some algorithms such as Lwf, although the CRI curve is high, the FA value is very low. This is also reflected in the CRI-past and FA-past curves of Figure 3 , Figure 4

[0042] In Figure 5 , the CRI score of the same test image belonging to the table class is shown, and the quantitative score of the focus area stability of different algorithms and its offset degree are basically consistent with the subjective observation results. At the same time, the CRI-past score results of each algorithm are decomposed into classification correct sample CRI score results and classification error CRI score results, such as Figure 6 ​As shown, the CRI average score of the test samples classified correctly is higher than that of the samples classified incorrectly. This indicates that the region of interest stability CRI is the basis for the final classification performance of the continuous learning algorithm.

[0043] The CRI-FA curves of different algorithms in the continuous learning of each task stage are plotted as Figure 7 As shown, the DV value and the RL value and the comprehensive index DR are calculated, as shown in Table 1, and the s-curve normalization parameters are al = 2 and a2 = 20. Although the DV value of the algorithm without sample replay such as Ewc and Lwf is better than that of the baseline algorithm, the final DR is lower than that of the baseline algorithm due to the poor correlation performance, indicating that the mapping relationship between the region of interest stability and the final classification accuracy of these algorithms has changed greatly, although they maintain a high classification accuracy. The DR value of the algorithm using the old sample replay scheme is obviously improved compared with the baseline algorithm, indicating that the stability and mapping relationship between the region of interest stability and the classification accuracy are well maintained.

[0044] Figure 7 The distance between the observation points and the standard points of different algorithms CRI-FA in the continuous learning of each task stage (the color from light to dark represents the distance from small to large) and the distance to the standard line compare (shown by the dashed line in the figure).

[0045] Table 1: Comparison of DV, RL and DR values of different algorithms in continuous learning experiments

[0046] Methods Finetune Ewc Lwf Replay Icarl Wa DV (normalized) 82.13 80.73 75.32 72.55 70.81 67.01 RL (normalized) 43.25 47.94 53.00 30.87 23.82 27.56 DR 64.48 61.30 60.08 77.60 83.13 81.53

Claims

1. A continuous learning performance evaluation method based on class activation region deviation metric, the method comprising: Step 1: For each image recognition task t learned in successive training, when model M has completed the training set {x} for the current image recognition task... t ,y t After learning from}, for the test set of the current image recognition task The i-th image In its true category Activation map on Save and set the standard class activation map spectrum for this image, and let z t For all The set representation; and simultaneously calculate the final classification accuracy FA for the test set of all learned image recognition task categories after the image recognition task is completed. t The calculation is performed using the following formula: where N m represents the number of test set samples of the mth task; Step 2: Compute old image recognition task test data Class activation map spectrum z on the model after completing learning of the current image recognition task t m,t , for the case of computing the region of interest stability; for the test data set of the current image recognition task t, there is z t,t = z t ; for t = 0, there is no old image recognition task test data class activation map spectrum z m,t ; Step 3: Old image recognition task test data class activation map spectrum z m,t and the standard class activation map spectrum standard map z produced by the first training m Probability normalization preprocessing is performed to obtain the normalized result z' m,t and z' m In order to measure the similarity of the attention area, the probability distribution is normalized using the normalization exponential function shown in public 2; for a class activation map spectrum z, the processing process is shown in the following formula: wherein z' represents the class activation map spectrum after probability normalization, p and q represent the points with p and q as the horizontal and vertical coordinates in the class activation map spectrum, and z(p, q) represents the point with p and q as the horizontal and vertical coordinates in the class activation map spectrum; wherein p is a nonlinear mapping amplification parameter, and here p = 0.1 is taken to amplify the class activation map spectrum with a numerical distribution range of 0-1 to 0-10 to fully reflect the difference in the change of the attention region of different algorithms; Step 4: Normalizing the probabilities z' m,t and z' m The SIM matrix similarity measure is performed, which is a comparison of two matrices by taking the minimum value at each corresponding position and summing them up. If the two distributions are identical, the final score is 1, and if the two distributions have no intersection, the final score is 0. The formula is: SIM(z′ m,t ,z′ m )=∑ p ∑ q MIN(z′ m,t (p,q),z′ m (p,q)) (3) z' m,t (p, q) denotes a point in z' m,t with horizontal and vertical coordinates p, q, respectively m (p, q) denotes a point in z' m with horizontal and vertical coordinates p, q, respectively In each task stage t, the test image samples of the previous old image recognition task are mapped to the class activation map spectrum and the standard map for testing to obtain a similarity score, and the average value is taken as the score CRI of the model focus region stability t ; wherein, represents the class activation map spectrum of the i-th image of the m-th task test set on the t-th task model, represents the standard map spectrum of the i-th image of the m-th task test set on the m-th task model; In addition, when 0≤m t -past, the classification accuracy of the old image recognition task under the condition of 0≤m t -past; Step 5: Construct deviation DV and correlation RL according to the classification accuracy of the model; the specific method is to take the stability of the attention region CRI t as the abscissa, and take the classification accuracy FA t as the ordinate, take the coordinate (CRI0, FA0) of the first task as the standard point, and the Euclidean distance between the subsequent continuous learning tasks (CRI t , FA t ) and the standard point is defined as the deviation DV and the average value; in addition, the straight line with a slope of 1 passing through the standard point is set as the standard line, and the distance from the subsequent task point to the standard line is taken as the correlation RL and the average value, and the specific calculation formula is: wherein T represents that there are T tasks in common in class incremental continuous learning; Step 6: Synthesize the deviation and the correlation, use the S-shaped curve to normalize and multiply the two indexes respectively, and form the evaluation index DR by subtracting 1; a good continuous learning algorithm should approach 1, and a poor algorithm will approach 0; wherein a1 and a2 are S-curve normalization parameters of DV and RL respectively, and a1 = 2a2.

Images