Class incremental learning method and product based on dynamic class prototype generation mechanism

Abstract

The present invention proposes a class incremental learning method and product based on a dynamic class prototype generation mechanism, which belongs to the field of artificial intelligence and includes the following steps: 1) dividing the class incremental data set and allocating it to T+1 training stages as training samples; 2) training the feature extractor and classifier of the current stage; 3) using the feature representation of the samples of the current stage to define the feature density, and calculating the category prototype based on the feature density; 4) using the prototype drift estimation method based on topology correction to update the category prototypes of all previous stages; 5) using the updated category prototypes of all previous stages to obtain pseudo features of each category; 6) defining a multi-objective loss function to learn a new feature space and classifier weights; 7) repeating steps 3) to 6) until the training of all incremental stages is completed; 8) after each training stage, using the model to evaluate all learned categories.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a class incremental learning method and product based on a dynamic class prototype generation mechanism. Background Art

[0002] In recent years, deep neural networks have been increasingly applied to dynamic environments, requiring them to continuously learn new knowledge from new data. However, continuous learning can lead to catastrophic forgetting, where the model's performance on old data deteriorates dramatically. Therefore, continuously learning new knowledge without forgetting the old knowledge has become a core requirement for many computer vision tasks (e.g., image recognition and object detection). Class incremental learning (CIL), a typical continuous learning task, aims to recognize new class samples while maintaining the ability to recognize old class samples. In CIL tasks, catastrophic forgetting has two main causes: (a) the neural network parameters associated with the old knowledge are rapidly changed to adapt to the new class data (i.e., network drift); and (b) because class samples from different CIL stages have not been trained together, they cannot form a good decision boundary between them (i.e., inter-class confusion between stages).

[0003] To mitigate catastrophic forgetting, many existing works use methods based on replaying old samples. However, these methods have two major limitations: (a) for some application scenarios with limited device memory, it is impossible to continuously store old samples of increasing categories; (b) the data privacy implications of storing old samples are often unacceptable. Summary of the Invention

[0004] This application provides a class incremental learning method and product based on a dynamic class prototype generation mechanism to solve the problem of how to alleviate catastrophic forgetting in class incremental learning without old samples. Based on the structural characteristics of this method, an efficient class incremental learning model and training method without old samples are invented.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present application provides a class incremental learning method based on a dynamic class prototype generation mechanism, comprising the following steps:

[0007] Step 1) The image dataset is divided into multiple non-overlapping datasets according to categories, and the multiple non-overlapping datasets are assigned to a base class stage and T incremental stages. The training samples of each training stage are the datasets assigned to the current stage.

[0008] Step 2) uses the training samples of the current stage in step 1) to train the feature extractor and classifier to learn the feature space and classifier weights of the current stage;

[0009] Step 3) using the feature extractor to obtain the feature representation of the training samples in the current stage, defining the feature density, and calculating the prototypes of each category in the current stage based on the feature density;

[0010] Step 4), in each incremental stage, for the prototypes of each category in all previous stages calculated in step 3), the prototype drift estimation method based on topology correction is used to update the prototypes of each category in all previous stages;

[0011] Step 5), in each incremental stage, using the prototypes of each category in all previous stages updated in step 4), pseudo features of each category in all previous stages are generated;

[0012] Step 6) In each incremental stage, based on the feature representation and classification prediction results of the training samples in the current stage in step 3), and the pseudo features and classification prediction results of each category in all previous stages in step 5), a multi-objective loss function is defined to learn a new feature space and classifier weights;

[0013] Step 7), repeat steps 3) to 6) until all incremental stage training is completed;

[0014] In step 8), the trained model is used to evaluate all learned categories with average accuracy and average forgetting rate as evaluation indicators, and a class incremental learning model based on the dynamic class prototype generation mechanism is obtained.

[0015] Based on the first aspect, in step 3), defining the feature density and calculating the prototypes of each category in the current stage based on the feature density includes:

[0016] For the category a∈C at stage t t , the initial feature space The definition is as follows:

[0017]

[0018] in, is the feature representation of the training sample at stage t, The feature representation of stage t is obtained by the feature extractor, i.e. y i are different categories in the training samples at stage t, is the new feature space;

[0019] Feature Representation The characteristic density D i The definition is as follows:

[0020]

[0021] Where E(·,·) represents the Euclidean distance, i and j represent the i-th and j-th training samples in stage t, respectively. is the feature representation of the jth training sample in stage t, and γ is a hyperparameter;

[0022] During the prototype generation process, the feature density D i Weight the features of category a, the initial prototype of category a The definition is as follows:

[0023]

[0024] Among them, Proto(·) is the prototype of category a, It is a feature representation The weight of D j Feature Representation The feature density of the initial prototype Represents the initial prototype set in stage t, category C 1：t Prototype collection Defined as: μ j is the prototype of the old category j in the previous t-1 stages.

[0025] Based on the first aspect, in step 4), the prototypes of each category in all previous stages are updated using a prototype drift estimation method based on topology correction, including the following steps:

[0026] In each incremental stage, the drift of the feature space in each incremental stage is calculated;

[0027] Using the drift of the feature space of each incremental stage, estimating the drift of the category prototypes of all previous stages;

[0028] The topological correction amount is calculated using the local topological relationship, and the prototypes of each category in all the previous stages are updated.

[0029] Based on the first aspect, in each incremental stage, the drift of the feature space of each incremental stage is calculated. The definition is as follows:

[0030]

[0031] Among them, category a∈C t , the feature representation of stage t Feature representation at stage t-1 f i t and f i t-1 are respectively generated by the feature extractor φ(x i θ t ) and φ(x i θ t-1 ) to obtain, Δf i Feature Representation Relative to feature representation The amount of drift;

[0032] The drift of the feature space of each incremental stage is used to estimate the drift of the category prototypes of all previous stages. The definition is as follows:

[0033]

[0034] Among them, category b∈C 1：t-1 , E(·,·) represents the Euclidean distance, For category b∈C 1：t-1 The prototype, is the feature representation of the j-th training sample in stage t, and η is a hyperparameter;

[0035] The topology correction amount is calculated by using the local topological relationship, and the prototypes of each category in all previous stages are updated. The topology correction amount Defined as The weighted sum of the K nearest neighbor drifts:

[0036]

[0037] in, It is the category prototype of all previous stages K nearest neighbors, Is a category prototype The drift amount, α is a hyperparameter;

[0038] The prototype topology correction drift of category b in all previous stages The definition is as follows:

[0039]

[0040] in, Is a category prototype The drift amount, Is a category prototype The topological correction amount, β is used to adjust the two contributions and Hyperparameters of

[0041] Update the prototype by stage to obtain the updated category prototype The definition is as follows:

[0042]

[0043] Based on the first aspect, in step 6), the multi-objective loss function is defined as follows:

[0044]

[0045] The definition is as follows:

[0046]

[0047] The definition is as follows:

[0048]

[0049] Among them, the feature representation of stage t is The dimension is d f ,Will It is represented by the feature extractor φ(·;θ t ) defined feature space, To predict the label, is the cosine classifier, y i is the true label, is the cross entropy loss, is the feature distillation loss, It is a pseudo feature constraint, λ1 and λ2 are used to adjust the two loss contributions and hyperparameters.

[0050] Based on the first aspect, the pseudo feature constraint The definition is as follows:

[0051]

[0052] in, yes The predicted probability, y i yes The true label of In each incremental stage, the updated category prototype is corrected by the prototype topology for drift The reconstructed pseudo feature space, pseudo feature space The definition is as follows:

[0053]

[0054] in, is the normalized pseudo feature of category b, e*σ is the Gaussian noise used to generate pseudo features, e and σ are related to have the same dimensions, Each dimension of σ represents the original feature space The standard deviation of the corresponding dimension in .

[0055] In the second aspect, an embodiment of the present application provides an image classification method based on a class incremental learning model based on a dynamic class prototype generation mechanism, wherein the image to be classified is input into the class incremental learning model based on a dynamic class prototype generation mechanism to obtain an image classification result of the image to be classified.

[0056] In a third aspect, an embodiment of the present application provides a class incremental learning system based on a dynamic class prototype generation mechanism, including:

[0057] A class incremental dataset partitioning module, used in step 1), divides the image dataset into multiple non-overlapping datasets according to categories, and assigns the multiple non-overlapping datasets to one base class stage and T incremental stages, with the training samples of each training stage being the dataset assigned to the current stage;

[0058] The feature space extraction and classification module is used in step 2) to train the feature extractor and classifier using the training samples of the current stage in step 1) to learn the feature space and classifier weights of the current stage;

[0059] A prototype calculation module, used in step 3), uses the feature extractor to obtain the feature representation of the training samples in the current stage, defines the feature density, and calculates the prototype of each category in the current stage based on the feature density;

[0060] A topology correction module, used in step 4), updates the prototypes of each category in all previous stages calculated in step 3) in each incremental stage by using a prototype drift estimation method based on topology correction;

[0061] The pseudo-feature generation module is used in step 5) to generate pseudo-features for each category in all previous stages in each incremental stage using the prototypes of each category in all previous stages updated in step 4);

[0062] A multi-objective loss function calculation module is used in step 6) to define a multi-objective loss function in each incremental stage based on the feature representation and classification prediction results of the training samples in the current stage in step 3) and the pseudo features of each category and their classification prediction results in all previous stages in step 5) to learn a new feature space and classifier weights;

[0063] The incremental learning model training module is used in step 7) and repeats steps 3) to 6) until all incremental stage training is completed;

[0064] The class incremental learning model evaluation module, used in step 8), uses the trained model to evaluate all learned categories with average accuracy and average forgetting rate as evaluation indicators, and obtains a class incremental learning model based on the dynamic class prototype generation mechanism.

[0065] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; and when the one or more programs are executed by the processor, implementing a method as described in any one of the above-mentioned first aspects.

[0066] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the above-mentioned first aspects.

[0067] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0068] The embodiment of the present invention provides a class incremental learning method and product based on a dynamic class prototype generation mechanism, wherein step 1) the image dataset is divided into a plurality of non-overlapping datasets according to categories, and the plurality of non-overlapping datasets are assigned to a base class stage and T incremental stages, and the training samples of each training stage are the datasets assigned to the current stage; step 2) the training samples of the current stage are used to train the feature extractor and the classifier to learn the feature space and classifier weights of the current stage; step 3) the feature extractor is used to obtain the feature representation of the training samples of the current stage, the feature density is defined, and the prototypes of each category of the current stage are calculated based on the feature density; step 4) in each incremental stage, for the prototypes of each category of all previous stages calculated in step 3), a topological correction method is used. The positive prototype drift estimation method updates the prototypes of each category in all previous stages; step 5), in each incremental stage, the prototypes of each category in all previous stages updated in step 4) are used to generate pseudo features of each category in all previous stages; step 6), in each incremental stage, based on the feature representation of the training samples in the current stage and their classification prediction results in step 3), and the pseudo features of each category in all previous stages and their classification prediction results in step 5), a multi-objective loss function is defined to learn a new feature space and classifier weight; step 7), steps 3) to 6) are repeated until the training of all incremental stages is completed; step 8), all learned categories are evaluated using the trained model with average accuracy and average forgetting rate as evaluation indicators to obtain a class incremental learning model based on a dynamic class prototype generation mechanism.

[0069] Inspired by the latest findings from brain cognitive science that the topological characteristics of visual stimuli are an important component of defining visual memory, the present invention solves the problems of catastrophic forgetting and network drift in incremental learning of classes without old samples based on a dynamic class prototype generation mechanism. A class incremental learning framework MATE based on a dynamic class prototype generation mechanism is constructed by a feature extractor, a cosine classifier, DBP (feature density-based prototype generation) and TPD (topology-corrected prototype drift estimation). MATE dynamically reconstructs the pseudo-feature space of the old class to solve the problem of invisibility of old samples in class incremental learning. The initial old category prototype is calculated based on feature density, and the old category prototype is continuously updated by a topology-corrected prototype drift estimation method to solve the network drift problem in incremental learning of classes without old samples. In a typical setting on a benchmark image classification dataset, the class incremental learning method based on a dynamic class prototype generation mechanism in this application achieves state-of-the-art performance compared with existing class incremental learning methods without old samples, and has the same level of performance as the classic replay-based class incremental method. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 A flow chart of a class incremental learning method based on a dynamic class prototype generation mechanism provided by an embodiment of the present invention;

[0072] Figure 2 A three-dimensional visualization of the feature space provided by an embodiment of the present invention;

[0073] FIG3( a ) is a diagram illustrating the topology correction effect of the class incremental learning framework MATE based on the dynamic class prototype generation mechanism provided by an embodiment of the present invention;

[0074] FIG3( b ) is a diagram showing the effect of a baseline model with simple drift estimation provided by an embodiment of the present invention;

[0075] FIG4( a ) is an architecture diagram of MATE, a class incremental learning framework based on a dynamic class prototype generation mechanism, provided in an embodiment of the present invention;

[0076] FIG4( b ) is an architecture diagram of DBP in MATE, a learning framework based on a dynamic class prototype generation mechanism provided by an embodiment of the present invention;

[0077] FIG4( c ) is an architecture diagram of TPD in MATE, a learning framework based on a dynamic class prototype generation mechanism provided by an embodiment of the present invention;

[0078] Figure 5 This figure compares the test accuracy of the embodiments of the present invention with other benchmark methods on the CIFAR-100, Tiny-ImageNet, and ImageNet-100 image classification datasets.

[0079] Figure 6 A structural block diagram of a class incremental learning system based on a dynamic class prototype generation mechanism provided by an embodiment of the present invention.

[0080] Figure 7 A schematic structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0082] Driven by practical needs, research on incremental class learning has received increasing attention in recent years. Incremental class learning aims to learn new knowledge from training samples of new classes while retaining the knowledge learned from samples of old classes.

[0083] To solve the problem of catastrophic forgetting, the method based on replaying old samples is the most mainstream method, but it has two major limitations: (a) for some application scenarios with limited device memory, it is impossible to continuously store more and more categories of old samples; (b) the data privacy involved in the stored old samples is often unacceptable.

[0084] The main reasons behind catastrophic forgetting include: (a) the neural network parameters associated with old knowledge are quickly changed to adapt to new category data (i.e., network drift); (b) since category samples from different CIL stages have not been trained together, they cannot form a good decision boundary between them (i.e., inter-stage inter-class confusion).

[0085] Inspired by the latest advances in brain cognitive science, this application addresses the issues of invisibility and network drift of old class samples based on a dynamic class prototype generation mechanism. This application also proposes MATE, a class incremental learning framework based on this dynamic class prototype generation mechanism. This framework calculates prototypes of old classes based on feature density and updates them using a prototype drift estimation method. It also adds a topological correction to prevent the topological relationships between prototypes from being disrupted.

[0086] Please see Figure 1 , Figure 1 A flow chart of a class incremental learning method based on a dynamic class prototype generation mechanism provided by an embodiment of the present invention includes the following steps:

[0087] S110: Step 1) The image dataset is divided into multiple non-overlapping datasets according to categories, and the multiple non-overlapping datasets are assigned to a base class stage and T incremental stages. The training samples of each training stage are the dataset assigned to the current stage. Each stage can only use the dataset of the current stage for training, and the old data of the previous stage is invisible.

[0088] The training phase of the model is divided into 1 base class phase and T incremental phases. In the tth training phase, the dataset assigned to phase t is defined as follows:

[0089]

[0090] Among them, (x i ,y i ) is the i-th training sample and its class label, that is, x i is the i-th training sample, y i is the class label of the i-th training sample, D t The label set is defined as C t , the label set C of each stage data t They do not overlap with each other.

[0091] S120: Step 2), using the training samples of the current stage in step 1), train the feature extractor and classifier to learn the feature space and classifier weights of the current stage; wherein the training model consists of the feature extractor φ(·; θ) and the cosine classifier ψ(·; ω);

[0092] For the training sample (x i ,y i )∈Dt, the feature representation f at stage t i t The definition is as follows:

[0093] f i t =φ(x i θt ),

[0094] Among them, f i t The dimension is d f ,Will It is represented by the feature extractor φ(·;θ t ) defined feature space; then, the feature representation f i t Mapped to the label space, its predicted probability The definition is as follows:

[0095]

[0096] in, is the cosine classifier, is a trainable weight; the label space is the space composed of the extracted feature vectors; softmax is a normalized exponential function, which is used to normalize the feature vectors classified by the cosine classifier to obtain the predicted probability.

[0097] S130: Step 3), using the feature extractor to obtain feature representations of the training samples in the current stage, defining feature density, and calculating prototypes of each category in the current stage based on the feature density;

[0098] In the current stage, the feature density corresponding to each category in the training samples is defined, and the prototype of each category in the current stage is calculated based on the feature density corresponding to each category, so as to reconstruct the pseudo feature space in the subsequent incremental stage to overcome the confusion between the categories; the categories in the current stage are the categories of the training samples in the data set assigned to the current stage;

[0099] For the category a∈C at stage t t , the initial feature space The definition is as follows:

[0100]

[0101] in, is the feature representation of the training sample at stage t, The feature representation of stage t is obtained by the feature extractor, i.e. f i t =φ(x i θ t ), y i are different categories in the training samples at stage t, Including the initial feature space and pseudo-feature space Afterwards, through the initial feature space Calculate the prototype of category a;

[0102] In order to maintain the model's ability to distinguish the category a, we need to pay more attention to those features that are difficult to classify correctly, so we use feature density D to evaluate the degree of difficulty in distinguishing features of each category; feature representation The characteristic density D i The definition is as follows:

[0103]

[0104] Where E(·,·) represents the Euclidean distance, i and j represent the i-th and j-th training samples in stage t, respectively. is the feature representation of the jth training sample in the stage, γ is a hyperparameter; the density of the feature D i The smaller the value of , the more likely the feature is to be difficult to identify correctly; see Figure 2 , Figure 2 A three-dimensional visualization of the feature space provided by an embodiment of the present invention. In order to more easily identify which features the prototype generation mechanism based on feature density pays more attention to, the T-SNE method is used to visualize the weight of each feature. The darker the color, the higher the weight corresponding to the feature. Figure 2 It can be seen that most of the highly weighted features are located at the edge of the cluster, indicating that the density of features at the edge of the cluster is lower, and these features are more likely to be difficult to correctly identify; feature density can be used to accurately evaluate the degree of difficulty in distinguishing each category.

[0105] During the prototype generation process, the feature density D i Weight the features, the initial prototype of category a The definition is as follows:

[0106]

[0107] Among them, Proto(·) is the prototype of category a, It is a feature representation The weight of D j Feature Representation The feature density of the initial prototype Represents the initial prototype set in stage t, category C 1：t Prototype collection Defined as: μ j is the prototype of the old category j in the previous t-1 stages.

[0108] S140: Step 4), in each incremental stage, for the prototypes of each category in all previous stages calculated in step 3), the prototype drift estimation method based on topology correction is used to update the prototypes of each category in all previous stages;

[0109] The prototype drift estimation method based on topology correction is used to update the prototypes of each category in all previous stages, including the following steps:

[0110] In each incremental stage, the drift of the feature space in each incremental stage is calculated;

[0111] Using the drift of the feature space of each incremental stage, estimating the drift of the category prototypes of all previous stages;

[0112] Calculate the topological correction amount using the local topological relationship and update the prototypes of each category in all the previous stages;

[0113] In each incremental stage, the drift of the feature space of each incremental stage is calculated The definition is as follows:

[0114]

[0115] Among them, category a∈C t , the feature representation of stage t Feature representation at stage t-1 f i t and f i t-1 are respectively generated by the feature extractor φ(x i θ t ) and φ(x i θ t-1 ) to obtain, Δf i Feature Representation Relative to feature representation The amount of drift;

[0116] The drift of the category prototypes of all previous stages is estimated by using the drift of the category feature space of each incremental stage The definition is as follows:

[0117]

[0118] Among them, category b∈C 1：t-1 , E(·,·) represents the Euclidean distance, For category b∈C 1：t-1 The prototype, is the feature representation of the j-th training sample in stage t, and η is a hyperparameter;

[0119] The topology correction amount is calculated by using the local topological relationship, and the prototypes of each category in all previous stages are updated. The topology correction amount Defined as The weighted sum of the K nearest neighbor drifts:

[0120]

[0121] Among them, in order to maintain the increase of the prototype drift Δμ t The old category prototype set after The topological relationship of the topology is used to calculate the topological correction amount. It is the category prototype of all previous stages K nearest neighbors, It's a prototype The drift amount, α is a hyperparameter;

[0122] The prototype topology correction drift of all previous stages of category b is defined as follows:

[0123]

[0124] in, Is a category prototype The drift amount, Is a category prototype The topological correction amount, β is used to adjust the two contributions and Hyperparameters of

[0125] Update the prototype by stage to obtain the updated category prototype The definition is as follows:

[0126]

[0127] See Figure 3, in which the squares represent the categories before the update, the circles represent the categories after the update, and the arrows represent the updates of the old categories to the new categories. Figure 3 shows the prototypes of 15 categories visualized using the T-SNE method, Figure 3(a) is a topology correction effect diagram of the class incremental learning framework MATE based on the dynamic class prototype generation mechanism of the present invention, and Figure 3(b) is a rendering of the baseline model with a simple drift estimation. Taking class 1 and class 2 as examples, in the class incremental learning framework MATE based on the dynamic class prototype generation mechanism, the local topological relationship between class 1 and class 2 is maintained (see the dotted arrows), while in the baseline model with only a simple drift estimation, their local topological relationship has changed significantly. It can be seen that the class incremental learning framework MATE based on the dynamic class prototype generation mechanism of the present application can maintain the local topological relationship between the old and new categories relative to the baseline model with only a simple drift estimation.

[0128] S150: Step 5), in each incremental stage, using the prototypes of each category in all previous stages updated in step 4), pseudo features of each category in all previous stages are generated;

[0129] S160: Step 6), in each incremental stage, based on the feature representation and classification prediction results of the training samples in the current stage in step 3), and the pseudo features and classification prediction results of each category in all previous stages in step 5), a multi-objective loss function is defined to learn a new feature space and classifier weights;

[0130] The overall definition of the multi-objective loss function is as follows:

[0131]

[0132] The definition is as follows:

[0133]

[0134] The definition is as follows:

[0135]

[0136] Among them, the feature representation f of stage t i t The dimension is d f ,Will It is represented by the feature extractor φ(·;θ t ) defined feature space, To predict the label, is the cosine classifier, y i is the true label, is the cross entropy loss, is the feature distillation loss, It is a pseudo feature constraint, λ1 and λ2 are used to adjust the two loss contributions and When learning new training samples, the feature space will change. When the change is large, the incremental learning model will not be able to accurately distinguish the old categories. It can prevent the features of new training samples from changing significantly in the model, so that the features extracted from new training samples in different stages of the model remain consistent. After the pseudo features generated based on the old category prototype are classified by the classifier, it is ensured that the classifier can distinguish the pseudo features generated by the old category;

[0137] The pseudo-feature constraints The definition is as follows:

[0138]

[0139] in, yes The predicted probability, y i yes The true label of In each incremental stage, the updated category prototype is corrected by the prototype topology for drift The reconstructed pseudo feature space, pseudo feature space The definition is as follows:

[0140]

[0141] in, is the normalized pseudo feature of category b, e*σ is the Gaussian noise used to generate pseudo features, e and σ are related to have the same dimensions, Each dimension of σ represents the original feature space The standard deviation of the corresponding dimension in .

[0142] S170: Step 7), repeat steps 3) to 6) until all incremental stage training is completed;

[0143] S180: Step 8), using the trained model, with average accuracy and average forgetting rate as evaluation indicators, evaluate all categories that have been learned at each stage, and obtain a class incremental learning model based on a dynamic class prototype generation mechanism;

[0144] For example, the feature extractor and cosine classifier are trained on the dataset D in the tth training stage. t The incremental learning model is continuously trained on the basis of the training. After the training of stage t is completed, the trained incremental learning model is used to classify all the learned categories C 1: Test on.

[0145] In the above implementation process, through step 1), the image data set is divided into multiple non-overlapping data sets according to categories, and the multiple non-overlapping data sets are assigned to 1 base class stage and T incremental stages, and the training samples of each training stage are the data sets assigned to the current stage; step 2), the feature extractor and classifier are trained using the training samples of the current stage to learn the feature space and classifier weights of the current stage; step 3), the feature extractor is used to obtain the feature representation of the training samples of the current stage, the feature density is defined, and the prototypes of each category of the current stage are calculated based on the feature density; step 4), in each incremental stage, for the prototypes of each category of all previous stages calculated in step 3), the prototype drift estimation method based on topology correction is used to estimate the prototypes of the prototypes. The prototypes of each category in all previous stages are updated; step 5), in each incremental stage, the prototypes of each category in all previous stages updated in step 4) are used to generate pseudo features of each category in all previous stages; step 6), in each incremental stage, based on the feature representation of the training samples in the current stage and their classification prediction results in step 3), and the pseudo features of each category in all previous stages and their classification prediction results in step 5), a multi-objective loss function is defined to learn a new feature space and classifier weight; step 7), steps 3) to 6) are repeated until the training of all incremental stages is completed; step 8), all learned categories are evaluated using the trained model with average accuracy and average forgetting rate as evaluation indicators to obtain a class incremental learning model based on a dynamic class prototype generation mechanism.

[0146] Based on the class incremental learning method of the above-mentioned dynamic class prototype generation mechanism, the present invention proposes a learning framework MATE based on the dynamic class prototype generation mechanism. Please refer to Figure 4, Figure 4 (a) is the architecture diagram of the learning framework MATE based on the dynamic class prototype generation mechanism of the present invention, Figure 4 (b) is the architecture diagram of DBP (prototype generation based on feature density) in the learning framework MATE based on the dynamic class prototype generation mechanism, and Figure 4 (c) is the architecture diagram of TPD (topology corrected prototype drift estimation) in the learning framework MATE based on the dynamic class prototype generation mechanism. Among them, the solid origin represents the features of each category, and the solid cross represents the prototype of each category. For example, in the t-1 stage, through the old feature extractor φ(x i θ t-1 ) Extract old category C 1：t-1 Features And classify the features, generate prototypes based on feature density (DBP), define feature density according to the feature identification of the old category, and calculate the prototypes of each old category based on the feature weighting of feature density; in the t stage, through the new feature extractor φ(x i θ t ) Extract new category C t Feature representation And classify the features to construct the initial feature space Since the feature space of the old category does not match the feature space of the new category, the old category prototype is updated based on the topologically corrected prototype drift estimation (TPD) to obtain the updated old category prototype; pseudo features are generated based on the updated old category prototype Reconstructing pseudo feature space Based on the initial feature space and pseudo-feature space Constructing a new feature space Using cosine classifier Classify each category prototype in the new feature space.

[0147] Inspired by the latest brain cognitive science, the present invention solves the problems of invisibility of old class samples and network drift respectively based on the dynamic class prototype generation mechanism. The class incremental learning framework MATE based on the dynamic class prototype generation mechanism is composed of feature extractors, cosine classifiers, DBP (feature density-based prototype generation) and TPD (topology-corrected prototype drift estimation). MATE dynamically reconstructs the pseudo-feature space of the old class to solve the catastrophic forgetting problem in incremental learning of classes without old samples. The initial old category prototype is calculated based on feature density, and the old category prototype is continuously updated by the prototype drift estimation method based on topology correction to solve the network drift problem in incremental learning of classes without old samples. In a typical setting on a benchmark image classification dataset, the class incremental learning method based on the dynamic class prototype generation mechanism in this application achieves state-of-the-art performance compared with the existing class incremental learning method without old samples, and has the same level of performance as the classic replay-based class incremental method.

[0148] Please see Figure 5 , Figure 5 This figure compares the test accuracy of the embodiments of the present invention with other benchmark methods on the CIFAR-100, Tiny-ImageNet, and ImageNet-100 image classification datasets.

[0149] Among them, the compared benchmark methods include: iCaRL, a method that introduces the rule of the closest representative sample mean into this field; EEiL, a method that uses balanced training and cross-distillation loss function; LUCIR, a method that uses cosine normalization, little forgetting constraint and inter-class separation to alleviate the problem of catastrophic forgetting; MUC, a method that uses a multi-classifier paradigm; LwF, a method that uses knowledge distillation loss function to retain old knowledge; PASS, a method that memorizes representative prototypes for old classes and uses self-supervised learning; SSRE, a method that uses a main branch expansion and side branch update strategy and a main branch distillation scheme to transfer invariant knowledge. Figure 5In the figure, (a) is a line graph comparing the test accuracy of the method and other baseline methods based on the 5-stage incremental setting on the CIFAR-100 dataset; (b) is a line graph comparing the test accuracy of the method and other baseline methods based on the 10-stage incremental setting on the CIFAR-100 dataset; (c) is a line graph comparing the test accuracy of the method and other baseline methods based on the 20-stage incremental setting on the CIFAR-100 dataset; (d) is a line graph comparing the test accuracy of the method and other baseline methods based on the 5-stage incremental setting on the Tiny-ImageNet dataset; (e) is a line graph comparing the test accuracy of the method and other baseline methods based on the 5-stage incremental setting on the Tiny-ImageNet dataset. (f) is a line chart comparing the test accuracy of this method with other baseline methods on the Tiny-ImageNet dataset under the 10-stage incremental setting; (g) is a line chart comparing the test accuracy of this method with other baseline methods on the ImageNet-100 dataset under the 5-stage incremental setting; (h) is a line chart comparing the test accuracy of this method with other baseline methods on the ImageNet-100 dataset under the 10-stage incremental setting. Figure 5 It can be seen from the data in that the class incremental learning method based on dynamic class prototype generation mechanism in this application has achieved the most advanced performance compared with the existing class incremental learning method without old samples, and has the same level of performance as the classic replay-based class incremental method.

[0150] Based on the same inventive concept, the present invention also proposes an image classification method of a class incremental learning model based on a dynamic class prototype generation mechanism, wherein the image to be classified is input into the class incremental learning model based on a dynamic class prototype generation mechanism to obtain the image classification result of the image to be classified.

[0151] Based on the same inventive concept, the present invention also proposes a class incremental learning system based on a dynamic class prototype generation mechanism, please refer to Figure 6 , Figure 6 The structural block diagram of a class incremental learning system based on a dynamic class prototype generation mechanism provided by an embodiment of the present invention includes:

[0152] The class incremental dataset partitioning module 110 is used in step 1) to partition the image dataset into multiple non-overlapping datasets according to categories, and assign the multiple non-overlapping datasets to one base class stage and T incremental stages, with the training samples of each training stage being the dataset assigned to the current stage;

[0153] The feature space extraction and classification module 120 is used in step 2) to train the feature extractor and classifier using the training samples of the current stage in step 1) to learn the feature space and classifier weights of the current stage;

[0154] A prototype calculation module 130, used in step 3), uses the feature extractor to obtain feature representations of the training samples in the current stage, defines feature density, and calculates prototypes of each category in the current stage based on the feature density;

[0155] A topology correction module 140 is used in step 4) to update the prototypes of each category in all previous stages calculated in step 3) in each incremental stage using a prototype drift estimation method based on topology correction;

[0156] The pseudo feature generation module 150 is used in step 5) to generate pseudo features for each category in all previous stages in each incremental stage using the prototypes of each category in all previous stages updated in step 4);

[0157] A multi-objective loss function calculation module 160 is used in step 6) to define a multi-objective loss function in each incremental stage based on the feature representation and classification prediction results of the training samples in the current stage in step 3) and the pseudo features of each category and their classification prediction results in all previous stages in step 5) to learn a new feature space and classifier weights;

[0158] The incremental learning model training module 170 is used in step 7) to repeat steps 3) to 6) until all incremental stage training is completed;

[0159] The class incremental learning model evaluation module 180 is used in step 8) to use the trained model to evaluate all learned categories with average accuracy and average forgetting rate as evaluation indicators to obtain a class incremental learning model based on a dynamic class prototype generation mechanism.

[0160] In the above implementation process, the class incremental dataset partitioning module 110 is used in step 1) to divide the image dataset into multiple non-overlapping datasets according to categories, and the multiple non-overlapping datasets are assigned to 1 base class stage and incremental stages, and the training samples of each training stage are the datasets assigned to the current stage; the feature space extraction and classification module 120 is used in step 2) to use the training samples of the current stage in step 1) to train the feature extractor and classifier, and learn the feature space and classifier weights of the current stage; the prototype calculation module 130 is used in step 3) to obtain the feature representation of the training samples of the current stage using the feature extractor, define the feature density, and calculate the prototypes of each category in the current stage based on the feature density; the topology correction module 140 is used in step 4) in each incremental stage, for the prototypes of each category of all previous stages calculated in step 3), the prototype drift estimation method based on topology correction is used to estimate the previous prototypes. The prototypes of each category in all stages are updated; the pseudo-feature generation module 150 is used in step 5) to generate pseudo-features for each category in all previous stages in each incremental stage using the prototypes of each category in all previous stages updated in step 4). The multi-objective loss function calculation module 160 is used in step 6) to define a multi-objective loss function in each incremental stage based on the feature representations and classification prediction results of the training samples in the current stage in step 3) and the pseudo-features and classification prediction results of each category in all previous stages in step 5), and learn a new feature space and classifier weights. The class incremental learning model training module 170 is used in step 7) to repeat steps 3) to 6) until all incremental stages are trained. The class incremental learning model evaluation module 180 is used in step 8) to evaluate all learned categories using the trained model using average accuracy and average forgetting rate as evaluation indicators, thereby obtaining a class incremental learning model based on a dynamic class prototype generation mechanism. Inspired by the recent findings in brain cognitive science that the topological characteristics of visual stimuli are an important component of defining visual memory, the present invention, based on a dynamic class prototype generation mechanism, solves the problems of catastrophic forgetting and network drift in class incremental learning without old samples. MATE, a class incremental learning framework based on a dynamic class prototype generation mechanism, is constructed through a feature extractor, a cosine classifier, DBP (feature density-based prototype generation), and TPD (topology-corrected prototype drift estimation). MATE dynamically reconstructs the pseudo-feature space of old classes to address the issue of old samples being invisible in class incremental learning. Initial old class prototypes are calculated based on feature density and continuously updated using a topology-corrected prototype drift estimation method to address the network drift issue in class incremental learning without old samples.In a typical setting on a benchmark image classification dataset, the class incremental learning method based on a dynamic class prototype generation mechanism in this application achieves state-of-the-art performance compared with existing class incremental learning methods without old samples, and has the same level of performance as the classic replay-based class incremental method.

[0161] See also Figure 7 , Figure 7 A schematic structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device includes a memory 101, a processor 102 and a communication interface 103, and the memory 101, the processor 102 and the communication interface 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, such as the program instructions / modules corresponding to the class incremental learning system based on the dynamic class prototype generation mechanism provided in an embodiment of the present application, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0162] Among them, the memory 101 can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0163] The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0164] I understand. Figure 7 The structure shown is only for illustration, and the electronic device may also include Figure 7 More or fewer components than shown, or with Figure 7 Different configurations shown. Figure 7 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0165] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0166] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0167] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0168] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0169] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A class incremental learning method based on a dynamic class prototype generation mechanism, characterized in that: The following steps are involved: Step 1) Divide the image dataset into multiple non-overlapping datasets according to categories, and assign the multiple non-overlapping datasets to a base class stage and Incremental stages, the training samples of each training stage are the data sets assigned to the current stage; Step 2) Use the training samples of the current stage in step 1) to train the feature extractor and classifier to learn the feature space and classifier weights of the current stage; Step 3) using the feature extractor to obtain the feature representation of the training samples in the current stage, defining the feature density, and calculating the prototypes of each category in the current stage based on the feature density; Step 4) In each incremental stage, for the prototypes of each category in all previous stages calculated in step 3), calculate the drift of the feature space in each incremental stage ; The drift of the feature space of each incremental stage is used to estimate the drift of the category prototypes of all previous stages. ; Calculate the topological correction amount using the local topological relationship and update the prototypes of each category in all previous stages; Step 5) In each incremental stage, the prototypes of each category in all previous stages updated in step 4) are used to generate pseudo features of each category in all previous stages; Step 6) In each incremental stage, based on the feature representation and classification prediction results of the training samples in the current stage in step 3), and the pseudo features and classification prediction results of each category in all previous stages in step 5), a multi-objective loss function is defined to learn a new feature space and classifier weights; Step 7) Repeat steps 3) to 6) until all incremental training stages are completed; Step 8) Use the trained model to evaluate all learned categories with average accuracy and average forgetting rate as evaluation indicators, and obtain a class incremental learning model based on the dynamic class prototype generation mechanism; in, The definition is as follows: ， Among them, the category ,stage Feature representation ,stage t Feature representation , and are respectively generated by the feature extractor and get, Feature Representation Relative to feature representation The amount of drift; The definition is as follows: ， Among them, the category , represents the Euclidean distance, The prototype, For the stage No. The feature representation of the training samples, is a hyperparameter; Topology correction amount Defined as The weighted sum of the K nearest neighbor drifts: ， in, , It is the category prototype of all previous stages K nearest neighbors, Is a category prototype The drift amount, is a hyperparameter; All previous stage categories The prototype topology corrects the drift The definition is as follows: in, yes It is used to adjust and Hyperparameters of Updated category prototype , defined as follows: 。 2. A class incremental learning method based on a dynamic class prototype generation mechanism according to claim 1, characterized in that: In step 3), the feature density is defined, and the prototypes of each category in the current stage are calculated based on the feature density, including: For the stage Category , the initial feature space The definition is as follows: ， in, For the stage Feature representation of training samples, , For the stage Different categories in the training samples, , is the new feature space; Feature Representation The characteristic density The definition is as follows: ， in, represents the Euclidean distance, Represents the stages , For the stage Middle The feature representation of the training samples, is a hyperparameter; During prototyping, feature density For categories The features are weighted, and the categories Initial prototype The definition is as follows: ， in, It is a feature representation The weight of , Feature Representation The feature density of the initial prototype , Representative stage The initial prototype set in category Prototype collection Defined as: , , For the front Old categories in stages prototype.

3. The class incremental learning method based on dynamic class prototype generation mechanism according to claim 1 is characterized in that: In step 6), the multi-objective loss function is defined as follows: ， The definition is as follows: ， The definition is as follows: Among them, the stage Feature representation The dimension is ,Will Represented by the feature extractor The feature space defined, is the true label, is the cross entropy loss, is the characteristic distillation loss, is a pseudo-feature constraint, and It is used to adjust the two loss contributions and hyperparameters.

4. The class incremental learning method based on dynamic class prototype generation mechanism according to claim 3 is characterized in that: The pseudo-feature constraints The definition is as follows: ， in, , yes The predicted probability of yes The true label of In each incremental stage, the updated category prototype is corrected by the prototype topology for drift The reconstructed pseudo feature space, pseudo feature space The definition is as follows: ， in, is a category The normalized pseudo features of is the Gaussian noise used to generate pseudo features, and have the same dimensions, , Each dimension represents the original feature space The standard deviation of the corresponding dimension in .

5. An image classification method based on a class incremental learning model with a dynamic class prototype generation mechanism, characterized in that: The image to be classified is input into a class incremental learning model based on a dynamic class prototype generation mechanism to obtain an image classification result of the image to be classified. The class incremental learning model based on a dynamic class prototype generation mechanism is obtained by the class incremental learning method based on a dynamic class prototype generation mechanism as described in any one of claims 1 to 4.

6. A class incremental learning system based on a dynamic class prototype generation mechanism, characterized in that: include: The class incremental dataset partitioning module is used in step 1 to divide the image dataset into multiple non-overlapping datasets according to categories, and assign the multiple non-overlapping datasets to a base class stage and Incremental stages, the training samples of each training stage are the data sets assigned to the current stage; Feature space extraction and classification module, used in step 2), uses the training samples of the current stage in step 1) to train the feature extractor and classifier, and learns the feature space and classifier weights of the current stage; A prototype calculation module, used in step 3), uses the feature extractor to obtain the feature representation of the training samples in the current stage, defines the feature density, and calculates the prototype of each category in the current stage based on the feature density; The topology correction module is used in step 4. In each incremental stage, for the prototypes of each category in all previous stages calculated in step 3), the drift of the feature space of each incremental stage is calculated. ; The drift of the feature space of each incremental stage is used to estimate the drift of the category prototypes of all previous stages. ; Calculate the topological correction amount using the local topological relationship and update the prototypes of each category in all previous stages; The pseudo-feature generation module, used in step 5), generates pseudo-features for each category in all previous stages in each incremental stage using the prototypes of each category in all previous stages updated in step 4); The multi-objective loss function calculation module, used in step 6), defines a multi-objective loss function in each incremental stage based on the feature representation and classification prediction results of the training samples in the current stage in step 3), as well as the pseudo features of each category and their classification prediction results in all previous stages in step 5), and learns a new feature space and classifier weights. Class incremental learning model training module, used in step 7), repeat steps 3) to 6) until all incremental stage training is completed; The class incremental learning model evaluation module, used in step 8, uses the trained model to evaluate all learned categories using average accuracy and average forgetting rate as evaluation indicators to obtain a class incremental learning model based on the dynamic class prototype generation mechanism; in, The definition is as follows: ， Among them, the category ,stage Feature representation ,stage t Feature representation , and are respectively generated by the feature extractor and get, Feature Representation Relative to feature representation The amount of drift; The definition is as follows: ， Among them, the category , represents the Euclidean distance, For the stage No. The feature representation of the training samples, is a hyperparameter; Topology correction amount Defined as The weighted sum of the K nearest neighbor drifts: ， in, , It is the category prototype of all previous stages K nearest neighbors, Is a category prototype The drift amount, is a hyperparameter; All previous stage categories The prototype topology corrects the drift The definition is as follows: in, yes It is used to adjust and Hyperparameters of Updated category prototype , defined as follows: 。 7. An electronic device, characterized in that: include: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images