A Semantic Knowledge-Guided Class-Incremental Learning Method and System

Abstract

The present invention proposes a semi - incremental learning method and system guided by semantic knowledge, which relates to the field of artificial intelligence technology. A semi - incremental learning framework SOUL is composed of a CNN feature extractor and a Bi - GCN classifier. This framework uses semantic knowledge extracted from class labels to construct an inter - class relationship graph and learn the Bi - GCN classifier. Based on the Bi - GCN classifier, the inter - class relationship is transferred from the semantic modality to the image classifier weights to solve the inter - class confusion problem. A local topology preservation constraint is designed to divide the global topology relationship of the learned feature space into a set of local topology relationships and maintain these local relationships in each semi - incremental learning stage to prevent the topology relationship of the learned feature space from being destroyed. By combining the local topology preservation strategy and the SOUL framework, the semi - incremental learning method of the present invention achieves state - of - the - art performance accuracy in typical settings on benchmark image classification datasets.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to a class-incremental learning method and system guided by semantic knowledge. Background Art

[0002] Since neural network models are often deployed in a constantly changing dynamic environment, the models are required to continuously learn new knowledge and adapt to new image categories. Driven by practical needs, in recent years, the research on class-incremental learning has received increasing attention. It attempts to learn new knowledge from new class training instances and retain the learned knowledge of old classes. In class-incremental learning, the training data of new classes are provided in stages. While the new training data are being trained, the classifier is extended, and the old and new training data are evaluated together at each stage. A simple method for class-incremental learning is to fine-tune the base network model using the new class training data. However, this method will cause the problem of catastrophic forgetting, that is, the performance of the model improves rapidly on new classes while deteriorating sharply on old classes. The main reasons behind catastrophic forgetting include: 1) network drift, the superior data fitting ability of neural networks causes the feature space quickly learned from the old class training data to drift to the feature space of new class data; 2) inter-class confusion, the boundaries between old and new classes are not well established because they have never been trained together.

[0003] Some existing class-incremental learning methods prevent inter-class confusion by introducing replay-based methods. However, when many samples of new classes but few or no samples of old classes are used in the class-incremental learning stage, it will lead to new class bias (data imbalance). There are also some methods that adopt the way of expanding and then pruning the incremental learning model to improve the effect of class-incremental learning. However, these methods are limited by the large number of parameters and high computational complexity. Summary of the Invention

[0004] The purpose of the present invention is to provide a class-incremental learning method and system guided by semantic knowledge to improve the problem in the prior art that when many samples of new classes but few or no samples of old classes are used in the class-incremental learning stage, it will lead to new class bias (data imbalance). There are also some methods that adopt the way of expanding and then pruning the incremental learning model to improve the effect of class-incremental learning. However, these methods are limited by the large number of parameters and high computational complexity.

[0005] In a first aspect, an embodiment of the present application provides a class-incremental learning method guided by semantic knowledge, including the following steps:

[0006] Obtain a class-incremental data set containing multiple categories, and divide the class-incremental data set into training samples of multiple training stages according to categories, and ensure that the categories in the training samples of different training stages do not overlap;

[0007] The feature extractor is used to extract features from the training samples in the different training stages respectively and train them, and feature spaces in different stages are constructed;

[0008] According to the class labels in the class incremental dataset, semantic embeddings of the class labels in different training stages are obtained, and an inter-class relationship graph for each stage is constructed respectively according to the semantic embeddings of the class labels in the different training stages;

[0009] According to the inter-class relationship graphs for each stage, a classifier is designed, and classification results of the training samples in the different training stages are calculated;

[0010] The feature spaces in the different stages are synchronously maintained with local similarity relationships by adopting a local topology preservation strategy;

[0011] A multi-objective loss function is defined, and the class incremental learning model is trained by using the local topology preservation constraint of the local similarity relationship to obtain a trained model.

[0012] Based on the first aspect, in some embodiments of the present invention, the step of obtaining semantic embeddings of the class labels in different training stages according to the class labels in the class incremental dataset, and constructing an inter-class relationship graph for each stage according to the semantic embeddings of the class labels in the different training stages includes the following steps:

[0013] According to the class labels in the class incremental dataset, the Word2Vec algorithm is respectively used to obtain semantic embeddings of the class labels in the current training stage;

[0014] The semantic embeddings of the class labels in each training stage are used to construct an inter-class relationship graph for each stage, wherein the nodes of the inter-class relationship graph represent all the classes learned in the current stage, and the edge between any two of the nodes represents the semantic relationship between classes.

[0015] Based on the first aspect, in some embodiments of the present invention, the semantic relationship between classes includes a general relationship and a core relationship. For the inter-class relationship graph, two symmetric adjacency matrices are defined, respectively representing the general relationship and the core relationship between all the classes learned in stage k:

[0016]

[0017]

[0018] Among them, N k is the number of all the classes learned in stage k, dist(·, ·) is the Euclidean distance, l i and l jThey are the semantic embeddings of class labels i and j respectively, η represents a hyperparameter, and P(·) represents a pruning operation for pruning weak edges and retaining the core semantic relationships in the inter-class relationship graph.

[0019] Based on the first aspect, in some embodiments of the present invention, the classifier is a Bi-GCN classifier, and the Bi-GCN classifier includes two GCN propagation functions to and the corresponding semantic embedding matrix as the inputs of the two GCN propagation functions, and the outputs of the two GCN propagation functions and as two sets of classifier weights, and the Bi-GCN classifier ψ is defined as:

[0020]

[0021] where, is the estimated likelihood value that a given sample x i belongs to class c at stage k, ρ is a hyperparameter controlling the contributions of the two GCN classifiers, corresponding to two sets of classification weights for class i respectively.

[0022] Based on the first aspect, in some embodiments of the present invention, the strategy of synchronously maintaining local similarity relationships in the feature spaces of different stages by using a local topology preservation strategy includes the following steps:

[0023] Calculate the similarity matrices of the same batch of training samples in the current stage k and the previous stage k - 1, and obtain the similarity matrix in the current stage k as S k , and the similarity matrix in the previous stage k - 1 as S k-1 ;

[0024] Use a sliding window to divide S k and S k-1 into local similarity sub-matrices S k (i) and S k-1 (i), where when the size m of the sliding window is greater than the step size n of the sliding window, the sum of the local similarity relationships in all the local similarity sub-matrices maintains the global similarity relationship;

[0025] Normalize the local similarity sub-matrices S k (i) and S k-1 (i) respectively to obtain normalized sub-matrices;

[0026] According to the normalized sub-matrices, use the mean absolute error loss function to synchronously maintain local similarity relationships.

[0027] Based on the first aspect, in some embodiments of the present invention, the multi-objective loss function includes a classification cross-entropy loss function, a knowledge distillation loss function for penalizing the change in output probabilities during the adjacent class incremental learning stage, and a local topology preservation constraint; the overall loss function in stage k is expressed as:

[0028]

[0029] where |C o | and |C n | respectively represent the number of old classes and new classes in the current class incremental learning stage, is the classification cross-entropy loss function; is the knowledge distillation loss function for penalizing the change in output probabilities during the adjacent class incremental learning stage; is the local topology preservation constraint, and α, γ, and δ represent hyperparameters for balancing the contributions of the three loss functions.

[0030] Based on the first aspect, in some embodiments of the present invention, the local topology preservation constraint is defined as:

[0031]

[0032] where and represent normalized submatrices, L1(·, ·) is the mean absolute error loss function, and Id is the number of submatrices calculated by the sliding window.

[0033] In a second aspect, an embodiment of the present application provides a class incremental learning system guided by semantic knowledge, including:

[0034] A class incremental dataset partitioning module for obtaining a class incremental dataset containing multiple categories, and dividing the class incremental dataset into training samples for multiple training stages by category, and ensuring that there is no overlap between the categories in the training samples of different training stages;

[0035] A feature space extraction module for using a feature extractor to extract and train features from the training samples of different training stages respectively, and constructing feature spaces for different stages;

[0036] A class relationship graph construction module for obtaining semantic embeddings of class labels in different training stages according to the class labels in the class incremental dataset, and constructing class relationship graphs for each stage according to the semantic embeddings of class labels in different training stages;

[0037] A classification module for designing a classifier according to the class relationship graphs of each stage, and calculating classification results of the training samples of different training stages;

[0038] A local topology preservation module, which is used to adopt a local topology preservation strategy for the feature spaces of different stages to synchronously preserve local similarity relationships;

[0039] A class-incremental learning model training module, which is used to define a multi-objective loss function and train a class-incremental learning model by using the local topology preservation constraint of the local similarity relationship to obtain a trained model.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory for storing one or more programs; a processor. When the one or more programs are executed by the processor, the method described in any one of the first aspects above is implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any one of the first aspects above is implemented.

[0042] The embodiment of the present invention has at least the following advantages or beneficial effects:

[0043] The embodiment of the present invention provides a class-incremental learning method and system guided by semantic knowledge. By obtaining a class-incremental data set including multiple categories, and dividing the class-incremental data set into training samples of multiple training stages according to categories, and ensuring that the categories in the training samples of different training stages do not overlap; then using a feature extractor to learn the feature space of the training samples in different stages; using the semantic embedding of the class labels learned in the current stage to construct an inter-class relationship graph for each stage; then designing a classifier to calculate the possibility that the feature belongs to different categories in different stages, and at the same time proposing a local topology preservation strategy to synchronously preserve local similarity relationships; and defining a multi-objective loss function, and jointly optimizing the training task with the local topology preservation constraint; continuously training the model using the feature extractor and the classifier. After completion, the model tests all the learned categories to evaluate the model.

[0044] Inspired by the latest developments in brain cognitive science, the present invention is based on semantic knowledge guidance and combines the ideas of regularization-based methods and replay-based methods to solve the network drift problem and the inter-class confusion problem respectively. By constructing a class-incremental learning framework SOUL consisting of a CNN feature extractor and a Bi-GCN classifier, the framework uses semantic knowledge extracted from class labels to build an inter-class relationship graph and learn the Bi-GCN classifier. Based on the Bi-GCN classifier, the inter-class relationship is transferred from the semantic modality to the image classifier weights to solve the inter-class confusion problem. At the same time, a local topology preservation constraint is designed to divide the global topological relationship of the learned feature space into a set of local topological relationships and maintain these local relationships in each class-incremental learning stage to prevent the topological relationship of the learned feature space from being destroyed. By combining the local topology preservation strategy and the SOUL framework, the class-incremental learning method of the present invention achieves state-of-the-art performance accuracy in typical settings on benchmark image classification datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0046] Figure 1 Flowchart of a semantic-knowledge-guided class-incremental learning method provided by an embodiment of the present invention;

[0047] Figure 2 Architecture diagram of the semantic-knowledge-guided class-incremental learning framework SOUL provided by an embodiment of the present invention;

[0048] Figure 3 Illustration diagram of the local topology preservation constraint provided by an embodiment of the present invention;

[0049] Figure 4 Comparison of test accuracies of the present invention with other benchmark methods on the CIFAR-100, ImageNet-100, and ImageNet-1000 image classification datasets provided by an embodiment of the present invention;

[0050] Figure 5 Block diagram of the structure of a semantic-knowledge-guided class-incremental learning system provided by an embodiment of the present invention;

[0051] Figure 6 Block diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0052] Icons: 110 - Class Incremental Dataset Partitioning Module; 120 - Feature Space Extraction Module; 130 - Inter - class Relationship Graph Construction Module; 140 - Classification Module; 150 - Local Topology Preservation Module; 160 - Class Incremental Learning Model Training Module; 170 - Model Testing Module; 101 - Memory; 102 - Processor; 103 - Communication Interface. Detailed Implementation Manner

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Usually, the components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0055] The following will describe in detail some embodiments of the present application with reference to the accompanying drawings. Without conflict, the following various embodiments and the various features in the embodiments can be combined with each other.

[0056] Driven by real - world requirements, in recent years, the research on class incremental learning has received increasing attention. Class incremental learning attempts to learn new knowledge from new class training instances and retain the learned knowledge of old classes.

[0057] In class incremental learning, the training data of new classes is provided in stages. While the new training data is being trained, the classifier is extended, and the old and new training data are evaluated together at each stage. A simple method of class incremental learning is to fine - tune the base network model using the new class training data. However, this method will lead to the problem of catastrophic forgetting, that is, the performance of the model improves rapidly on new classes while deteriorating sharply on old classes. The main reasons behind catastrophic forgetting include: 1) Network drift. The superior data fitting ability of neural networks causes the feature space quickly learned from old - class training data to drift to the feature space of new - class data; 2) Inter - class confusion. The boundaries between old and new classes are not well established because they have never been trained together.

[0058] Inspired by the latest developments in brain cognitive science, the present invention is based on semantic knowledge guidance and combines the ideas of regularization-based methods and replay-based methods to solve the network drift problem and the inter-class confusion problem respectively. We propose SOUL, which consists of a CNN feature extractor and a Bi-GCN classifier. Based on the Bi-GCN classifier, the inter-class relationship is transferred from the semantic modality to the weights of the image classifier to solve the inter-class confusion problem. At the same time, we design a local topology preservation constraint to prevent the topological relationship of the learned feature space from being destroyed.

[0059] Please refer to Figures 1 - 3 , Figure 1 which is a flowchart of a semantic knowledge-guided class-incremental learning method provided by an embodiment of the present invention, Figure 2 which is an architecture diagram of the semantic knowledge-guided class-incremental learning framework SOUL provided by an embodiment of the present invention, Figure 3 which is an explanatory diagram of the local topology preservation constraint provided by an embodiment of the present invention.

[0060] The semantic knowledge-guided class-incremental learning method includes the following steps:

[0061] Step S110: Obtain a class-incremental data set containing multiple categories, and divide the class-incremental data set into training samples of multiple training stages according to categories, and ensure that the categories in the training samples of different training stages do not overlap; the class-incremental data set includes data of multiple categories, and each category of data corresponds to a class label.

[0062] Step S120: Use a feature extractor to extract features from the training samples of different training stages and train them respectively to construct feature spaces of different stages; the feature extractor is also constantly learning during the training process of different stages of the model, learning the feature spaces of different stages. The feature space is found from the above feature extraction that extracting features from the original data is to map the original data to a higher-dimensional space, and the features in the feature space are a higher-dimensional abstraction of the original data.

[0063] The above feature extractor can be represented as a feature extractor φ with parameters θ k For a training sample (x i , y i ), its feature representation in stage k is defined as: where has a dimension of d f , and is represented as the feature space learned by the feature extractor φ(·; θ k ), where x i represents the sample pattern, and y i represents the label category.

[0064] Step S130: Obtain the semantic embeddings of class labels at different training stages according to the class labels in the class incremental dataset, and construct an inter-class relationship graph for each stage according to the semantic embeddings of the class labels at different training stages; semantic embeddings can convert class labels into vectors, which is convenient for constructing the inter-class relationship graph for each stage.

[0065] Among them, the steps of constructing the inter-class relationship graph for each stage may include the following steps:

[0066] First, according to the class labels in the class incremental dataset, use the Word2Vec algorithm to obtain the semantic embeddings of the class labels in the current training stage respectively; the Word2Vec algorithm is simple and efficient, and is especially suitable for obtaining high-precision word vector representations from large-scale and ultra-large-scale corpora. The vectorization of words is to mathematize the words in the language, that is, to represent a word as a vector.

[0067] Then, use the semantic embeddings of the class labels in each training stage to construct an inter-class relationship graph for each stage. Among them, the nodes of the inter-class relationship graph represent all the classes learned in the current stage, and the edge between any two nodes represents the semantic relationship between classes. Specifically, use the Word2Vec algorithm to obtain the semantic embeddings of all the class labels learned in the current stage k, and use them to construct an inter-class relationship graph for each stage k. 。 is an undirected graph, where the nodes represent all the classes learned in the current stage, and the edge between any two nodes represents the semantic relationship between classes. is a subgraph of, which means that the inter-class relationship in the current stage k will be included in the next stage k + 1.

[0068] Among them, the semantic relationship between classes includes a general relationship and a core relationship. For the inter-class relationship graph, two symmetric adjacency matrices are defined, respectively representing the general relationship and the core relationship between all the classes learned in stage k:

[0069]

[0070]

[0071] Among them, N k is the number of all the classes learned in stage k, dist(·, ·) is the Euclidean distance, l i and l j are the semantic embeddings of class labels i and j respectively, η represents a hyperparameter, and P(·) represents a pruning operation, which is used to prune weak edges and retain the core semantic relationship in the inter-class relationship graph.

[0072] Step S140: Design a classifier according to the inter-class relationship graphs of the respective stages, and calculate the classification results of the training samples in the different training stages; the classification results of the training samples in the different training stages represent the probabilities that the features belong to different classes in different stages.

[0073] The classifier can be a Bi-GCN classifier ψ, and the classifier ψ consists of two GCN propagation functions which take and the corresponding semantic embedding matrix as inputs and calculate the next layer as follows:

[0074]

[0075] where d0 is the dimension of the semantic embedding l i For

[0076]

[0077] where is a diagonal matrix with diagonal elements , is a trainable weight matrix, I is the identity matrix, and βI is a self-enhancing term to alleviate the over-smoothing problem. Similarly, for

[0078]

[0079] where

[0080] Assume that the two GCN propagation functions produce the final outputs:

[0081] Then and can be regarded as two sets of classifier weights, where correspond to the two sets of classification weights for class i respectively.

[0082] Therefore, the Bi-GCN classifier ψ can be defined as:

[0083]

[0084] where is the estimated likelihood value that the given sample x i belongs to class c at stage k, ρ is a hyperparameter that controls the contributions of the two GCN classifiers, is the feature representation at stage k.

[0085] Step S150: Adopt a local topology preservation strategy for the feature spaces in different stages to synchronously preserve local similarity relationships; specifically, it includes the following steps:

[0086] First, calculate the similarity matrices of the same batch of training samples in the current stage k and the previous stage k - 1, and obtain the similarity matrix for the current stage k as S k , and the similarity matrix in the previous stage k - 1 as S k-1 ; The elements of S k and S k-1 are calculated as follows:

[0087]

[0088]

[0089] where · is the vector inner product, S k , b is the training batch size.

[0090] Then, use a sliding window to divide S k and S k-1 into local similarity sub - matrices S k (i) and S k-1 (i). Among them, when the size m of the sliding window is greater than the step size n of the sliding window, the sum of the local similarity relationships in all the local similarity sub - matrices maintains the global similarity relationship;

[0091] Then, normalize the local similarity sub - matrices S k (i) and S k-1 (i) respectively to obtain normalized sub - matrices; Normalization can enhance the similarity difference between samples in local relationships, thereby reducing the optimization difficulty.

[0092] Finally, according to the normalized sub - matrices, adopt the mean absolute error loss function to synchronously preserve local similarity relationships. Among them, the local topology preservation constraint is defined as:

[0093]

[0094] where and represent the normalized sub - matrices, L1(·, ·) is the mean absolute error loss function, and Id is the number of sub - matrices calculated by the sliding window.

[0095] Step S160: Define a multi - objective loss function, and use the local topology preservation constraint of the local similarity relationship to train the class - incremental learning model to obtain a trained model.

[0096] Among them, the multi-objective loss function includes the cross-entropy loss function for classification, the knowledge distillation loss function that penalizes the change in output probability in the incremental learning stage of adjacent classes, and the local topology preservation constraint;

[0097] The overall loss function in stage k is expressed as:

[0098]

[0099] Among them, |C o | and |C n | represent the numbers of old classes and new classes in the current class incremental learning stage, respectively. is the cross-entropy loss function for classification; is the knowledge distillation loss function that penalizes the change in output probability in the incremental learning stage of adjacent classes; is the local topology preservation constraint, and α, γ, and δ represent hyperparameters that balance the contributions of the three loss functions. The above different loss functions adopt adaptive weights to achieve a dynamic trade-off between plasticity and stability.

[0100] The above cross-entropy loss function for classification can be expressed as:

[0101]

[0102] Among them, σ represents the softmax function, is the estimated likelihood vector corresponding to x i , x i represents the sample pattern, and y i represents the label category.

[0103] The above knowledge distillation loss function that penalizes the change in output probability in the incremental learning stage of adjacent classes can be expressed as:

[0104]

[0105] Among them, τ represents the distillation temperature value, respectively represent the output probabilities of the c-th class in the current and previous stages k and k - 1, x i represents the sample pattern, and y i represents the label category.

[0106] The above and can adopt adaptive weights to achieve a dynamic balance between plasticity and stability.

[0107] It should be noted that the above steps S110 - S160 are used to identify each step and do not represent the logical sequence of the steps. In actual use, the order of the steps can be arbitrarily exchanged, and it does not play a role in limiting the order here. Among them, steps S120 - S140 are for formulating the model structure, and steps S150 - S160 are for training the model. Generally speaking, steps S120 - S140 and steps S150 - S160 can be processed in parallel to quickly obtain the class incremental learning model. Step S170: Use the trained model to test all the learned classes. The above test process can be: continuously train the model using the feature extractor and the Bi - GCN classifier on the joint training set After completing the training of the joint training set, further fine - tune the model on the balanced sample set E 1 ∪…∪E k-1 ∪E k This sample set is taken from all the learned classes The model is tested on all the learned classes

[0108] In the above implementation process, by obtaining a class incremental dataset containing multiple classes, and dividing the class incremental dataset into training samples of multiple training stages by category, and ensuring that the classes in the training samples of different training stages do not overlap; then using the feature extractor to learn the feature space of the training samples in different stages; using the semantic embedding of the class labels learned in the current stage to construct an inter - class relationship graph for each stage; then designing a classifier to calculate the possibility that the feature belongs to different classes in different stages, and at the same time proposing a local topology preservation strategy to synchronously maintain the local similarity relationship; and defining a multi - objective loss function, and jointly optimizing the training task with the local topology preservation constraint; continuously training the model using the feature extractor and the classifier, and after completion, testing the model on all the learned classes to evaluate the model.

[0109] ​Inspired by the latest developments in brain cognitive science, the present invention is based on semantic knowledge guidance, combining the ideas of regularization-based methods and replay-based methods to solve the network drift problem and the inter-class confusion problem respectively. Through a class-incremental learning framework SOUL composed of a CNN feature extractor and a Bi-GCN classifier, the framework uses semantic knowledge extracted from class labels to construct an inter-class relationship graph and learn the Bi-GCN classifier. Based on the Bi-GCN classifier, the inter-class relationship is transferred from the semantic modality to the image classifier weights to solve the inter-class confusion problem. At the same time, a local topology preservation constraint is designed to divide the global topological relationship of the learned feature space into a set of local topological relationships and maintain these local relationships in each class-incremental learning stage to prevent the topological relationship of the learned feature space from being destroyed. By combining the local topology preservation strategy and the SOUL framework, the class-incremental learning method of the present invention achieves state-of-the-art performance accuracy in typical settings on benchmark image classification datasets.

[0110] Please refer to Figure 4 , Figure 4 for the comparison of the test accuracies of the present invention with other benchmark methods on the CIFAR-100, ImageNet-100, and ImageNet-1000 image classification datasets.

[0111] Among them, the comparison benchmark methods include: the method LwF that uses a knowledge distillation loss function to retain old knowledge, the method iCaRL that introduces the nearest representative sample mean rule into this field, the method EEiL that uses a balanced training and cross-distillation loss function, the method LUCIR that utilizes cosine normalization, less forgetting constraint, and inter-class separation to alleviate the catastrophic forgetting problem, the method PODNet with a spatial distillation loss function, and the method TPCIL that retains old knowledge by maintaining the global topological relationship of the feature space. In the figure, (a) is a line chart comparing the test accuracies of this method and other benchmark methods under an incremental setting of 5 stages on the CIFAR-100 dataset; (b) is a line chart comparing the test accuracies of this method and other benchmark methods under an incremental setting of 10 stages on the CIFAR-100 dataset; (c) is a line chart comparing the test accuracies of this method and other benchmark methods under an incremental setting of 25 stages on the CIFAR-100 dataset; (d) is a line chart comparing the test accuracies of this method and other benchmark methods under an incremental setting of 5 stages on the ImageNet-100 dataset; (e) is a line chart comparing the test accuracies of this method and other benchmark methods under an incremental setting of 10 stages on the ImageNet-100 dataset; (f) is a line chart comparing the test accuracies of this method and other benchmark methods under an incremental setting of 5 stages on the ImageNet-1000 dataset; (g) is a line chart comparing the test accuracies of this method and other benchmark methods under an incremental setting of 10 stages on the ImageNet-1000 dataset. It can be seen from the data in the figure that the test accuracy of the method of the present invention is higher than that of other methods.

[0112] Based on the same inventive concept, the present invention also proposes a class-incremental learning system guided by semantic knowledge. Please refer to Figure 5 , Figure 5 which is a structural block diagram of a class-incremental learning system guided by semantic knowledge provided by an embodiment of the present invention. The class-incremental learning system guided by semantic knowledge includes:

[0113] A class-incremental dataset partitioning module 110, configured to obtain a class-incremental dataset including multiple categories, and divide the class-incremental dataset into training samples of multiple training stages according to categories, and ensure that there is no overlap between the categories in the training samples of different training stages;

[0114] A feature space extraction module 120, configured to use a feature extractor to extract and train features from the training samples of different training stages respectively, and construct feature spaces of different stages;

[0115] The inter-class relationship graph construction module 130 is used to obtain the semantic embeddings of class labels in different training stages according to the class labels in the class incremental dataset, and respectively construct the inter-class relationship graphs of each stage according to the semantic embeddings of the class labels in different training stages;

[0116] The classification module 140 is used to design a classifier according to the inter-class relationship graphs of each stage, and calculate the classification results of the training samples in different training stages;

[0117] The local topology preservation module 150 is used to synchronously preserve the local similarity relationship by adopting a local topology preservation strategy for the feature spaces of different stages;

[0118] The class incremental learning model training module 160 is used to define a multi-objective loss function, and train the class incremental learning model by using the local topology preservation constraint of the local similarity relationship to obtain the trained model;

[0119] The model testing module 170 is used to test all learned categories by using the trained model.

[0120] In the above implementation process, the class incremental dataset division module 110 obtains a class incremental dataset containing multiple categories, and divides the class incremental dataset into training samples of multiple training stages according to categories, and ensures that the categories in the training samples of different training stages do not overlap; the feature space extraction module 120 uses a feature extractor to learn the feature space of the training samples in different stages; the inter-class relationship graph construction module 130 uses the semantic embedding of the class labels learned in the current stage to construct an inter-class relationship graph for each stage; the classification module 140 designs a classifier to calculate the possibility that the feature belongs to different categories in different stages, and the local topology preservation module 150 proposes a local topology preservation strategy to synchronously maintain the local similarity relationship; the class incremental learning model training module 160 defines a multi-objective loss function, and simultaneously collaboratively optimizes the training task with the local topology preservation constraint; continuously trains the model using the feature extractor and the classifier, and after completion, the model testing module 170 tests all the learned categories to evaluate the model. Inspired by the latest developments in brain cognitive science, the present invention, guided by semantic knowledge, combines the ideas of regularization-based methods and replay-based methods to solve the network drift problem and the inter-class confusion problem respectively. Through a class incremental learning framework SOUL composed of a CNN feature extractor and a Bi-GCN classifier, the framework uses semantic knowledge extracted from class labels to construct an inter-class relationship graph and learn the Bi-GCN classifier, and based on the Bi-GCN classifier, transfers the inter-class relationship from the semantic modality to the image classifier weights to solve the inter-class confusion problem. At the same time, a local topology preservation constraint is designed to divide the global topology relationship of the learned feature space into a set of local topology relationships and maintain these local relationships in each class incremental learning stage to prevent the topology relationship of the learned feature space from being destroyed. By combining the local topology preservation strategy and the SOUL framework, the class incremental learning method of the present invention achieves state-of-the-art performance accuracy in typical settings on benchmark image classification datasets.

[0121] Please refer to Figure 6 , Figure 6 which is a schematic structural block diagram of an electronic device provided by an embodiment of the present application. The electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, the processor 102, and the communication interface 103 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, such as program instructions / modules corresponding to a class incremental learning system guided by semantic knowledge provided by an embodiment of the present application. The processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0122] Among them, the memory 101 can be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

[0123] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0124] It can be understood that Figure 6 The structure shown is only schematic, and the electronic device may also include more or fewer components than those shown Figure 6 in it, or have a configuration different from that Figure 6 shown. Figure 6 Each component shown in it can be implemented by hardware, software, or a combination thereof.

[0125] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0126] In addition, in each embodiment of this application, the various functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0127] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0128] The above are only the preferred embodiments of this application and are not used to limit this application. For those skilled in the art, this application can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.

[0129] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A class-incremental learning method guided by semantic knowledge, characterized in that, Including the following steps: Obtain a class incremental dataset including multiple categories, divide the class incremental dataset into training samples of multiple training stages according to categories, and ensure that there is no overlap between the categories in the training samples of different training stages. The class incremental dataset includes data of multiple image categories, and each image category data corresponds to a class label; Use a feature extractor to perform feature extraction and training on the training samples of different training stages respectively, and construct feature spaces of different stages; According to the class labels in the class incremental dataset, obtain semantic embeddings of the class labels of different training stages, and respectively construct inter-class relationship graphs for each stage according to the semantic embeddings of the class labels of different training stages; Design a classifier according to the inter-class relationship graphs of each stage, and calculate the classification results of the training samples of different training stages; Adopt a local topology preservation strategy for the feature spaces of different stages to synchronously maintain local similarity relationships; Define a multi-objective loss function, and use the local topology preservation constraint of the local similarity relationship to train the class incremental learning model to obtain a trained model; Among them, the step of obtaining semantic embeddings of the class labels of different training stages according to the class labels in the class incremental dataset, and respectively constructing inter-class relationship graphs for each stage according to the semantic embeddings of the class labels of different training stages includes: According to the class labels in the class incremental dataset, use the Word2Vec algorithm to obtain semantic embeddings of the class labels of the current training stage respectively; Use the semantic embeddings of the class labels of each training stage to construct inter-class relationship graphs for each stage. Among them, the nodes of the inter-class relationship graph represent all classes learned in the current stage, and the edge between any two nodes represents the semantic relationship between classes.

2. The semantic knowledge-guided class incremental learning method according to claim 1, wherein The semantic relationships between the classes include general relationships and core relationships. For the class relationship graph, two symmetric adjacency matrices are defined, respectively representing the general relationships and core relationships between all classes learned in stage k: Among them, N k is the number of all classes learned in stage k, dist(·, ·) is the Euclidean distance, l i and l j are the semantic embeddings of class labels i and j respectively, η represents a hyperparameter, and P(·) represents a pruning operation for pruning weak edges and retaining the core semantic relationships in the class relationship graph.

3. The semantic knowledge-guided class incremental learning method according to claim 1, characterized in that The classifier is a Bi-GCN classifier, and the Bi-GCN classifier includes two GCN propagation functions, with and the corresponding semantic embedding matrix as the inputs of the two GCN propagation functions, and the outputs of the two GCN propagation functions and as two sets of classifier weights. The Bi-GCN classifier ψ is defined as: where is the estimated likelihood that the given sample x i belongs to class c at stage k, and ρ is a hyperparameter that controls the contributions of the two GCN classifiers. correspond to two sets of classification weights for class i, respectively.

4. The class incremental learning method guided by semantic knowledge according to claim 1, characterized in that The step of adopting a local topology preservation strategy for the feature spaces of different stages to synchronously maintain local similarity relationships includes the following steps: Calculate the similarity matrix of the same batch of training samples in the current stage k and the previous stage k - 1, and obtain the similarity matrix of the current stage k as S k , and the similarity matrix in the previous stage k - 1 is S k-1 ; Divide S using a sliding window k and S k-1 into local similarity sub-matrices S k (i) and S k-1 (i), where when the size m of the sliding window is greater than the step size n of the sliding window, the sum of the local similarity relationships in all the local similarity sub-matrices maintains the global similarity relationship; Normalize the local similarity sub-matrices S k (i) and S k-1 (i) respectively to obtain the normalized sub-matrices; According to the normalized sub-matrix, use the mean absolute error loss function to synchronously maintain local similarity relationships.

5. The class incremental learning method guided by semantic knowledge according to claim 1, wherein The multi-objective loss function includes a cross-entropy loss function for classification, a knowledge distillation loss function for penalizing the change in output probability in adjacent class incremental learning stages, and a local topology preservation constraint; the overall loss function of stage k is expressed as: where, |C o | and |C n | represent the numbers of old classes and new classes in the current class incremental learning stage respectively, is the cross-entropy loss function for classification; is the knowledge distillation loss function for penalizing the change in output probabilities in the adjacent class incremental learning stage; is the local topology preservation constraint, and α, γ, and δ represent hyperparameters for balancing the contributions of the three loss functions.

6. The semantic knowledge-guided class incremental learning method according to claim 5, characterized in that The local topology preservation constraint is defined as: Among them, and represent normalized submatrices, L1(·,·) is the mean absolute error loss function, and Id is the number of submatrices calculated by the sliding window.

7. A semantic knowledge-guided class incremental learning system, characterized in that, Including: A class incremental dataset division module for obtaining a class incremental dataset including multiple categories, dividing the class incremental dataset into training samples of multiple training stages according to categories, and ensuring that there is no overlap between the categories in the training samples of different training stages. The class incremental dataset includes data of multiple image categories, and each image category data corresponds to a class label; A feature space extraction module for using a feature extractor to perform feature extraction and training on the training samples of different training stages respectively, and constructing feature spaces of different stages; An inter-class relationship graph construction module for obtaining semantic embeddings of the class labels of different training stages according to the class labels in the class incremental dataset, and respectively constructing inter-class relationship graphs for each stage according to the semantic embeddings of the class labels of different training stages; A classification module, configured to design a classifier according to the inter-class relationship graphs of the respective stages, and calculate the classification results of the training samples in the different training stages; A local topology preservation module, configured to synchronously preserve the local similarity relationship by adopting a local topology preservation strategy for the feature spaces of the different stages; A class incremental learning model training module, configured to define a multi-objective loss function, and train a class incremental learning model by using the local topology preservation constraint of the local similarity relationship to obtain a trained model; Wherein, the inter-class relationship graph construction module is further configured to: According to the class labels in the class incremental dataset, respectively use the Word2Vec algorithm to obtain the semantic embeddings of the class labels in the current training stage; Use the semantic embeddings of the class labels in the respective training stages to construct the inter-class relationship graphs of the respective stages, wherein the nodes of the inter-class relationship graphs represent all the classes learned in the current stage, and the edges between any two of the nodes represent the semantic relationships between classes.

8. An electronic device, characterized in that, Including: A memory, configured to store one or more programs; A processor; When the one or more programs are executed by the processor, the method described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1-6 is implemented.

Images