Small sample class incremental learning method based on hidden attribute enhancement network

Abstract

The invention discloses a small sample class incremental learning method based on a hidden attribute enhancement network, which applies hidden attributes to generalization of small sample image data, captures generalizable information in an image by using a self-attention mechanism and a hidden attribute memory module, and takes the information as learnable hidden attributes. And then a hidden attribute memory pool is constructed, and generalization of new class image recognition is realized by using hidden attributes learned from the base class. Compared with an existing small sample class incremental learning method, the small sample class incremental learning method based on the hidden attribute enhancement network can improve the classification accuracy and improve the generalization ability of identifying new classes, and the generalization ability of the new classes is superior to that of a traditional small sample class incremental learning method.

Description

Technical Field

[0001] The present invention relates to the fields of pattern recognition and machine learning, as well as small sample learning and incremental learning technology, and in particular to a small sample class incremental learning method based on a latent attribute incremental network. Background Art

[0002] In recent years, machine learning models have achieved remarkable results in various fields such as natural language processing and computer vision. Humans generally have the ability to continuously learn to recognize object categories, such as being able to learn to recognize elephants without forgetting the previously learned ability to recognize cats and dogs. This ability is often referred to as incremental learning. Inspired by humans' incremental learning ability, class incremental learning was proposed and applied to machine learning models.

[0003] In incremental class learning, the model is required to learn new knowledge on continuous tasks without forgetting previously learned knowledge. The key issue is how to effectively acquire new category knowledge while not forgetting old knowledge. However, due to the high cost of data annotation and the difficulty in obtaining rare category samples, it becomes impractical to train models using large-scale datasets in practical applications. Therefore, small sample incremental class learning has attracted more and more attention.

[0004] In the incremental learning of small-sample classes for image classification, the model aims to continuously learn new categories using limited data, and it faces the dual challenges of catastrophic forgetting and poor generalization ability for new categories. To solve the above problems, recent studies have proposed three different methods: meta-learning-based methods, replay-based methods, and feature and feature space-based methods. The meta-learning-based method simulates the real incremental learning stage by constructing a pseudo-incremental learning task in the base task, so that the model learns to learn on a small number of samples. The replay-based method prevents the model from forgetting old knowledge by retaining some old class samples to form a replay buffer. The feature and feature space-based method synthesizes or amplifies the base class image samples to synthesize some pseudo-new class image samples, which can reserve a part of the feature space for the real new class and avoid confusion between the distribution of new and old class data.

[0005] However, these methods all learn global features for classification, often ignoring discriminative local information, resulting in weak knowledge generalization ability. When faced with new incremental tasks, such as recognizing new animal categories like tigers and leopards, global features have weak generalization ability, while local features, such as the texture of claws or hair, are more generalizable and transferable. These fine-grained details are crucial to distinguish visually similar animals belonging to different categories. Summary of the invention

[0006] In response to the problems of existing methods, the present invention changes the way previous methods learn features and proposes a latent attribute enhancement network to enhance the features of image objects, in which latent attributes can greatly improve the generalization of new classes, and the latent attributes learned from the base class can be generalized to different new classes.

[0007] The present invention proposes a latent attribute enhancement network to enhance image object features, where latent attributes refer to generalizable features in the image, such as shape or color, and latent attributes are automatically learned to focus on key local features. In order to solve the overfitting problem that is prevalent in small sample learning of image classification, we freeze the backbone network parameters after the basic training stage and use meta-learning to simulate the realistic incremental learning process. The simulation involves constructing multiple small sample incremental tasks to reproduce real incremental learning scenarios. The small sample class incremental learning model based on the latent attribute enhancement network proposed in the present invention can effectively focus on local discriminant information and enhance the generalization of new class learning.

[0008] The technical solution steps of the present invention are as follows:

[0009] Step 1: Pre-train image classification model;

[0010] S11 Input: image data of base task.

[0011] S12 base task pre-training: Input the image data of the base task into the image classification model for training.

[0012] S13 output: pre-trained image classification model.

[0013] Step 2: Learning latent attributes

[0014] S21 Pseudo-incremental learning: Sample the image data of the base task and randomly select some images from it as the support set and query set.

[0015] S22 inputs the support set and query set sampled in the previous step, as well as the latent attributes, into the latent attribute enhancement network. The latent attribute enhancement network consists of three parts: the image classification model pre-trained in step 1, the auxiliary memory module, and the Transformer-based knowledge interaction module. The auxiliary memory module stores the latent attributes. Input the support set and latent attributes into the latent attribute enhancement network, then input the query set and latent attributes into the network, generalize the knowledge in the support set captured by the latent attributes to the query set, and finally perform loss calculation and parameter update.

[0016] S23 model loss: Using local cross entropy loss to train the latent attribute enhancement network can better capture generalized local information.

[0017] S24 model reasoning: The network parameters are frozen in the reasoning stage, and the latent attributes trained on the base task can generalize the knowledge to new tasks well.

[0018] Compared with the previous technology, this paper proposes a latent attribute enhancement network that can focus on discriminative local visual areas to capture fine-grained features. At the same time, this paper constructs a Transformer-based knowledge interaction module that can achieve information fusion between local image areas and latent attributes, thereby enhancing the model's ability to handle complex and detailed information and effectively improving the generalization of the model.

[0019] Auxiliary memory module

[0020] Due to data limitations and privacy constraints in incremental learning of small sample classes, only the data of the current task can be used during training, which may lead to severe catastrophic forgetting. Traditional playback methods alleviate the problem of catastrophic forgetting by retaining the data of old tasks to train new models. However, these methods require a lot of memory space. In addition, the imbalance between old and new class data will lead to biased recognition ability of the model. The present invention proposes an auxiliary memory module for learning latent attributes, which can not only retain the knowledge of old categories, but also be generalized to new categories.

[0021] For each image x,x∈R H×W×C (where H, W, and C represent the height, width, and number of channels of the image, respectively). We use the pre-trained classification model Φ θ (x) Extract its feature map:

[0022] F=Φ θ (x),F∈R H '×W' ×C '

[0023] Where θ is the learnable parameter of the classification model. Then, we reshape the feature map to obtain the final image feature token T, where T∈R L×C ', L = H' × W', L represents the number of image tokens. The hidden attribute in the auxiliary memory module proposed in this invention is denoted by P LA ,P LA ∈R N×C ', where N represents the number of hidden attribute settings. LA The parameters in the network are randomly initialized and automatically updated by gradient descent. The present invention aims to enable the network to automatically learn latent attributes that not only preserve the information of old categories but also generalize to new categories and use them to enhance the generalization of image features.

[0024] Transformer-based knowledge interaction module

[0025] Traditional small sample class incremental learning methods usually focus on learning global features to train classification models, which may be affected by noisy background information, resulting in the model being unable to capture finer-grained discriminative features, leading to classification errors. Therefore, focusing on discriminative local features is crucial to improving the robustness of recognition models. The present invention proposes a Transformer-based knowledge interaction module for fusing information from latent attributes and visual features, focusing on discriminative semantic areas to enhance visual feature representation.

[0026] The sampled support set images and query set images in pseudo-incremental learning are sent to the image classification model to obtain the corresponding feature tokens. Assume that the token of the support image is denoted by T s , the token of the query image is denoted by T q , the hidden attribute is recorded as P LA The knowledge interaction module can be expressed as:

[0027]

[0028] Q=W q ×concat[T,P LA ]

[0029] K=W k ×concat[T,P LA ]

[0030] V=W v ×concat[T,P LA ]

[0031] Among them, T indicates support for token T s Or query token T q . Visual tokens are feature maps output by image classification models, which eliminates the extra tokenization step required by traditional self-attention mechanisms. The present invention simplifies the processing flow and fully utilizes the feature extraction capabilities of image classification models. Due to the interaction between latent attributes and each local area, key distinguishing areas can be effectively located, thereby enhancing feature representation.

[0032] Network training phase

[0033] In the training phase, the present invention divides the training set into a support set and a query set through pseudo-incremental learning, where the support set is denoted as S and the query set is denoted as Q. During the training process, the latent attributes first interact with the visual features of the support set and then with the visual features of the query set. The latent attributes can simultaneously learn the effective areas in the support set image and the query set image, thereby enhancing the generalization ability of the model. After the knowledge interaction module, we obtain the enhanced support set visual features and latent attributes:

[0034]

[0035] KIM stands for Transformer-based knowledge interaction module. represents the enhanced visual feature token of the i-th image of class c from the support set S, Represents the enhanced latent attribute.

[0036] The enhanced visual features and latent attributes of the query set are:

[0037]

[0038] in represents the enhanced visual feature token of the i-th image of class c from the support set Q, Represents the enhanced latent attribute.

[0039] The category prototype of each category of visual token is as follows:

[0040]

[0041] where n c Represents the number of images in category c.

[0042] In order to obtain discriminative local features, the present invention imposes classification constraints on each position in the query feature map, and performs classification by measuring the cosine similarity between the mth query image token and the cth category prototype to obtain the classification score:

[0043]

[0044] in is the enhanced query set image feature. σ represents the sigmoid function. GAP represents global average pooling, and cosine represents the cosine distance. The cross entropy loss during training can be expressed as:

[0045]

[0046] where y c Represents the label of class c.

[0047] Network inference phase

[0048] After the training phase, the present invention fixes the parameters of the network to prevent overfitting. For subsequent new tasks, we use the latent attributes stored in the auxiliary memory module to enhance the feature representation of the support image and the query image respectively. Then, the category prototype P is calculated by the enhanced support image features. * The prediction for the t-th query image can be expressed as:

[0049]

[0050] in, is the enhanced feature representation of the t-th query image, P i * is the enhanced visual prototype of the i-th category.

[0051] Compared with previous small-sample incremental learning methods that mainly focus on global features and ignore local discriminative information, the present invention can effectively capture local features that are crucial for distinguishing similar-looking animals belonging to different categories and is more generalizable.

[0052] The present invention proposes a latent attribute enhancement network, which uses latent attributes to capture generalizable and discriminative information in samples to improve the adaptability to new tasks.

[0053] Comprehensively consider the capture of class-level and instance-level information: The present invention samples the training images multiple times through pseudo-incremental learning to capture the key information of the image at the instance level. The model parameters are frozen during inference, and the category prototype is used for classification. This enables the model to better retain the knowledge of old tasks, and freezing the model parameters effectively alleviates the overfitting problem.

[0054] Stronger feature robustness: Traditional methods only consider the distribution of global features in space, and after freezing the network parameters, there will usually be confusion between the distribution of new and old classes. The present invention proposes learning the network from local features, which effectively avoids the confusion problem in traditional methods.

[0055] Improved classification accuracy: Since the present invention can capture generalizable information in different images, it performs well in generalization ability of new tasks. The image classification model is no longer overly biased towards new tasks, which significantly reduces the misclassification of previous categories and improves the classification accuracy of new and old categories, ultimately achieving an improvement in incremental learning performance of small sample classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a specific implementation flow chart of the present invention.

[0057] Figure 2 The experimental results of small-sample incremental learning on the Mini-ImageNet dataset are shown in Figure 2. The final performance is measured by the accuracy (SA) and average accuracy (AA) on each task compared with different small-sample incremental learning methods.

[0058] Figure 3The experimental results of small-shot incremental learning on the CIFAR100 dataset are shown. The final performance is measured by the accuracy (SA) and average accuracy (AA) on each task compared with different small-shot incremental learning methods.

[0059] Figure 4 The experimental results of small sample incremental learning on the CUB200 dataset are shown in Figure 2. The final performance is measured by the accuracy (SA) and average accuracy (AA) on each task compared with different small sample incremental learning methods.

[0060] Figure 5 It is a line chart showing the average accuracy trend of each task on the Mini-ImageNet, CIFAR100, and CUB200 datasets. DETAILED DESCRIPTION

[0061] To accurately evaluate our invention, we conduct extensive experiments on three benchmark few-shot class incremental learning datasets: CIFAR100, mini-ImageNet, and CUB200. CIFAR100 contains 100 classes, each with 500 training images and 100 test images. mini-ImageNet is extracted from ImageNet and contains the same number of classes and images as CIFAR100. CUB200, which is commonly used for few-shot learning, is a more fine-grained dataset containing 200 bird species, each with 30 training images and 30 test images.

[0062] For CIFAR100 and mini-ImageNet, we selected 60 categories as base tasks and divided the remaining 40 categories into 8 tasks, using an experimental setting of 5 categories and 5 samples per task, resulting in a total of 9 training tasks. In CUB200, 100 categories were set as base tasks, and the remaining 100 categories were divided into 10 tasks, using an experimental setting of 10 categories and 5 samples per task, resulting in a total of 11 training tasks. The performance of the model is evaluated by testing the accuracy of the model on each task and the average accuracy of all tasks. During the training phase, only the data of the current task can be used; during the testing phase, the performance of the model on all categories encountered is evaluated.

[0063] On the CIFAR100 dataset, Figure 2 As shown in Figure 2, our method improves the average accuracy by 1.28% over the state-of-the-art TEEN method. Through feature enhancement of latent attributes, we are able to learn more discriminative feature representations, thereby improving recognition accuracy.

[0064] On the mini-ImageNet dataset, such as Figure 3As shown in the figure, our method outperforms the latest FSCL method in terms of average accuracy, with the accuracy of the last task increased by 1.52% and the average accuracy increased by 1.85%. Compared with the method of using knowledge distillation to alleviate catastrophic forgetting, our last task accuracy increased by 2.81% and the average accuracy increased by 1.11%. This shows that our method can effectively generalize the latent attribute knowledge captured in the base task to the new task, effectively alleviating the problem of catastrophic forgetting.

[0065] On the CUB200 dataset, Figure 4 As shown in , compared with the latest TEEN method, our method improves the accuracy of the last task by 0.94% and the average accuracy by 0.95%. Figure 5 As shown in the figure, our method outperforms other comparison methods and demonstrates excellent performance. This further demonstrates that our method can continuously enhance the generalization of image classification models in all tasks, verifies its effectiveness in incremental learning tasks for small sample classes, and highlights its advantages as a potential solution.

Claims

1. A small sample class incremental learning method based on latent attribute enhancement network, characterized in that: The implementation steps of this method are as follows: Step 1: Pre-train image classification model; S11 input: image data of base task; S12 base task pre-training: input the image data of the base task into the image classification model for training; S13 output: pre-trained image classification model; Step 2: Learning of latent attributes; S21 Pseudo-incremental learning: Sample the image data of the base task and randomly select some images from it as the support set and query set; S22 inputs the sampled support set and query set, as well as the latent attributes, into the latent attribute enhancement network. The latent attribute enhancement network consists of three parts: the image classification model pre-trained in step 1, the auxiliary memory module, and the Transformer-based knowledge interaction module. The auxiliary memory module stores the latent attributes. The support set and the latent attributes are input into the latent attribute enhancement network, and then the query set and the latent attributes are input into the network. The knowledge in the support set captured by the latent attributes is generalized to the query set, and finally the loss calculation and parameter update are performed. S23 model loss: Using local cross entropy loss to train the latent attribute enhancement network can better capture generalized local information; S24 model reasoning: The network parameters are frozen in the reasoning stage, and the latent attributes trained on the base task can generalize the knowledge to new tasks well.

2. According to claim 1, a small sample class incremental learning method based on latent attribute enhancement network is characterized in that: In the latent attribute enhancement network, the implementation process of the auxiliary memory module is as follows: For each image x,x∈R H×W×C , where H, W, C represent the height, width and number of channels of the image respectively, and the pre-trained classification model Φ θ (x) Extract its feature map: F=Φ θ (x),F∈R H '×W' ×C ' Where θ is the learnable parameter of the classification model; then, the feature map is reshaped to obtain the final image feature tokenT, where T∈R L×C ',L=H'×W', L represents the number of image tokens, H' and W' represent the height and width of the feature map, and C' represents the number of channels of the feature map; the hidden attribute in the auxiliary memory module is denoted by P LA ,P LA ∈R N×C ', where N represents the number of hidden attribute settings; P LA The parameters in the network are initialized randomly and updated automatically by gradient descent.

3. The small sample class incremental learning method based on latent attribute enhancement network according to claim 1 is characterized in that: The implementation process of the Transformer-based knowledge interaction module in the latent attribute enhancement network is as follows: the Transformer-based knowledge interaction module is used to fuse the information of latent attributes and visual features, focusing on the discriminative semantic areas to enhance the visual feature representation; The sampled support set images and query set images in pseudo-incremental learning are sent to the image classification model to obtain the corresponding feature tokens. Assume that the token of the support image is denoted by T s , the token of the query image is denoted by T q , the hidden attribute is recorded as P LA ; The knowledge interaction module is expressed as: Q=W q ×concat[T,P LA ] K=W k ×concat[T,P LA ] V=W v ×concat[T,P LA ] Among them, T indicates support for token T s Or query token T q ; concat represents the concatenation operation, which concatenates the input feature token with the hidden attribute; Q, K, and V represent the query, key, and value in the Transformer self-attention mechanism respectively; W q , W k , W v They represent the mapping matrix, which maps the concatenated feature tokens to the hidden attributes; softmax represents the softmax function. is a scaling factor to prevent gradient explosion during gradient descent.

4. The small sample class incremental learning method based on latent attribute enhancement network according to claim 1 is characterized in that: In the latent attribute enhancement network, the implementation process of the network training phase is as follows: In the training phase, the training set is divided into a support set and a query set through pseudo-incremental learning, where the support set is denoted as S and the query set is denoted as Q. During the training process, the latent attributes first interact with the visual features of the support set and then interact with the visual features of the query set. Hidden attributes are learned from valid regions in both support set images and query set images, thereby enhancing the generalization ability of the model. The visual features and latent attributes of the support set are enhanced after the knowledge interaction module: KIM stands for Transformer-based knowledge interaction module. represents the i-th image from the c-th class of the support set S, represents the enhanced visual feature token of the i-th image of the c-th class from the support set S, represents the enhanced latent attribute; The enhanced visual features and latent attributes of the query set are: in represents the i-th image of the c-th class from the query set Q, represents the enhanced visual feature token of the i-th image of the c-th category from the query set Q, represents the enhanced latent attribute; The category prototype of each category of visual token is as follows: where n c represents the number of images in category c; In order to obtain discriminative local features, classification constraints are imposed on each position in the query feature map. Classification is performed by measuring the cosine similarity between the mth query image token and the cth category prototype to obtain the classification score: in is the enhanced query set image feature; σ represents the sigmoid function; GAP represents global average pooling, and cosine represents the cosine distance; the cross entropy loss during training is expressed as: where y c represents the label of the c-th class, and N represents the number of images in the c-th class.

5. The small sample class incremental learning method based on latent attribute enhancement network according to claim 1 is characterized in that: In the latent attribute enhancement network, the implementation process of the network reasoning stage is as follows: After the training phase, the parameters of the network are fixed to prevent overfitting. For subsequent new tasks, the latent attributes stored in the auxiliary memory module are used to enhance the feature representation of the support image and the query image respectively. Then, the category prototype P is calculated using the enhanced support image features. * ; The prediction of the t-th query image is expressed as: in, is the enhanced feature representation of the t-th query image, It is the enhanced visual prototype of the i-th category, and the argmax function means taking the highest value of the prediction score of the t-th picture among all categories.

Images