Method and device for few-shot class-incremental learning

Abstract

The present disclosure provides methods and systems for few-shot class-incremental learning. According to one aspect, a method is provided. The method includes obtaining at least one weight attention map of a first network and updating weights of a second network using the at least one weight attention map, wherein the second network is a modulation network. The method further includes generating at least one feature attention map of the second network based on the at least one weight attention map of the first network and an input image set of at least one class. The method further includes generating at least one feature map of the first network based on the input image set of the at least one class and updating the at least one feature map of the first network based on the feature attention map of the second network.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and interest in the sequence of U.S. Patent Application No. 17 / 527,295, filed November 16, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to the field of machine learning, and more particularly to methods and apparatus for incremental learning of meta-few-shot classes. Background Technology

[0004] Classical machine learning algorithms are typically developed for classifying a finite number of classes. Existing learning algorithms attempt to train models to learn new classes while preventing or minimizing the forgetting of already learned classes (e.g., prior knowledge). However, these algorithms cannot effectively adapt pre-trained models to new classes without catastrophically forgetting prior knowledge. Therefore, finding a balance between adapting to new knowledge and preventing or minimizing forgetting remains a challenge in the field of machine learning. In addition to this balance challenge, existing machine learning algorithms face further challenges in adapting to new knowledge due to the limited number of training samples reflecting the real world. Other limitations of existing algorithms will be further described in this paper.

[0005] Therefore, there is a need for methods and devices for incremental learning of meta-few-shot classes that eliminate or mitigate one or more limitations of existing technologies.

[0006] This background information is provided to disclose information that the applicant believes may be relevant to the present invention. No of the above information should be considered or construed as constituting prior art relative to the present invention. Summary of the Invention

[0007] This invention provides a method and system for incremental learning of meta-few-shot classes. According to a first aspect, a method is provided. The method includes: acquiring at least one weight attention map of a first network. The method further includes: updating the weights of a second network using the at least one weight attention map of the first network, wherein the second network is a modulation network. The method further includes: generating at least one feature attention map of the second network based on the at least one weight attention map of the first network and an input image set of at least one class. The method further includes: generating at least one feature map of the first network based on the input image set of the at least one class. The method further includes: updating the at least one feature map of the first network based on the feature attention map of the second network. The method can provide enhanced model training, wherein the model's weight selection is learned and based on the current state of model knowledge and input images of new classes.

[0008] In some embodiments, the first network is a prediction network. In some embodiments, updating the weights of the second network includes multiplying the weights of the second network by the at least one weight attention map of the first network. In some embodiments, the at least one weight attention map represents the current state of the first network, the current state indicating learned knowledge. The method can utilize known knowledge for weight selection in the model. The model can provide a balance between learning new knowledge and maintaining old knowledge.

[0009] In some embodiments, the method further includes providing the first network with a training dataset indicating n new classes. In some embodiments, the method further includes adding n nodes to a classifier of the first network, wherein the classifier includes one or more existing nodes indicating one or more learned classes among the at least one class, each of the existing nodes indicating a learned class. In some embodiments, the method further includes merging the n nodes with the one or more existing nodes by concatenation. In some embodiments, the method further includes updating the first network based on the training dataset. In some embodiments, the method further includes evaluating the updated first network based on test data indicating the new classes and the learned classes. The method can provide augmented training based on simulated incremental learning. The method can provide a learnable backbone for the model.

[0010] In some embodiments, the training and test datasets are derived from a sequential dataset. In some embodiments, the sequential dataset and the input image set are derived from a base dataset indicating base classes, which include the n new classes and the at least one of the original classes. This approach can alleviate the problem of learning new knowledge while forgetting old knowledge.

[0011] In some embodiments, each of the first network and the second network is a deep neural network. The method can train scalable architectures (i.e., deep neural networks).

[0012] In some embodiments, the n new classes come from a class domain different from the at least one class. The method can provide a training model for applications based on different domains.

[0013] According to a second aspect, a second method is provided. The method includes: providing a training dataset indicating n new classes to a network. The method further includes: adding n nodes to a classifier of the network, wherein the classifier includes one or more existing nodes indicating one or more learned classes, each of the existing nodes indicating a learned class. The method further includes: merging the n nodes with the one or more existing nodes via concatenation. The method further includes: updating the first network based on the training dataset. The method further includes: evaluating the updated network based on test data indicating the new classes and the learned classes. The method can provide simulated training in a continuous-round manner (i.e., incremental sessions). The method can provide evaluation in each incremental session based on new knowledge learned in the incremental session and old knowledge learned from previous sessions. Therefore, the method can provide model optimization aligned with the evaluation.

[0014] In some embodiments, the network is a prediction network. In some embodiments of one or more of the above aspects, the training dataset represents an n-way k-shot classification problem. In some embodiments, the evaluation includes generating a loss function for the updated prediction network based on the test data. This approach is better able to handle forgetting old knowledge.

[0015] In some embodiments of one or more of the above aspects, updating the prediction network includes updating the weight parameters of the backbone of the prediction network based on the training dataset.

[0016] In some embodiments of the foregoing aspects, the method further includes updating the weight parameters of the backbone of the prediction network based on the generated loss function. The method can provide an incrementally updated learnable backbone.

[0017] According to a third aspect, an apparatus is provided, the apparatus comprising a module for performing the methods of one or more of the foregoing aspects.

[0018] According to a fourth aspect, an apparatus is provided, comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method described in one or more of the preceding aspects.

[0019] According to a fifth aspect, a computer-readable medium is provided, wherein the computer-readable medium stores program code executable by a device, the program code being used to perform the methods described in one or more of the preceding aspects.

[0020] According to a sixth aspect, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer performs the methods described in one or more of the above aspects.

[0021] According to a seventh aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the methods described in one or more of the preceding aspects.

[0022] Optionally, in one implementation, the chip may further include the memory. The memory stores the instructions, and the processor executes the instructions stored in the memory. When the instructions are executed, the processor performs the method of the first aspect.

[0023] According to an eighth aspect, an electronic device is provided. The electronic device includes the motion recognition device described in any one or more of the foregoing aspects.

[0024] Other aspects of the invention provide apparatus and systems for implementing the methods disclosed herein. For example, wireless stations and access points may be configured with machine-readable storage including instructions that, when executed by a processor of these devices, configure the devices to perform the methods disclosed herein.

[0025] Embodiments have been described above in conjunction with various aspects of the present invention, and these embodiments can be implemented based on these aspects. Those skilled in the art will understand that embodiments can be implemented in conjunction with the aspects described therein, but may also be implemented together with other embodiments of that aspect. It will be apparent to those skilled in the art that embodiments are mutually exclusive or incompatible with each other. Some embodiments may be described in conjunction with one aspect, but may also be applicable to other aspects, as will be apparent to those skilled in the art. Attached Figure Description

[0026] Other features and advantages of the present invention will become apparent from the following specific embodiments, taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 An embodiment of the present invention provides a classic machine learning system for image classification.

[0028] Figure 2 An embodiment of the present invention provides a system for incremental learning of classes.

[0029] Figure 3 An embodiment of the present invention provides a system for incremental learning of few-shot classes.

[0030] Figure 4The embodiment of the present invention illustrates a weight selection mechanism for updating new classes.

[0031] Figure 5A and Figure 5B The present invention illustrates a neuromodulated meta-learning (ANML) method provided by an embodiment of the present invention.

[0032] Figure 6 The overall architecture of the learning method provided by an embodiment of the present invention is shown.

[0033] Figure 7 An image classification model provided by an embodiment of the present invention is shown.

[0034] Figure 8A An embodiment of the present invention provides a bidirectional guided adaptive module.

[0035] Figure 8B A flowchart for bidirectional guided adaptation for the first layer is shown, according to an embodiment of the present invention.

[0036] Figure 9 The present invention illustrates a few-shot class incremental learning (FSCIL) method provided by an embodiment of the present invention.

[0037] Figure 10 A flowchart of the pseudo-FSCIL provided by an embodiment of the present invention is shown.

[0038] Figure 11 Experimental results provided by embodiments of the present invention are shown.

[0039] Figure 12 A schematic structural diagram of the system architecture provided by an embodiment of the present invention is shown.

[0040] It should be noted that the same features are identified by the same reference numerals in all the accompanying drawings. Detailed Implementation

[0041] Classic machine learning algorithms are typically developed to classify a predefined number of classes. Figure 1 An embodiment of the present invention provides a classic machine learning system for image classification. For example... Figure 1As shown, the image dataset 102 is typically divided into a training set 104 and a test set 106, where the training and test data include classes of the same type (e.g., the same category ID), as will be understood by those skilled in the art. Dataset 102 is always independently and identically distributed (IID). Typically, the model is trained on the training set 104 and tested on the test set 106. Model evaluation is usually performed on a single set. During deployment, the model may only classify the classes it observed during training. Therefore, the model may not learn additional classes.

[0042] Incremental learning aims to enable trained models to continue learning new classes during testing time. Figure 2 An embodiment of the present invention provides a system for incremental class learning. System 200 assumes the more realistic condition that data enters the system sequentially. Initially, at 202, a model can be trained using a dataset with J classes. After deployment, the trained model can be trained using data from new classes. At each learning stage, the model may be fed only data from the new classes or trained with data from the new classes, so the model may learn only from the fed data. For example, at learning stage 204 associated with time step t, the model is trained on a task based on N class classification (already trained with J classes). In addition to learning from new classes (e.g., N classes) based on the fed data at 204, the model is also forced not to forget the old classes (J classes).

[0043] Therefore, after each training phase, the model is evaluated based on all classes trained during and before the training phase. For example, after learning phase 204 (associated with time step t), at point 206, the model is evaluated on all J+N classes. The expectation is that the model learns new classes (N classes) but does not forget old classes (J classes). Accordingly, the model is evaluated on test data for the current class and previous classes.

[0044] One challenge of class incremental learning is that the new classes (e.g., N classes) at each learning stage do not overlap with the previous classes (e.g., J classes). Therefore, System 200 can be called non-IID.

[0045] Few-shot class incremental learning is another variant of class incremental learning with additional conditions. In addition to the system 200 mentioned above, few-shot class incremental learning assumes that at each time step t (i.e., at each learning stage), the model can only observe a small number of images for each new class.

[0046] Figure 3An apparatus system for incremental learning with few-shot classes, provided by an embodiment of the present invention, is illustrated. In each learning phase, a model can be trained to perform a new task of N-way k-shot classification. The N-way k-shot classification problem may include N new classes, each with k images. Before deployment, at point 302, the model can be trained using a dataset of J classes. After deployment, in learning phase 304 associated with time step t, the model can be trained based on the N-way k-shot classification problem. Subsequently, at point 306, the model can be evaluated based on both the J and N classes.

[0047] System 300 (few-shot incremental learning) may be more practical than System 200 because users may only need a small number of images for training each new class. However, having only a small number of images for training each new class can lead to overfitting with System 300. Furthermore, few-shot incremental learning may suffer from the same limitations as other incremental learning techniques, such as catastrophic forgetting and the ability to learn new knowledge with very few training samples. Finding a balance between learning new knowledge and not forgetting old knowledge remains a challenge for incremental learning techniques.

[0048] As those skilled in the art will understand, there are trade-offs in addressing the limitations of the prior art. In preserving prior knowledge (e.g., by allowing fewer model weights to be learned), the model's adaptability may be limited. Similarly, by increasing the model's adaptability (e.g., by allowing more model weights to be learned), prior knowledge may be destroyed or forgotten.

[0049] Figure 4 An embodiment of the invention provides a weight selection mechanism for updating new classes. Mechanism 400 can alleviate the challenge of forgetting old knowledge. In each incremental step D, the 10% of model parameters with the lowest weight magnitudes can be selected for training. The remaining parameters are fixed. This method is based on the understanding that old knowledge is encoded in the weights, particularly the weights with larger magnitudes.

[0050] Therefore, at incremental step D(2) 402, weight 404 can be selected for training. The prior knowledge at 402 can be encoded in weight 412, which is based on data 410. Similarly, at incremental step D(3) 406, weight 408 can be selected for training. The prior knowledge at 406 can be encoded in weights 412 and 404, which is based on data 412.

[0051] The first problem with weight selection mechanisms is that they are artificially engineered (e.g., the metric used for selection (amplitude), the percentage of weights (10%)). The selection of low-amplitude weights depends on the data and can vary from class to class. Furthermore, selecting low-amplitude weights for training based on the understanding that they have less impact on forgetting old knowledge may not be a correct general rule. The second problem with weight selection mechanisms is that model optimization is inconsistent (directly related) to the goals of preventing the forgetting of old knowledge (the forgetting problem) and learning new knowledge (the adaptation problem).

[0052] Figure 5A and Figure 5B This illustrates a neuromodulated meta-learning (ANML) method. ANML is a meta-learning method based on automatic weight selection for incremental learning. (Reference) Figure 5A In ANML, the neural modulation network 502 automatically selects weights in the prediction network 504 for updating. The neural modulation network generates a gating graph to achieve selective activation. The neural modulation network 502 generates element-wise gating 506 for the forward activation of the prediction network, thereby achieving selective activation of the prediction network. Based on the output of the neural modulation network 502, important and unimportant weights of the prediction network 504 can be determined. Compared to the weight selection mechanism 400, ANML automates manual engineering design operations (e.g., weight selection for updates).

[0053] refer to Figure 5B The algorithm attempts to simulate test scenarios using training data. Training data (new knowledge) is sequentially fed into the model, and the model parameters are updated. After the model is updated, it is evaluated based on both the new and old knowledge. The evaluation is based on sampling random images from old and new (future) classes. Therefore, meta-optimization includes samples from future time steps, which may not be optimal for handling the forgetting problem. Furthermore, meta-optimization does not cover all previous classes. As those skilled in the art will understand, ANML techniques have several limitations.

[0054] The first limitation is that the feature extractor (i.e., the backbone in the prediction network 504) is fixed during testing. Therefore, the parameters of the backbone are not updated as the model network learns. A fixed backbone can further challenge the classifier's image classification, thus limiting the overall model performance. Therefore, the embodiments described herein provide a learnable backbone.

[0055] The second limitation is that only the last layer is gated, which restricts the scalability of the technique. Since ANML is based on shallow networks, it is not as effective for deeper networks.

[0056] The third limitation is that the neural modulation network 502 is unaware of the current state of the prediction network 504 (e.g., the degree of knowledge or class learned at each incremental step). Therefore, the output 506 of the neural modulation network 502 tends to be the same regardless of the learning stage of the prediction network, which limits model optimization.

[0057] The embodiments described herein can alleviate the limitations of few-shot incremental learning for image classification, including the forgetting problem (forgetting old knowledge) and adaptation (learning new knowledge). The embodiments described herein can use a bidirectional guided adaptation module to guide weight selection and model training based on new classes. The embodiments described herein can provide a model learning method that learns new knowledge (new classes) while being less prone to forgetting old knowledge (the forgetting problem). In some embodiments, the learning method can be based on meta-learning-based algorithms.

[0058] The embodiments described herein provide a fully optimization-based approach for incremental learning of few-shot classes. These embodiments can leverage a two-layer optimization phase from meta-learning to mitigate the problem of learning new knowledge versus forgetting old knowledge.

[0059] The embodiments described herein can provide an improved learning architecture for incremental learning. Figure 6 The overall architecture of the learning method provided by an embodiment of the present invention is illustrated. Architecture 600 may include a modulation network 602 and a prediction network 604. In some embodiments, the prediction network 604 may include a backbone 606 (e.g., a feature extractor) and a classifier 608. As those skilled in the art will understand, each of the backbone network 606 and the modulation network 602 may have similar neural network structures (e.g., the same number of layers and the same weight parameters).

[0060] The backbone 606 may include layers 620, 622, 624, and 628. Similarly, the modulation network 602 may include layers 630, 632, 634, and 638. In one embodiment, the backbone network 606 and the modulation network 602 may have corresponding layers for reference... Figure 8A The described bidirectional guided adaptive mechanisms interact with each other. For example, the first layer 620 of the backbone 606 can correspond to the first layer 630 of the modulation network 602, and layers 620 and 630 together can be referred to as a pair of corresponding layers.

[0061] In some embodiments, the weights of the backbone network (e.g., at each layer 620, 622, 624, ..., 628) can already be encoded with prior knowledge because the weights of the prediction network (i.e., the backbone 606) encode learned or prior knowledge. Therefore, learned or prior knowledge can be encoded in the weights associated with each backbone layer (e.g., 620, 622, 624, ..., 628).

[0062] In one embodiment, at time t, prediction network 606 and modulation network 602 may feed input 601 (new knowledge) to 610 and 612, respectively. For example, input 601 may include an image dataset indicating one or more new classes.

[0063] In some embodiments, the weights associated with layer 620 of backbone 606 can be updated based on the received input 610. Thereafter, information indicating the weights associated with first layer 620 can be sent (640) to the corresponding first layer 630 of modulation network 602. The weight information can be sent in the form of an attention map (e.g., a weighted attention map).

[0064] The corresponding layer 630 can update its weight parameters based on the received weighted attention map. In some embodiments, the update can be a multiplication operation. Subsequently, in some embodiments, the corresponding layer 630 can generate a feature attention map based on its updated weights and the received input 601, the feature attention map indicating the features of input 601 based on the updated weights. Similarly, layer 620 can generate a feature map based on its associated weights and the received input 601, the feature map indicating the features of input 601 based on the associated weights of layer 620. The feature attention map generated by layer 630 can be applied to the feature map of layer 620 by, for example, multiplication 642, as shown. Those skilled in the art will understand that feature attention maps and feature maps can be similar, but their value ranges can be different. As mentioned above, a feature attention map can indicate the features of an input image and is the output of a network layer (e.g., modulation layer 630). The value of a feature attention map may be between [0,1]. A feature map can also indicate the features of an input image and is the output of a network layer (e.g., prediction layer 620). Feature maps can typically have values ​​outside the range [0,1].

[0065] In some embodiments, the second layer 622 of the backbone receives (644) a combined feature map generated from multiplication 642. The weights of layer 622 can be transmitted (646) as an associated weight attention map to the corresponding layer 632 of the modulation network 602, as shown. The corresponding layer 632 can receive (646) the weight attention map of layer 622, and can also receive (648) a feature attention map associated with layer 630, as shown. Based on the received (646) weight attention map and the received (648) feature attention map, the weights associated with layer 632 can be updated. Then, a feature attention map associated with layer 632 can be generated based on the updated weights of layer 632. Similarly, layer 622 can generate a feature map based on its associated updated weights and the received (644) combined feature map. The generated feature map of layer 632 can be applied to the feature map of layer 622, for example, by multiplication 650, as shown.

[0066] The same process described for reference pairs 632 and 622 can be performed with the remaining pairs (e.g., pairs 634 and 624, ... and pairs 638 and 628). The feature maps generated by the last pairs 638 and 628 can be combined by multiplication 652 (the feature attention map generated by layer 638 is applied to the feature map generated by layer 628) and sent to classifier 608 (654).

[0067] In some embodiments, at least one weight attention map of the prediction network 604 may be obtained. In some embodiments, the weights of the modulation network 602 may be updated using at least one weight attention map of the prediction network 604. In some embodiments, at least one feature attention map of the modulation network 602 may be generated based on at least one weight attention map of the prediction network 604 and an input image set of at least one class (e.g., input 601). In some embodiments, at least one feature map of the prediction network 604 may be generated based on an input image set of at least one class. In some embodiments, the feature map of the prediction network 604 may be updated based on the feature attention map of the modulation network.

[0068] refer to Figure 6 The described embodiments can provide improved architectures based on increased network depth and additional pattern-gated graphs. Therefore, the embodiments described herein can provide a scalable architecture compared to existing technologies. As those skilled in the art will understand, existing technologies are based on simple, shallow networks, which may fail when applied to deep neural networks. Therefore, the embodiments described herein can be applied to deep neural networks.

[0069] The embodiments described herein may also provide a bidirectional guided adaptive module, as referenced in [reference]. Figure 8A As described, this makes the current state of the prediction network known to the neural modulation network 602. The current state of the prediction network can refer to the degree to which the prediction network has been learned (e.g., the weights of the layers in the backbone 606), with each backbone layer transmitting the degree to which the prediction network has been learned to the corresponding modulation layer, as described herein.

[0070] Figure 7 An image classification model provided by an embodiment of the present invention is illustrated. The image classification model 700 may be similar to a prediction network 604. Model 700 may include a backbone 606 and a classifier 608 comprising multiple nodes. Each node may be associated with a corresponding unique data class.

[0071] In some embodiments, the backbone 606 may include stacked convolutional layers. In some embodiments, the backbone 606 may include stacked convolutional layers followed by fully connected layers. The backbone 606 may be referred to as a feature extractor.

[0072] In one embodiment, input 601 can be fed into backbone 606. Backbone 606 can generate a feature map or feature attention map based on the input image. The generated feature attention map or feature map can indicate one or more nodes 704, each node indicating a data class (e.g., image). The feature map or feature attention map can be fed into classifier 608. Classifier 608 can include one or more fully connected (FC) nodes (e.g., 704 and 706), each node indicating a data class.

[0073] In one embodiment, a new FC (Full-Complete) classification node (e.g., 706) can be added to the classifier for training based on training data (e.g., input 601). The number of newly added classification nodes can be based on the number of classes present in the training data, since each node can correspond to a specific class. Each newly added node can be fully connected to existing nodes, as shown in the figure.

[0074] Figure 8A A bidirectional guided adaptive module provided by an embodiment of the present invention is illustrated. Module 800 can reflect the process performed by each pair of corresponding layers (one layer from the backbone 606, and the other from the modulation network 602), as shown in the reference. Figure 6 As stated above.

[0075] In one embodiment, the weight attention map (indicating weight information) of layer 806 of backbone 606 can be sent (840) to the corresponding layer 802 of modulation network 602. The corresponding layer 802 can update its weight parameters based on the received weight attention map. In some embodiments, the update can be a multiplication operation. Subsequently, in some embodiments, the corresponding layer 802 can generate a feature attention map based on its updated weights and the received input 601. Similarly, layer 806 can generate a feature map based on its associated weights and the received input (e.g., input 601). The generated feature attention map of layer 802 can be applied, for example, by multiplication 842 to the generated feature map of layer 806, as shown.

[0076] refer to Figure 6 and Figure 8A Predict the network parameters θ P (For example, at each layer) this is processed by several fully-connected (FC) layers to generate a weight attention map. The weight attention map is then applied to the weights θ of the modulated network. M (For example, at each layer). The modulation network can receive new knowledge (new class of data) and generate feature attention maps to be applied (e.g., by multiplication 842) to the feature maps of the prediction network (at each layer).

[0077] Accordingly, the prediction parameter θ PThe current state of the prediction network can be provided (e.g., old knowledge it has already learned). The modulation network can combine the current state of the prediction network with new knowledge to adapt. Adaptation can refer to guiding the prediction network to update.

[0078] Figure 8B A flowchart for bidirectional guided adaptation for the first layer is shown, according to an embodiment of the present invention. Figure 8B The flowchart can reflect the reference Figure 8A and Figure 6 The described embodiments.

[0079] In one embodiment, at position 854, the prediction layer θ can be processed by a convolutional layer. P The weights. For example, the prediction layer can refer to... Figure 8A Layer 806. At 856, a weighted attention map can be generated, which represents the weights of the prediction layer.

[0080] In one embodiment, at 858, the modulation layer θ can be updated based on the weighted attention map generated by the prediction layer. M The weights of the modulation layer can be updated by multiplying the modulation layer weights by the weight attention map generated by the prediction layer. The modulation layer refers to the corresponding layer in the modulation network, for example, Figure 8A Layer 802 in the middle.

[0081] In one embodiment, at 860, the modulation layer receives input (e.g., input 601) to generate a feature attention map. At 864, the feature attention map associated with the modulation layer can be sent to the prediction layer. At 866, the prediction layer can also receive input (e.g., input 601) to generate a feature map based on the weights of the prediction layer.

[0082] In one embodiment, at 868, the feature map generated by the prediction layer can be updated by multiplication based on the feature attention map generated by the modulation layer. At 870, the updated feature map is then passed to the next layer.

[0083] Those skilled in the art will understand that a base class refers to a large amount of data for each class. Training based on base classes can yield well-trained base models. Base class training can be similar to offline training (freely collecting large amounts of data) to develop an effective classifier for the trained classes. Since test data is unavailable, continuous learning can be simulated to evaluate the trained model; this simulation can be called pseudo-meta few-shot class incremental learning (Pseudo Meta FSCIL). Pseudo-meta FSCIL can be based on... Figure 9 The method shown.

[0084] Figure 9 This paper illustrates a pseudo-meta few-shot class incremental learning (FSCIL) method provided by embodiments of the present invention. As described herein, since the current incremental step is only available at test time, the embodiments described herein can use a base class to simulate test incremental scenarios to train the model.

[0085] refer to Figure 9 The input to method 900 can include a learning rate (α, β). The input can also include a prediction network (θ). P ) and modulation network (θ) M The pre-trained weights are derived from a base class. The input can also include the base class (D). 0 The training set (e.g., images that can be acquired offline).

[0086] Method 900 may include initializing the model with pre-trained weights at 901. In some embodiments, the method may further include sampling the dataset from the base class at 903 to simulate the incremental step. As described herein, base data can be used to simulate sequential data for incremental learning. Sequential data D s It can include pseudo-training images (S j ) and query test image (Q j Therefore, the model can be based on pseudo-training images (S). j Training is performed on the test image (Q). j The test was conducted on [the platform]. For each training cycle, the session sequence was first sampled, with each session including the new class to be learned.

[0087] The method may also include resetting (e.g., clearing) the accumulation set (e.g., being cleared) at 904. In some embodiments, the method may include training a model based on operations from 905 to 913. Operations 906 to 912 may be performed for each of N sessions.

[0088] In some embodiments, at position 906, n new classifier FC nodes can be added to the classifier (e.g., classifier 606 for a prediction network). The number of n new classifier FC nodes can be based on the number of new classes in each session. For example, if the sequence includes a 5-way 5-shot classification problem (indicating 5 new classes, each with 5 images), then 5 new classifier FC nodes can be added to the classifier. The n new classifier FC nodes can be randomly initialized. Thereafter, only these n new classifier FC nodes can be trained over several epochs.

[0089] At point 907, new category FC nodes can be combined with old category FC nodes, for example, through cascading. Therefore, new nodes can be merged with old nodes.

[0090] In some embodiments, after merging the new node with the old node, the entire model can be updated K times to adapt to the new session class, as described in operations 908 to 910. The updates can be based on pseudo-training images (S... j The pseudo-training images may include (Reference Input) and Y j s (Refer to the input labels). After updating the entire model, it is assumed that the model has learned new knowledge at session j. Therefore, the adapted model is now capable of classifying the new class.

[0091] In some embodiments, the adapted model can be evaluated based on the forgetting problem. Therefore, at 911, the test set (query set) of the current session at session j can be accumulated into the accumulation set. Since training is based on new images at time t, it should be evaluated based on all classes (old and new knowledge) that the model has already observed; therefore, all query images are accumulated. Subsequently, at 912, the loss is calculated based on the accumulation set to measure the forgetting problem. In some embodiments, the original model θ can then be updated. P,M The original model can be updated based on the calculated gradient. Therefore, the meta-parameters θ can be updated based on the forgetting evaluation. P,M The original model, which includes a prediction network and a modulation network, refers to the model before adaptation (i.e., the model before 908).

[0092] In some embodiments, method 900 is based on bilayer meta-learning, which refers to 909 and 912. Bilayer meta-learning refers to training an augmented model based on updates according to 909 and further updates according to 912. The method can measure the forgetting level of the adapted model. Therefore, the method can ensure that the dilemma of adaptation and forgetting can be minimized.

[0093] As described herein, the embodiments can simulate incremental learning. At each time step t, n new class-FC nodes can be added to the classifier (e.g., classifier 606), where n can be based on the new class on which the model will be trained at time step t (e.g., in each session j). Once the new nodes are added, the model (e.g., the new class-FC nodes) can be trained several times, for example, K times. As those skilled in the art will understand, the number of times the model can be trained at each session (e.g., K times) can be based on the number of times to avoid overfitting.

[0094] As described above, in some embodiments, the model can be trained K times based on new knowledge (new class) at each session j. Since the model may have a tendency to forget old knowledge, in some embodiments, the trained model can be evaluated based on old knowledge to minimize the forgetting problem. Therefore, after training, data from both the old and new classes is accumulated for testing the model. In some embodiments, the adapted model can be evaluated based on the forgetting problem, i.e., based on data from old knowledge (old class). In some embodiments, the adapted model can be evaluated based on new knowledge to assess the model's adaptation to the new knowledge.

[0095] Based on the performance of the adapted model on the old knowledge, the original model can be updated (e.g., the weight parameters can be updated).

[0096] Figure 10 A flowchart of the pseudo-element FSCIL provided in an embodiment of the present invention is shown. Flowchart 100 can reflect the reference... Figure 9 The described embodiments.

[0097] At point 1002, the data sequence can be sampled to simulate continuous learning. The embodiment described with reference to 1002 can reflect the embodiment described with reference to 903.

[0098] For each time step in the sequence, at 1040, one or more operations described in references 1004, 1006, 1008, 1010, 1012, 1014 and 1016 can be performed.

[0099] At point 1004, a new FC (Full-Complete) node for classification can be added to the classifier and training can begin. The embodiment described with reference to 1004 reflects the embodiment described with reference to 906. At point 1006, similar to the embodiment described with reference to 907, the newly added node can be merged with the new node.

[0100] In some embodiments, at 1008, the entire model can be updated K times using a small number of new images (indicating new classes) fed at that time step. The embodiments described with reference to 1008 may reflect the embodiments described with reference to 908 and 909.

[0101] In some embodiments, at 1010, a query (test) set can be accumulated, similar to the embodiment described with reference to 911. The accumulated query set can then be used to evaluate the adapted or updated model. At 1014, a loss, such as cross-entropy loss, can be calculated. As those skilled in the art will understand, the loss can reflect the ability to cope with forgetting old knowledge and adapting to new knowledge. At 1016, the original model can be updated based on the calculated loss.

[0102] At point 1018, the same process can be performed for the next time step, such as process 1040. As those skilled in the art will understand, the entire process can be repeated for some cycles, such as 100 cycles.

[0103] Figure 11 Experimental results provided by embodiments of the present invention are shown. As illustrated, replacing the current method (e.g., state-of-the-art, SOTA) with a metal-trained backbone 606 can improve performance. The experimental results (i.e., the numbers shown in the table) are the top-1 accuracy at the final session on CIFAR100 (a dataset for classification). The baseline represents the model pre-trained only on the base classes and evaluated directly in the incremental step. PML represents the model pre-trained on the base classes and evaluated using a reference class. Figure 9 The described embodiments further train the model. Figure 11 In the code, "+4 mask" indicates that neural modulation was applied to the prediction network at four different locations. "+guided adaptive" indicates that bidirectional guided neural modulation has been added to the prediction network.

[0104] The embodiments described herein can provide enhanced model optimization involving less human engineering compared to existing techniques. The embodiments described herein can provide a fully learning training approach. Full learning indicates that neural modulation is learned to execute the weight selection mechanism described herein (rather than human engineering). The embodiments described herein can provide learnable layers (e.g., each layer is learnable), resulting in improved performance in terms of overfitting (for new classes) and forgetting old knowledge. The embodiments described herein can provide a balanced trade-off between forgetting and adaptation. The embodiments described herein can provide an automatic weight selection mechanism.

[0105] The embodiments described herein can provide a bidirectional guided adaptive module that can use a modulation network to guide the updates of a prediction network.

[0106] The embodiments described herein can provide simulated training in a sequential, round-by-round manner (i.e., incremental sessions). The embodiments described herein can provide evaluation in each incremental session based on new knowledge learned in the incremental session and old knowledge learned from previous sessions. Therefore, the embodiments can provide model optimization aligned with the evaluation. The embodiments described herein can provide two-layer optimization to address the forgetting problem after adaptation. Two-layer optimization refers to augmented training based on updates according to 909 and further updates according to 912.

[0107] The embodiments described herein can have applications where incremental classes have larger domain gaps compared to base training classes. This application reflects real-world scenarios where users can incrementally add new classes from different domains. Since the data collected for training cannot cover all users' classes, the model can be pre-trained to enable it to continue learning. The pre-trained model can then be incrementally trained based on data collected by users through manual annotation. Therefore, the pre-trained model can become a user-personalized model based on incremental learning from user data.

[0108] Figure 12 This is a schematic structural diagram of the system architecture provided in an embodiment of the present invention. As shown in system architecture 1200, data acquisition device 1260 is used to acquire data and store it in database 1230. The stored data may include training data and test data as described herein. Training device 1220 generates a target model or rule 1201 based on the training data maintained in database 1230. Target model or rule 1201 may refer to the trained network described herein, such as the backbone network, prediction network, and other networks used in different embodiments herein.

[0109] Optionally, one or more methods described herein can be processed by a CPU, by a combination of a CPU and a GPU, or by other processors suitable for neural network computations, without being processed by a GPU. This application does not impose any limitations on this.

[0110] The target model 1201 can be used for downstream tasks (e.g., one or more embodiments described herein). For example, a downstream task could be an image classification task. In one embodiment, the model's input data could be an input image, and the output could be a custom-trained model.

[0111] In some embodiments, the data maintained in database 1230 may not necessarily be collected by data acquisition device 1260, but may also be obtained from other devices via reception. It should be noted that training device 1220 may not necessarily train and optimize the target model or rule 1201 entirely based on the data maintained in database 1230, but may train the model using data obtained from the cloud or other locations. The above description should not be construed as a limitation on the embodiments of the present invention.

[0112] The target module / rule 1201 acquired through training by the training device 1220 can be applied to different systems or devices, such as the execution device 1210. The execution device 1210 can be a terminal, such as a mobile terminal, tablet computer, laptop computer, AR / VR device, in-vehicle terminal, etc., or it can be a server, cloud, etc. The execution device 1210 is equipped with an I / O interface 1212 for data interaction with external devices. Users can input data to the I / O interface 1212 through the client device 1240.

[0113] The preprocessing module 1213 can be used to preprocess inputs received from the I / O interface 1212.

[0114] During the preprocessing of input data by the execution device 1210 or the calculation module 1211 in the execution device 110, the execution device 1210 can call data, code, etc. from the data storage system 1250 for corresponding processing, or store the data, instructions, etc. obtained through the corresponding processing in the data storage system 1250.

[0115] I / O interface 1212 can return processing results to client device 1240 and provide the processing results to the user. It should be noted that training device 1220 can generate corresponding target models / rules 1201 based on different training data for different objectives or different tasks (the tasks described in this paper).

[0116] In some embodiments, a user can manually specify input data by performing operations on the screen provided by I / O interface 1212. In another case, client device 1240 can automatically send input data to I / O interface 1212. If client device 1240 needs to automatically send input data, it needs to obtain user authorization. The user can specify the corresponding permissions in client device 1240. The user can view the results output by execution device 1210 in client device 1240. Specific presentation formats can include displayed content, voice, actions, etc. Additionally, user device 1240 can act as a data collector, collecting the input data input to I / O interface 1212 and the output results output by I / O interface 1212 as new sampled data, and storing the new sampled data in database 1230. User device 1240 may not collect data, but I / O interface 1212 can directly store the input data input to I / O interface 1212 and the output results output from I / O interface 1212 as new sampled data in database 1230.

[0117] It should be noted that, Figure 12This is merely a schematic diagram of the system architecture provided by an embodiment of the present invention. The positional relationships between devices, components, modules, etc., shown in the figure do not constitute any limitation. For example, the data storage system 1250 is an external memory relative to the execution device 1210. In another case, the data storage system 1250 may be located within the execution device 1210.

[0118] One aspect of the present invention provides a method. The method includes: acquiring at least one weight attention map of a first network. The method further includes: updating the weights of a second network using the at least one weight attention map of the first network, wherein the second network is a modulation network. The method further includes: generating at least one feature attention map of the second network based on the at least one weight attention map of the first network and an input image set of at least one class. The method further includes: generating at least one feature map of the first network based on the input image set of the at least one class. The method further includes: updating the at least one feature map of the first network based on the feature attention map of the second network.

[0119] In some embodiments, the method further includes: generating a first weight attention map of a first layer of the first network, the first weight attention map representing weight parameters associated with the first layer of the first network. In some embodiments, the method further includes: updating the weight parameters of a corresponding first layer of a second network based on the first weight attention map. In some embodiments, the method further includes: generating a first feature attention map based on the updated weight parameters of the corresponding first layer and the input image set. In some embodiments, the method further includes: generating a first feature map based on the weight parameters of the first layer and the input image set. In some embodiments, the method further includes: generating a second feature map based on a multiplication of the first feature attention map and the first feature map. The method can provide learnable layers with improved performance in terms of overfitting and forgetting old knowledge.

[0120] In some embodiments, the method further includes: generating a corresponding weight attention map for each remaining layer of the first network, the corresponding weight attention map representing the weight parameters of each remaining layer of the first network. In some embodiments, the method further includes: updating the weight parameters of each remaining layer of the second network, and updating the corresponding weight attention map for each remaining layer of the second network based on the corresponding layer in the remaining layers of the first network. In some embodiments, the method further includes: generating a corresponding feature attention map associated with each remaining layer of the second network. In some embodiments, the method further includes: generating a corresponding feature map associated with each remaining layer of the first network. In some embodiments, the method further includes: generating a corresponding combined feature map based on the corresponding feature attention map and the corresponding feature map.

[0121] A convolutional neural network (CNN) is a deep neural network with a convolutional structure, and it is a deep learning architecture. Deep learning architectures represent multi-layered learning through machine learning algorithms at different abstraction layers. As a deep learning architecture, a CNN is a feed-forward artificial neural network. Each neuron in a feed-forward artificial neural network can respond to its input (e.g., an image).

[0122] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the above-described systems, devices and units will be referred to one or more corresponding embodiments described herein, and will not be repeated here.

[0123] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not performed. Furthermore, the communication connections shown or described can be implemented through some interfaces. Indirect communication connections between apparatuses or units can be implemented electronically, mechanically, or otherwise.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiment solution according to actual needs.

[0125] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0126] When these functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this invention, or parts contributing to the prior art, or some of the technical solutions, can be implemented in the form of a software product. This software product can be stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes any medium capable of storing program code, such as a USB flash drive, a removable hard disk, read-only memory (ROM), random-access memory (RAM), a magnetic disk, or an optical disk.

[0127] Embodiments of the present invention can be implemented using electronic hardware, software, or any combination thereof. In some embodiments, the present invention is implemented by one or more computer processors executing program instructions stored in memory. In some embodiments, the present invention is implemented in part or in whole in hardware, for example, using one or more field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs) to rapidly perform processing operations.

[0128] The actions associated with the method described herein can be implemented as coded instructions in a single computer program product. In other words, a computer program product is a computer-readable medium on which software code is recorded, which executes the method when the computer program product is loaded into memory and executed on the microprocessor of a wireless communication device.

[0129] Furthermore, each operation of the method can be performed on any computing device such as a personal computer, server, or PDA, and based on one or more program units, modules, or objects, or a portion thereof, generated from any programming language such as C++ or Java. Additionally, each operation, or a file or object implementing each operation, can be performed by dedicated hardware or circuit modules designed for this purpose.

[0130] The above description is merely some specific implementations of this application and is not intended to limit the scope of protection of this application. Any variations or substitutions easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0131] It should be understood that although specific embodiments of the technology have been described herein for illustrative purposes, various modifications can be made without departing from the scope of the technology. Therefore, the specification and drawings are to be considered merely as an illustration of the invention as defined in the appended claims, and any and all modifications, variations, combinations, or equivalents falling within the scope of the invention are contemplated. In particular, computer program products or program elements, or program storage devices such as magnetic or optical, magnetic tape, or optical discs, are provided within the scope of this technology for storing machine-readable signals for controlling the operation of a computer according to the method of this technology and / or constructing some or all of its components according to the system of this technology.

[0132] Although the invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof can be made without departing from the invention. Therefore, the specification and drawings are to be regarded merely as an illustration of the invention as defined in the appended claims and any and all modifications, variations, combinations, or equivalents falling within the scope of the invention are to be considered.

Claims

1. A method for incremental learning of meta-few-shot classes, characterized in that, include: Obtain at least one weight attention map of the first network; The weights of the second network are updated using the at least one weight attention map of the first network, wherein the second network is a modulation network; Based on the at least one weight attention map of the first network and the input image set of at least one class, at least one feature attention map of the second network is generated. Based on the input image set of the at least one class, at least one feature map of the first network is generated; The first network updates at least one feature map based on the feature attention map of the second network.

2. The method according to claim 1, characterized in that, The first network is a prediction network.

3. The method according to claim 1, characterized in that, Updating the weights of the second network includes multiplying the weights of the second network by the at least one weight attention map.

4. The method according to claim 1, characterized in that, The at least one weighted attention graph represents the current state of the first network, and the current state indicates the knowledge that has been learned.

5. The method according to claim 1, characterized in that, The method further includes: Provide the first network with a training dataset indicating n new classes; Add n nodes to the classifier of the first network, wherein the classifier includes one or more existing nodes that indicate one or more learning classes in the at least one class, each of the existing nodes indicating a learning class; The n nodes are merged with one or more existing nodes through cascading; Update the first network based on the training dataset; The updated first network is evaluated based on the test data indicating the new class and the learned class.

6. The method according to claim 5, characterized in that, The training and test datasets are derived from sequential datasets.

7. The method according to claim 6, characterized in that, The sequential dataset and the input image set are derived from a base dataset indicating a base class, which includes the n new classes and the at least one class.

8. The method according to claim 1, characterized in that, Each of the first network and the second network is a deep neural network.

9. The method according to claim 5, characterized in that, The n new classes come from class domains that are different from the at least one class.

10. The method according to any one of claims 5-7, characterized in that, The training dataset represents an n-way k-shot classification problem.

11. The method according to any one of claims 5-7, characterized in that, The updated first network, as evaluated, includes: A loss function is generated for the updated first network based on the test data.

12. The method according to any one of claims 5-7, characterized in that, Updating the first network based on the training dataset includes: The weight parameters of the backbone of the first network are updated based on the training dataset.

13. The method according to claim 11, characterized in that, The method further includes: The weight parameters of the backbone of the first network are updated based on the generated loss function.

14. A device for incremental learning of meta-few-shot classes, characterized in that, The device includes at least one processor and at least one non-transitory machine-readable medium storing executable instructions, wherein, when the executable instructions are executed by the at least one processor, the device is configured to: Obtain at least one weight attention map of the first network; The weights of the second network are updated using the at least one weight attention map of the first network, wherein the second network is a modulation network; Based on the at least one weight attention map of the first network and the input image set of at least one class, at least one feature attention map of the second network is generated. Based on the input image set of the at least one class, at least one feature map of the first network is generated; The first network updates at least one feature map based on the feature attention map of the second network.

15. The device according to claim 14, characterized in that, The at least one weighted attention graph represents the current state of the first network, and the current state indicates the knowledge that has been learned.

16. The device according to claim 14, characterized in that, The at least one processor is further configured to: Provide the first network with a training dataset indicating n new classes; Add n nodes to the classifier of the first network, wherein the classifier includes one or more existing nodes that indicate one or more learning classes in the at least one class, each of the existing nodes indicating a learning class; The n nodes are merged with one or more existing nodes through cascading; Update the first network based on the training dataset; The updated first network is evaluated based on the test data indicating the new class and the learned class.

17. The device according to claim 16, characterized in that, The evaluation of the updated first network includes: generating a loss function for the updated first network based on the test data.

18. The device according to claim 17, characterized in that, The at least one processor is further configured to: update the weight parameters of the backbone of the first network based on the generated loss function.

Images