A small sample class incremental learning method integrating synaptic remodeling and episodic memory network

Abstract

This invention discloses a few-shot incremental class learning method that integrates synaptic remodeling and episodic memory networks. The invention supplements a few-shot dataset with new class-specific virtual data generated by an episodic memory network decoder to alleviate the data gap problem in such datasets. The invention employs a synaptic remodeling method to generate virtual data to address the data gap issue in few-shot datasets and designs a forgetting-resistant learning model based on an episodic memory network to solve the problem of catastrophic forgetting of categories. The synaptic remodeling method first performs deep analysis on the existing limited data to extract highly representative episodic features, and then generates new virtual data using these features. These virtual data retain the key information of the original data while introducing appropriate variations and diversity, thereby enriching the training set. In this way, the model can be trained on larger and richer datasets, significantly improving its generalization ability and prediction accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and machine learning, and specifically relates to a small sample category incremental learning method integrating synaptic remodeling and episodic memory networks. Background Art

[0002] In the field of machine learning, incremental learning is an important learning paradigm, which refers to the ability of a model to continuously update and optimize itself by gradually learning new data or categories. This approach is particularly important in dynamic environments and practical applications, because in many practical scenarios, data is acquired gradually and the model needs to continuously adapt to new information and tasks.

[0003] In existing conventional technologies, the amount of new data or category data may be very limited and cannot provide enough information to effectively update the model. In this case, the model may find it difficult to accurately learn new category features due to the lack of sufficient training data. In addition, in the incremental learning process, when the model continuously receives new data, if there is no appropriate mechanism to retain old data or knowledge, the model may lose its ability to recognize old data due to excessive adjustment of new data. This phenomenon seriously affects the long-term performance and stability of the model.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a small sample category incremental learning method integrating synaptic remodeling and contextual memory networks, thereby solving the problems raised in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] A small sample category incremental learning method integrating synaptic remodeling and episodic memory network, comprising the following steps:

[0008] S1: Obtain basic category image features, build a visual semantic model Pix2pix, and use the visual semantic model Pix2pix to divide the input image into visual feature vectors and semantic feature vector

[0009] S2: Visual features and semantic features are respectively extracted through the attention mechanism to extract basic category features, and the importance parameter matrix is ​​constructed according to the long-term remodeling of synapses to form the synaptic remodeling weight matrix w 1 ;

[0010] S3: The scenario feature encoder encodes the scenario features and further performs synaptic remodeling on the scenario features to form the scenario weight matrix w 2;

[0011] S4: Use contrast loss function to constrain the consistency of visual features and semantic features to form a unified expression space;

[0012] S5: For new category input images, repeat the above feature extraction process to obtain new visual features and semantic features. The visual features and semantic features of the new category are used as context features. The decoder of the context memory network is used to generate virtual data of the new category. Finally, the virtual data of the new category generated by the context memory network decoder is added to the small sample data set.

[0013] Optionally, before using the decoder of the context memory network to generate virtual data of a new category, the following steps are also included:

[0014] A1: By regularizing the classification prediction loss, an anti-forgetting learning model is constructed to reduce the classification prediction loss;

[0015] A2: After completing the training of the old task, the context memory network is used to extract the visual features f of all samples v , semantic features f c and the alignment feature f a , and construct the scenario feature space F of the category i , whose expression is:

[0016] A3: Freeze the common feature parameters of the feature extractor close to the model input as the basic category shared features, and save the shared features in the scenario matrix In front of the training task, the weight of the shared feature is changed by Δw epi Set to a small value;

[0017] A4: When performing a new category recognition task, extract new features and calculate the cosine distance with the vectors in the scene feature space to obtain the classification loss L;

[0018] A5: Calculate the forgetting rate after each task is learned, and erase the neurons or input data with smaller weights.

[0019] Optionally, save the shared features in the context matrix In front of the shared features, the shared features will be given priority and utilized. When training new tasks, the influence of shared features will not be masked by new features. By limiting the change of shared feature weights, the shared features will not be excessively modified during the training of new tasks.

[0020] Optionally, by the formula L = L class +αL sort +βL epi Calculate the L of the classification loss, where Lclass is the cross entropy loss, L sort is the generalization loss between the old class feature distribution and the new class feature distribution, L epi is the context loss between the new class features and the old class features, and α and β are regular factors used to balance the loss terms.

[0021] Optionally, by formula To calculate the cross entropy loss L class , where M is the number of categories, s is the sign function, and pi c is the predicted probability that the sample belongs to category C.

[0022] Optionally, by formula To calculate the generalization loss L between the old class feature distribution and the new class feature distribution sort , where represents the forgetting of the i-th task after learning the k-th task, a m,n V represents the classification accuracy of the model on the nth task after learning the mth task. i As the basic category benchmark, V k are the first K feature vectors of the new category.

[0023] Optionally, by formula To calculate the scene loss L between the new class features and the old class features epi , where C is the matrix stored in the scenario The number of categories in , Δw epi is the weight change between the new category and the corresponding category in the scenario matrix.

[0024] Optionally, an importance parameter matrix is ​​constructed based on the long-term remodeling of synapses to form a synaptic remodeling weight matrix w 1 The steps are:

[0025] The attention mechanism focuses on the most important part of the image, extracts more representative features, and weights and adjusts the selected features according to the long-term remodeling of synapses. In this process, a sparse feature parameter matrix is ​​constructed.

[0026] The characteristic parameters processed by the long-term synaptic remodeling are constructed into a sparse importance parameter matrix, and the importance parameter matrix is ​​constructed into a synaptic remodeling weight matrix w 1 .

[0027] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0028] The present invention adopts a synaptic remodeling method to generate virtual data to alleviate the data missing problem of small sample data sets, and designs an anti-forgetting learning model based on a situational memory network to solve the problem of catastrophic forgetting of categories. The synaptic remodeling method first conducts a deep analysis of the existing small amount of data, extracts highly representative situational features, and then generates new virtual data through these features. These virtual data retain the key information of the original data while introducing appropriate changes and diversity, thereby enriching the training set. In this way, the model can be trained on a larger and richer data set, significantly improving its generalization ability and prediction accuracy. The situational memory network introduces a memory module to save the key features and weights of previous tasks. When the model learns a new task, these memory modules can effectively retain and utilize the information of the old task, avoiding the phenomenon that the model over-adjusts the new task data and forgets the knowledge of the old task.

[0029] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0031] In the figure:

[0032] Figure 1 Block diagram for incremental learning of few-shot categories.

[0033] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0034] The present invention will now be described in further detail with reference to the accompanying drawings.

[0035] See also Figure 1 As shown, in this embodiment, a small sample category incremental learning method integrating synaptic remodeling and contextual memory network is provided, comprising the following steps

[0036] S1: Obtain basic category image features, build a visual semantic model Pix2pix, and use the visual semantic model Pix2pix to divide the input image into visual feature vectors and semantic feature vector By separating visual features from semantic features, the model is able to learn and utilize low-level visual information and high-level semantic information at the same time, thereby improving learning and generalization capabilities. Visual and semantic features generated by the Pix2pix model can be further used for data enhancement, especially in small sample learning, where more virtual samples can be generated to enrich the dataset. Separating and capturing features helps retain knowledge of old tasks in incremental learning and improves the model's ability to resist forgetting.

[0037] S2: Visual features and semantic features are respectively extracted through the attention mechanism to extract basic category features, and the importance parameter matrix is ​​constructed according to the long-term remodeling of synapses to form the synaptic remodeling weight matrix w 1 ; By extracting basic category features through the attention mechanism, the key parts of the image can be captured more accurately, and irrelevant information can be ignored, thereby improving the accuracy of feature extraction. The constructed importance parameter matrix can effectively screen out features with high importance in multiple tasks, retain key features, and enhance the memory capacity of the model. The synaptic remodeling weight matrix simulates the synaptic change process in biological neural networks, retains the weights of important features, and improves the generalization ability of the model in new tasks. The synaptic remodeling weight matrix is ​​sparse, that is, only a few features are given larger weights, which reduces the burden of calculation and storage and improves the training efficiency of the model.

[0038] S3: The scenario feature encoder encodes the scenario features and further reshapes the synapses of the scenario features to form the scenario weight matrix w 2 ;

[0039] It should be noted that the scenario feature encoder can capture multi-level information of features by encoding scenario features. This capture of multi-level information helps the model understand and represent input data at different levels, thereby improving the feature expression ability. The scenario feature encoder not only extracts rich feature information, but also forms a scenario weight matrix through synaptic remodeling. The scenario weight matrix retains important scenario features, which can provide effective references in subsequent tasks, thereby enhancing the memory ability of the model. When training new tasks, the scenario feature encoder and the scenario weight matrix can effectively transfer knowledge from old tasks to new tasks. This knowledge transfer improves the performance and generalization ability of the model in new tasks. By maintaining important features of old tasks and dynamically adjusting weights, the model keeps memory of old tasks while continuously learning new tasks, thereby achieving higher training efficiency and robustness. This mechanism enables the model to always maintain high performance and stability when dealing with multi-task learning and small sample data sets, reflecting the effective combination of synaptic remodeling and scenario memory network.

[0040] S4: Use contrast loss function to constrain the consistency of visual features and semantic features to form a unified expression space;

[0041] It should be noted that: through the contrast loss function, visual features and semantic features are forced to align. This alignment ensures that in a unified expression space, visual and semantic information can complement and verify each other, improving the consistency of features. The contrast loss function reduces feature noise and improves the purity and representativeness of features by penalizing the inconsistency of visual features and semantic features, thereby improving the training effect of the model. The contrast loss function can also enhance the distinguishability between features, making features of different categories more obvious and distinguishable in a unified expression space. By forming a unified expression space, the model can better cope with changes in different data distributions and maintain a high level of robustness.

[0042] S5: For new category input images, repeat the above feature extraction process to obtain new visual features and semantic features. The visual features and semantic features of the new category are used as context features. The decoder of the context memory network is used to generate virtual data of the new category. Finally, the virtual data of the new category generated by the context memory network decoder is added to the small sample data set.

[0043] It should be noted that by repeating the feature extraction process, new visual features and semantic features can be extracted and corresponding virtual data can be generated. These virtual data significantly increase the diversity and richness of the small sample data set. By supplementing the generated virtual data to the small sample data set, the deficiencies of the original data set can be made up, so that the model can learn and generalize better. The visual features and semantic features of the new category are used as context features, and virtual data is generated through the decoder of the context memory network, which helps the model to better understand and remember the new category features. Generating a large amount of virtual data can effectively perform data enhancement and reduce the overfitting phenomenon of the model during training, thereby improving the robustness and stability of the model. This method not only expands the scope of training data, but also ensures the consistency of the learning process of new features with the old task knowledge through efficient encoding and decoding of context features, so that the model can keep the memory of old tasks while constantly learning new tasks, and achieve higher learning efficiency and generalization ability. In this way, the model can achieve better results in dealing with multi-task learning and small sample data sets, and show stronger adaptability and stability.

[0044] In this embodiment, before using the decoder of the context memory network to generate virtual data of a new category, the method further includes:

[0045] A1: By regularizing the classification prediction loss, an anti-forgetting learning model is constructed to reduce the classification prediction loss;

[0046] It should be noted that regularization adds constraints to the classification prediction loss to prevent the model from making significant modifications to important features and knowledge of old tasks when learning new tasks, thereby reducing forgetting. Regularization helps the model retain knowledge from old tasks, which can be transferred and used in new tasks, thereby improving the generalization ability of the model and making it perform better when faced with new data. By regularizing the classification prediction loss, drastic changes in model parameters during training can be avoided, the stability of the model can be enhanced, and the training process can be ensured to be smoother and more reliable.

[0047] A2: After completing the training of the old task, the context memory network is used to extract the visual features f of all samples v , semantic features f c and the alignment feature f a , and construct the scenario feature space F of the category i , whose expression is:

[0048] It should be noted that using the episodic memory network to extract features and construct a scenario feature space can effectively reduce the model's forgetting of old task knowledge when learning new tasks and improve its anti-forgetting ability. By extracting visual features, semantic features, and alignment features and constructing a scenario feature space, the model can capture and utilize multi-level information, improve generalization capabilities, and perform better when handling new tasks. By aligning features, ensuring that visual features and semantic features remain consistent in a unified expression space helps improve the model's learning effect and feature utilization efficiency.

[0049] A3: Freeze the common feature parameters of the feature extractor close to the model input as the basic category shared features, and save the shared features in the scenario matrix In front of the training task, the weight of the shared feature is changed by Δw epi Set to a small value;

[0050] Freeze the common feature parameters of the feature extractor near the model input as shared features of the base category. This helps retain key features and ensures that the model does not forget these basic features during the training of new tasks. When training new tasks, set the weight changes of shared features to small values ​​to limit the weight update amplitude of these features. This prevents the model from over-modifying the features of old tasks, thereby reducing forgetting. By freezing and retaining shared features and limiting their weight changes, the model training process is more stable. This can avoid the interference of new task training on old tasks and improve the stability of the overall model.

[0051] A4: When performing new category recognition tasks, new features are extracted and the cosine distance calculation is performed with the vectors in the scenario feature space to obtain the classification loss L; by extracting features of new categories, the model can obtain the latest and targeted feature information and improve the accuracy of the classification task. Using cosine distance calculation, the similarity between new features and scenario features can be effectively measured, thereby accurately evaluating the classification loss. Comparing new features with vectors in the scenario feature space helps maintain the consistency of feature representation and ensures that new and old task features are processed uniformly in the same space. By calculating the classification loss, the model parameters can be dynamically adjusted to perform better in new tasks. This adjustment helps the model adapt to new categories faster and improves overall learning efficiency.

[0052] A5: Calculate the forgetting rate after each task after learning, and erase the neurons or input data with smaller weights. Calculating and erasing neurons with smaller weights can reduce the existence of unimportant or redundant neurons, optimize the model structure, and save computing resources and storage space. Clearing input data that is no longer needed helps keep the data set clean and effective and avoids interference from redundant data. By calculating the forgetting rate after each task, the model can identify and retain the memory of important knowledge, reduce unnecessary forgetting, and improve generalization ability.

[0053] In this embodiment, the shared features are stored in the scenario matrix , so that the shared features will be given priority and utilized. When training new tasks, the impact of shared features will not be masked by new features, and by limiting the change of shared feature weights, the shared features will not be excessively modified during the training of new tasks. Storing shared features in front of the scenario matrix allows these features to be given priority and utilized when the model processes inputs. This ensures that the model first refers to the important features that have been learned, improving the accuracy and consistency of task processing. When training new tasks, the impact of shared features will not be masked by new features. This means that the key knowledge learned in old tasks will not be lost due to the addition of new tasks, thereby maintaining the consistency and coherence of knowledge. By limiting the change of shared feature weights, these features can be prevented from being excessively modified during the training of new tasks. In this way, the model can gradually adapt to the requirements of new tasks while maintaining the original knowledge. Limiting the change of shared feature weights also helps prevent the model from overfitting during the training of new tasks, because the model will not rely too much on new features, but will consider both new and old features comprehensively.

[0054] In this embodiment, by the formula L=L class +αL sort +βL epi Calculate the L of the classification loss, where L class is the cross entropy loss, L sortis the generalization loss between the old class feature distribution and the new class feature distribution, L epi is the contextual loss between new class features and old class features, and α and β are regularization factors used to balance the loss terms. By optimizing the comprehensive loss L, the model can find the best balance between multiple objectives and improve the overall training effect and performance. By introducing α and β as regularization factors in the formula, the influence of each loss term can be balanced to ensure that the model can not only accurately classify, but also effectively generalize between new and old tasks. By adjusting the values ​​of α and β, the weights of each loss term can be flexibly controlled to adapt to different task requirements and data characteristics.

[0055] In this embodiment, the formula To calculate the cross entropy loss L class , where M is the number of categories, s is the sign function, and pi c is the predicted probability that the sample belongs to category C.

[0056] In this embodiment, the formula To calculate the generalization loss L between the old class feature distribution and the new class feature distribution sort , where represents the forgetting of the i-th task after learning the k-th task, a m,n V represents the classification accuracy of the model on the nth task after learning the mth task. i As the basic category benchmark, V k is the first K feature vectors of the new category. By calculating the generalization loss L between the feature distribution of the old category and the feature distribution of the new category sort , which can effectively measure the generalization ability of the model between new and old tasks. By comparing the basic category features V i and the new category feature V k The distribution of can ensure that the model maintains stability and consistency between new and old tasks. By calculating the generalization loss and forgetting rate, the model can dynamically adjust the parameters during training to ensure the optimization and efficiency of the training process.

[0057] In this embodiment, the formula To calculate the scene loss L between the new class features and the old class features epi , where C is the scenario matrix stored in The number of categories in Δw epi is the weight change between the new category and the corresponding category in the context matrix. By calculating the context loss L between the new feature and the old feature epi , which can measure the consistency between new and old features. This helps ensure that the model maintains consistency with old task features when processing new tasks, improving overall stability. Calculate the weight change Δw between the new category and the corresponding category in the scenario matrixepi , which can effectively retain the key features of old tasks and reduce forgetting.

[0058] In this embodiment, an importance parameter matrix is ​​constructed based on the long-term remodeling of synapses to form a synaptic remodeling weight matrix w 1 The steps are:

[0059] The attention mechanism focuses on the most important part of the image, extracts more representative features, and weights and adjusts the selected features according to the long-term remodeling of synapses. In this process, a sparse feature parameter matrix is ​​constructed. Through the attention mechanism, the model can focus on the most important part of the image, thereby extracting more representative features. This helps to improve the accuracy of feature extraction and enables the model to more accurately capture the key information in the image.

[0060] The characteristic parameters processed by the long-term synaptic remodeling are constructed into a sparse importance parameter matrix, and the importance parameter matrix is ​​constructed into a synaptic remodeling weight matrix w 1 In the process of feature weighting and adjustment, a sparse feature parameter matrix is ​​constructed. This means that only a few features are given larger weights, and the weights of other features are smaller, thereby reducing the computational burden of the model and improving training efficiency.

[0061] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A small sample class incremental learning method integrating synaptic remodeling and episodic memory network, characterized in that: The following steps are involved: S1: Obtain basic category image features, build a visual semantic model Pix2pix, and use the visual semantic model Pix2pix to divide the input image into visual feature vectors and semantic feature vector S2: Visual features and semantic features are respectively used to extract basic category features through the attention mechanism, and the importance parameter matrix is ​​constructed according to the long-term remodeling of synapses to form the synaptic remodeling weight matrix w1; S3: The scenario feature encoder encodes the scenario features and further performs synaptic remodeling on the scenario features to form a scenario weight matrix w2; S4: Use contrast loss function to constrain the consistency of visual features and semantic features to form a unified expression space; S5: For the new category input image, repeat the above feature extraction process to obtain new visual features and semantic features. The new category visual features and semantic features are used as context features. The decoder of the context memory network is used to generate virtual data of the new category. Finally, the new category virtual data generated by the context memory network decoder is added to the small sample data set. Before using the decoder of the context memory network to generate virtual data of new categories, it also includes: A1: By regularizing the classification prediction loss, an anti-forgetting learning model is constructed to reduce the classification prediction loss; A2: After completing the training of the old task, the context memory network is used to extract the visual features f of all samples v , semantic features f c and the alignment feature f a , and construct the scenario feature space F of the category i , whose expression is: A3: Freeze the common feature parameters of the feature extractor close to the model input as the basic category shared features, and save the shared features in the scenario matrix In front of the training task, the weight of the shared feature is changed by Δw epi Set to a small value; A4: When performing a new category recognition task, extract new features and calculate the cosine distance with the vectors in the scene feature space to obtain the classification loss L; A5: Calculate the forgetting rate after each task after learning, and erase the neurons or input data with smaller weights; Save shared features in the context matrix In front of , the shared features will be given priority and utilized. When training new tasks, the influence of shared features will not be masked by new features. By limiting the change of shared feature weights, the shared features will not be excessively modified during the training of new tasks. By the formula L = L class +αL sort +βL epi Calculate the L of the classification loss, where L class is the cross entropy loss, L sort is the generalization loss between the old class feature distribution and the new class feature distribution, L epi is the context loss between the new class features and the old class features, and α and β are regular factors used to balance the loss terms.

2. A small sample category incremental learning method integrating synaptic remodeling and episodic memory network according to claim 1, characterized in that: By formula To calculate the cross entropy loss L class , where M is the number of categories, s is the sign function, and p ic is the predicted probability that the sample belongs to category C.

3. A small sample category incremental learning method integrating synaptic remodeling and episodic memory network according to claim 1, characterized in that: By formula To calculate the generalization loss L between the old class feature distribution and the new class feature distribution sort , where represents the forgetting of the i-th task after learning the k-th task, a m,n V represents the classification accuracy of the model on the nth task after learning the mth task. i As the basic category benchmark, V k are the first K feature vectors of the new category.

4. The small sample category incremental learning method integrating synaptic remodeling and episodic memory network according to claim 1, characterized in that: By formula To calculate the scene loss L between the new class features and the old class features epi , where C is the matrix stored in the scenario The number of categories in , Δw epi is the weight change between the new category and the corresponding category in the scenario matrix.

5. The small sample category incremental learning method integrating synaptic remodeling and episodic memory network according to claim 1, characterized in that: The steps of constructing the importance parameter matrix based on the long-term remodeling of synapses and forming the synaptic remodeling weight matrix w1 are as follows: The attention mechanism focuses on the most important part of the image, extracts more representative features, and weights and adjusts the selected features according to the long-term remodeling of synapses. In this process, a sparse feature parameter matrix is ​​constructed. The characteristic parameters that have undergone long-term synaptic remodeling processing are constructed into a sparse importance parameter matrix, and the importance parameter matrix is ​​constructed into a synaptic remodeling weight matrix w1.

Images