Small sample class incremental learning method based on robustness few-parameter fine tuning

Abstract

In the field of image recognition, small sample class incremental learning of a deep learning model is a challenging technical problem, and parameter iteration updating is performed on the basis of a pre-training model, so that the model can continuously learn new class knowledge only by using a limited number of labeled samples, and meanwhile, original class knowledge is reserved. The invention provides a small sample class incremental learning method based on robustness few-parameter fine tuning on the basis of a nearest neighbor class mean value classification method in order to solve the problem that parameter updating under extremely limited samples can cause deviation of a model and a large number of parameter updating can cause that the model forgets to have knowledge at the same time, and the invention provides the small sample class incremental learning method based on robustness few-parameter fine tuning on the basis of the nearest neighbor class mean value classification method. According to the method provided by the invention, efficient fine tuning of key parameters can be realized, and a flexible decision area is generated by controlling the number and amplitude of parameter adjustment, so that the problems are avoided.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition and image processing (G06V, G06F), and specifically relates to a small sample class incremental learning method based on robustness and few parameter fine-tuning. Background Art

[0002] Over the past decade, with the development of hardware equipment and the continuous optimization of network structure design [1] With the introduction of more data sets, traditional deep learning algorithms have shown impressive results in many tasks. [2,3] , such as image classification [4] , target detection [5] , semantic segmentation [6] These tasks that originally relied on a lot of artificial priors can now be completed efficiently with the help of deep learning integrated training methods, thus playing an important role in various scenarios of people's daily lives. In recent years, with the increasing maturity of traditional deep learning algorithms, many researchers have tried more complex generative, multimodal or 3D scene tasks, such as cross-modal generation (Vincent Figure [7] , Vincent Video [8] , 3D generation [9] Whether it is a traditional task with mature solutions or a newly proposed task that is still under research, their continuous progress is not only due to excellent network design, but more importantly, it relies on the support of a large amount of labeled training data. However, in many scenarios, the scenarios on which deep learning algorithms rely have the need for continuous expansion of knowledge, such as in the open set face recognition task.

[10] In this paper, the model is usually required to be able to recognize an increasing number of test subjects, so such deep learning algorithms usually need to have an update mechanism to continuously adapt to the new data distribution; in addition, in such scenarios, the number of samples used to update the algorithm is usually very scarce. On the one hand, this is because data collection is inherently difficult (for example, when updating the face recognition algorithm, usually only a small number of photos taken at the moment can be used). On the other hand, due to the high cost of real-time data annotation, the scarce labeled samples force people to reduce the number of samples required to train the deep learning algorithm to update itself. Taking into account the characteristics of the continuous expansion of knowledge and the scarcity of samples, researchers proposed the task of incremental learning of small sample classes.

[11] , exploring the use of only a small number of labeled samples to enable the algorithm to continuously update and recognize new object categories, and has received a certain degree of attention in the development of deep learning algorithms. In small samples, the number of single-category samples is small, for example, less than a specified number threshold.

[0003] The core challenge of small sample incremental learning technology is to overcome catastrophic forgetting.

[12] and overfitting

[13] In the current small sample incremental learning setting, the dataset can be divided into basic category data and incremental category data.

[14] , where each base category contains enough training samples for model initialization. In contrast, each incremental category only includes limited training samples to simulate incremental tasks. At the same time, when the model updates and learns incremental categories, the original training data involved in the previous task will no longer be available. Therefore, if general deep learning algorithms are used to directly learn incremental data, it will often lead to catastrophic forgetting of old category knowledge. As the number of updates increases, this forgetting phenomenon will continue to intensify, and eventually lead to complete forgetting of the base category or early incremental category, which is unbearable for algorithms that need to retain the original category knowledge as much as possible. In addition, since only a few training samples are available for incremental categories, if a small number of samples are used to update the model parameters, the general update method will usually lead to the model lacking sufficient generalization for incremental categories, and thus unable to learn new category knowledge through algorithm updates. Under the influence of catastrophic forgetting and overfitting, both the retention of original category knowledge and the extension of incremental category knowledge are very challenging.

[0004] Nearest neighbor classification method

[15] It has been proved by some work to be a better algorithm for dealing with small sample increment problems. Figure 1 As shown, there is room for optimization in the degree of cohesion and the location of cluster centers. On this basis, the present invention proposes to use a one-dimensional batch regularization mapping with adaptive parameters to achieve an approximate Gaussian distribution of features, and proposes a "similarity-based feature random reconstruction" method to generate a large number of pseudo features for new classes, and perform robust fine-tuning on the original cluster centers, thereby alleviating the bias and overfitting problems of the cluster centers. At the same time, the present invention introduces a low-parameter fine-tuning structure based on low-rank decomposition, and uses this structure as a branch of the backbone network to achieve robust updates of model parameters and enhance its cohesion ability. Since only a few key parameters are updated in this process, this parameter fine-tuning method can keep the original parameters of the model intact and has the ability to resist forgetting. Summary of the invention

[0005] The purpose of this invention is to study a small sample incremental model updating technology with low resource consumption and high universality based on the theories and methods of statistics and deep learning. It can update model parameters using extremely limited resources, so that it does not forget old knowledge while learning new knowledge, thereby completing the knowledge update.

[0006] This paper designs a small sample class incremental learning method based on robustness and few parameter fine-tuning, which is used to process image samples for image recognition. This technology has made innovations in many aspects, such as Gaussian distribution feature construction, incremental category feature reconstruction, and backbone network parameter update, thereby solving the pain points of other related technologies. Many experiments have shown that the technology proposed in this paper has obvious performance and efficiency advantages.

[0007] The present invention comprises the following steps:

[0008] Step S1: Gaussian distribution feature mapping;

[0009] Step S11: Perform one-dimensional batch normalization on the features output by the backbone network.

[0010] Step S12: adaptively scale the features obtained in S11.

[0011] Step S2: Extract basic category feature information and reconstruct small sample incremental category features;

[0012] Step S21: extract basic category features using step S1, calculate and record the sample mean and sample variance of each feature channel in the obtained basic category features.

[0013] Step S22: Use step S1 to extract incremental category features, calculate the similarity between the incremental category features and the mean of the basic category feature samples in S21, select multiple basic category replay features with high similarity, and reconstruct the small sample incremental category features.

[0014] Step S23: using the reconstructed features obtained in S31, updating the cluster center, and using the new cluster center for classification.

[0015] Step S3: fine-tuning the backbone network branch structure parameters;

[0016] Step S31: Initialize the low-parameter fine-tuning structure based on low-rank decomposition.

[0017] Step S32: Use a small number of samples in the incremental category and use the model structure initialized in S31 as the branch structure of the backbone network to fine-tune parameters.

[0018] Based on a thorough analysis of the deficiencies of other small sample incremental learning technologies, the present invention proposes a parameter fine-tuning technology with broader application prospects through two core technologies: cluster center update and backbone network update. Experimental results prove that the present invention has reached a very advanced level.

[0019] The beneficial effects of the present invention are:

[0020] 1. The present invention is based on the core idea of ​​robust parameter-less fine-tuning, which controls the location, quantity and degree of parameter updates so that model parameters can be updated efficiently within a limited range.

[0021] 2. The present invention studies the problem of small sample incremental learning in deep learning, which has important research significance and huge application value for large models and various AI applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the core idea of ​​the present invention.

[0023] Figure 2 It is a schematic diagram of basic category feature information extraction and incremental category feature reconstruction in the present invention.

[0024] Figure 3 Schematic diagram of the small sample incremental category feature reconstruction method of the present invention.

[0025] Figure 4 It is a schematic diagram of the fine-tuning structure with few parameters in the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in combination with specific implementation methods and with reference to the accompanying drawings. It should be understood that the embodiments provided below are only for the purpose of fully and comprehensively disclosing the present invention and fully conveying the technical concept of the present invention to those skilled in the art, but are not intended to limit the scope of the present invention.

[0027] Step S1: Gaussian distribution feature mapping;

[0028] In order to solve the problem of non-Gaussian distribution of feature distribution, the present invention proposes to use one-dimensional batch regularization and adaptive scaling method to map the features of arbitrary distribution output by the backbone network into an approximate Gaussian distribution that meets the requirements of the present invention.

[0029] Step S11: Perform one-dimensional batch normalization on the features output by the backbone network.

[0030] First, in small sample class incremental learning, the Resnet network structure is usually used

[16] As the backbone network, in this step, as shown in the following formula, the present invention uses the Resnet network model φ for feature extraction, where I is the knowledge (image) involved in this task, and y is the feature obtained through sample extraction.

[0031] y=φ(I)

[0032] Then, the feature y is mapped to a Gaussian distribution through one-dimensional batch normalization. Specifically, for an input feature batch {y(1) ,y (2) ,…,y (m)}, assuming that its dimension is d (i.e., there are d features), in one-dimensional batch normalization, calculate the mean μ and variance σ 2 The formula is shown below, where y ij Represents the j-th dimension feature value of the i-th feature of the batch, μ j and Represents the mean and variance of the jth dimension of the feature batch.

[0033]

[0034] Then normalize the features to get the normalized features Where ε is a small number (e.g. 10 -8 ) to prevent the denominator from being zero.

[0035] Step S12: Adaptively scale the features obtained in S11

[0036] Based on S11, this step introduces the parameters γ and β that can be automatically updated based on the gradient descent algorithm to the original Mapping is performed. The output after mapping is: The present invention hopes to map the features of each dimension in this way so that they conform to an approximately independent Gaussian distribution.

[0037] Step S2: Extract basic category feature information and reconstruct small sample incremental category features;

[0038] In this step, the present invention utilizes the features after Gaussian distribution mapping in step S1 to realize the recording of basic category features and the reconstruction of incremental category features, and based on the reconstructed features, realizes the robust fine-tuning of cluster center parameters, wherein the recording of basic category features and the reconstruction of incremental category features are as follows: Figure 2 shown.

[0039] Step S21: extract basic category features using step S1, calculate and record the sample mean and sample variance of each feature channel in the obtained basic category features.

[0040] In step S1, the present invention maps each feature dimension of the feature into a Gaussian distribution that is approximately independent and differently distributed. In this step, the present invention regards each channel feature value of each basic category as being sampled from an independent but non-identical one-dimensional data distribution, so that the sample mean and sample variance of each category i and channel j can be calculated in the basic category, where x i,j,k represents the eigenvalue of the kth sample of the jth channel of the i-th category, is the sample mean, is the corresponding sample variance, and m is the number of samples.

[0041]

[0042] For the basic category samples, the present invention records their sample mean and sample variance, and approximately regards them as the mean and variance of the feature population, that is,

[0043]

[0044] where μ i,j and are the mean and variance of the feature population of the jth channel of the i-th category, respectively.

[0045] Step S22: Use step S1 to extract incremental category features, calculate the similarity between the incremental category features and the mean of the basic category feature samples in S21, randomly select multiple high-similarity basic category replay features, and reconstruct the small sample incremental category features.

[0046] In this step, the mean value of the basic category feature recorded in step S21 is first used to calculate its similarity with the current incremental category feature. The feature extracted from the incremental category sample is x′, and the mean value of the i-th basic category feature is μ i , the similarity between the two is δ i , · represents the inner product, and ||·||2 represents the second-order norm.

[0047]

[0048] Then δ i Sort from high to low, select the first K′ as the high similarity category, randomly select K (K′>K) from them, and based on the feature mean μ i and variance σ i Perform feature sampling to achieve feature replay of the basic category, ~ represents Following this distribution, N(μ,σ 2 ) means the mean is μ and the variance is σ 2 And the dimensions are independent of each other (that is, the covariance matrix is ​​a diagonal matrix).

[0049]

[0050] Then use the sampled multiple base categories to replay the features For the incremental category original feature x′ i′ Reconstruction to obtain Where K is the number of high-similarity categories randomly selected above, i represents the i-th basic category, i′ represents the i′th incremental category, and j represents the j-th reconstructed feature under this category. is the feature mean of this category, such as Figure 3 shown.

[0051]

[0052] Step S23: Reconstruct features using the incremental categories obtained in S22, update cluster centers, and use the new cluster centers for classification.

[0053] In this step, the feature mean after reconstruction of the incremental category features is calculated as the cluster center parameter of the fine-tuned incremental category. Since the basic category itself has good clustering characteristics, there is no need to fine-tune the cluster center parameters.

[0054]

[0055] where c i′ represents the cluster center of the i'th incremental category, represents the jth reconstruction feature of the i'th incremental category, N represents the number of reconstruction samples of the i'th incremental category, and during testing, the features y and c of the test samples are used. i′ The distance between them determines the category l to which they belong, where y k and c i′,k are y and c respectively i′ The eigenvalue of the kth dimension.

[0056]

[0057] Step S3: fine-tuning the backbone network branch structure parameters;

[0058] In this step, after updating the cluster center parameters, a branch structure is built at a specific position of the backbone network, and the robustness of the backbone network is fine-tuned by adjusting the branch structure parameters to enhance the cohesion of the model. Figure 4 shown.

[0059] Step S31: Initialize the low-parameter fine-tuning structure based on low-rank decomposition.

[0060] When the model is adjusted for a new task, the change in its weights can be approximated by a decomposable low-rank matrix, rather than comprehensively adjusting all parameters of the entire model. In this step, the feature map output M after the penultimate layer module of the backbone network model is extracted. -2 And the feature map output M after the last level module -1 , the relationship between the two is M -1 =W -1 M -2 , where W -1 is the parameter weight of the last level module, that is, the parameter to be updated. Through low-rank decomposition, W -1 The change after update is expressed as ΔW-1 =BA, where B and A are two low-rank matrices whose ranks are much smaller than the dimension of the original weight matrix. In the actual algorithm, they are constructed by two 1×1 convolutions. In this way, during the fine-tuning process, only the parameters of the two low-rank matrices B and A need to be updated, without updating the entire W -1 , thus greatly reducing the number of trainable parameters.

[0061] When initializing the two low-rank matrices B and A, A is initialized using Xavier, while B is initialized with zero, ensuring that ΔW -1 =BA=0.

[0062] Step S32: Use a small number of samples in the incremental category to fine-tune the parameters of the model structure initialized by S31 as the branch structure of the backbone network. The branch structure adds new neurons without modifying the original network structure. During the incremental learning process, the backbone network parameters are frozen and only the parameters of the branch structure are updated.

[0063] On the basis of step S31, let the two low-rank one-dimensional convolution matrices B and A be bypassed, that is, W′ -1 =W -1 +ΔW -1 , where W -1 As the parameters of the last layer of the original backbone network remain unchanged, ΔW is updated using incremental category samples -1 The small number of parameters represented by can robustly obtain the updated parameters W′ -1 .

[0064] This paper applies the proposed model updating method to three public datasets CIFAR100

[17] CUB200

[18] and mini-imagenet

[19] The CIFAR100 and mini-imagenet datasets were iteratively updated 8 times, while the CUB200 was iteratively updated 10 times. In each round of multiple model updates, the present invention and other methods were improved. Specifically, the present method performed better than another widely used model parameter fine-tuning method MetaFSCIL on the mini-ImageNet, CIFAR100 and CUB200 datasets.

[20] In comparison, the average accuracies are 2.59%, 0.63%, and 5.14% higher respectively. In the last learning round, compared with MetaFSCIL, the accuracies of this method on the mini-ImageNet, CIFAR100, and CUB200 datasets are 2.22%, 1.38%, and 7.03% higher respectively.

[0065] Table 1 mini-Imagenet experimental indicators (unit: %)

[0066]

[0067] Table 2 cifar100 experimental indicators (unit: %)

[0068]

[0069] Table 3 cub200 experimental indicators (unit: %)

[0070]

[0071]

[0072] In summary, this method proposes a small-sample incremental learning method based on robust few-parameter fine-tuning. Experiments show that this technology has better efficiency and performance.

[0073] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

[0074] [1] Yin Xiaohang, Wang Yongcai, Li Deying. A review of medical image segmentation technology based on improved U-Net structure[J]. Journal of Software, 2021, 32(02): 519-550. DOI: 10.13328 / j.cnki.jos.006104.

[0075] [2] Guo Hua. Deep learning and its significance [J]. Curriculum. Textbooks. Teaching Methods, 2016, 36(11): 25-32. DOI: 10.19877 / j.cnki.kcjcjf.2016.11.005.

[0076] [3] Yu Kai, Jia Lei, Chen Yuqiang, et al. Deep learning yesterday, today and tomorrow[J]. Journal of Computer Research and Development, 2013, 50(09): 1799-1804.

[0077] [4] Zheng Yuanpan, Li Guangyang, Li Ye. A review of research on the application of deep learning in image recognition[J]. Computer Engineering and Applications, 2019, 55(12): 17.

[0078] [5] A review of vision-based object detection and tracking[J]. Acta Automatica Sinica, 2016, 42(10):24.

[0079] [6] Tian Xuan, Wang Liang, Ding Qi. A review of image semantic segmentation methods based on deep learning[J]. Journal of Software, 2019, 30(2): 29.

[0080] [7] Zhang Zeyu, Wang Tiejun, Guo Xiaoran, et al. A review of AI painting research[J]. Computer Science and Technology Research, 2024, 18(06): 1404-1420.

[0081] [8] Zhu Guanghui, Wang Xiwen. Core technology, operation mechanism and future scenarios of artificial intelligence video model Sora[J]. Journal of Xinjiang Normal University (Philosophy and Social Sciences Edition), 2024, 45(04): 149-156. DOI: 10.14100 / j.cnki.65-1039 / g4.20240320.001.

[0082] [9] Long Xiaoxiao, Cheng Xinjing, Zhu Hao, et al. Frontiers of 3D Vision[J]. Journal of Image and Graphics, 2021, 26(06): 1389-1428.

[0083]

[10] Xu Jiayao, Sun Yuan. Open scene face recognition system and its application[J]. Command Information System and Technology, 2014, 5(04): 39-42+47. DOI: 10.15908 / j.cnki.cist.2014.04.015.

[0084]

[11] Tao X, Hong X, Chang

[0085]

[12] I J Goodfellow,M Mirza,D Xiao,et al.An empirical investigation ofcatastrophicforgetting in gradient-based neural networks[J].arXiv preprintarXiv:13126211,2013.

[13] Y Zou,S Zhang,Y Li,et al.Margin-based few-shot class-incremental learning withclass-level overfitting mitigation[J].Advances inneural information processingsystems,2022,35:27267-27279.

[0086]

[14] Chen W Y,Liu Y C,Kira Z,et al.Acloser look at few-shotclassification[J].arXivpreprint arXiv:1904.04232,2019.

[0087]

[15] Shi G,Chen J,Zhang W,et al.Overcoming catastrophic forgetting inincrementalfew-shot learning by finding flat minima[J].Advances in neuralinformation processingsystems,2021,34:6747-6761.

[0088]

[16] He K,Zhang X,Ren S,et al.Deep residual learning forimagerecognition[C] / / Proceedings of the IEEE conference on computer visionand patternrecognition.2016:770-778.

[0089]

[17] Glorot X,Bengio Y.Understanding the difficulty of training deepfeedforwardneural networks[C] / / Proceedings of the thirteenth internationalconference on artificialintelligence and statistics.JMLR Workshop andConference Proceedings,2010:249-256.

[0090]

[18] Wah C,Branson S,Welinder P,et al.The caltech-ucsd birds-200-2011dataset[J].2011.

[0091]

[19] Russakovsky O,Deng J,Su H,et al.Imagenet large scale visualrecognitionchallenge[J].International journal of computer vision,2015,115:211-252.

[0092]

[20] Chi Z,Gu L,Liu H,et al.Metafscil:Ameta-learning approach for few-shot classincremental learning[C] / / Proceedings of the IEEE / CVF conference oncomputer visionand pattern recognition.2022:14166-14175.

Claims

1. A small sample class incremental learning method based on robust parameter fine-tuning, characterized in that: The following steps are involved: Step S1: Gaussian distribution feature mapping; Step S11: Perform one-dimensional batch regularization on the features output by the backbone network; Step S12: adaptively scaling the features obtained in S11; Step S2: extracting base class feature information and reconstructing incremental category features; Step S21: extracting base class features using step S1, calculating and recording the sample mean and sample variance of each feature channel in the obtained base class features; Step S22: extract incremental category features using step S1, calculate the similarity between the incremental category features and the mean of the base category feature samples in S21, select multiple base category replay features with high similarity, and reconstruct the incremental category features; Step S23: using the reconstructed features obtained in S31, updating the cluster center, and using the new cluster center for classification; Step S3: fine-tuning the backbone network branch structure parameters; Step S31: Initializing a fine-tuning structure with few parameters based on low-rank decomposition; Step S32: Use the model structure initialized in S31 as the branch structure of the backbone network to perform parameter fine-tuning.

2. The method according to claim 1, characterized in that In step S11, the one-dimensional batch regularization method is used to map the features of the arbitrary distribution output by the backbone network into a statistically significant Gaussian distribution, where φ is the backbone network parameter, I is the sample input to the backbone network, y is the extracted feature, and μ j and are the sample mean and variance of the j-th dimension of the one-dimensional batch normalization, y ij and are the sample features before and after the j-th dimension mapping of the i-th sample; y=φ(I) Here, ε is a small number used to prevent the denominator from being zero.

3. The method according to claim 1, characterized in that In step S12, the obtained features are adaptively scaled using learnable parameters γ and β, so that each feature dimension is mapped to a Gaussian distribution that is approximately independent and differently distributed, x ij is the feature after mapping, 4. The method according to claim 1, characterized in that: In step S21, the sample mean and sample variance of each feature channel in the obtained base class feature are calculated and recorded, where x i,j,k represents the eigenvalue of the kth sample of the jth channel of the i-th category, is the sample mean, is the corresponding sample variance, m is the number of samples, μ i,j and are the mean and variance of the feature population of the i-th category and the j-th channel respectively.

5. The method according to claim 1, characterized in that In step S22, the similarity δ between the incremental category feature and the mean of the base category feature samples in S21 is calculated i , then δ i Sort from high to low, select the first K′ as the high similarity category, randomly select K from them to replay the basic category features, K′>K, and reconstruct the incremental category features. First, the basic category feature mean μ i Similarity δ with the current incremental category feature x′ i , · represents the inner product, and ||·||2 represents the second-order norm Based on the feature mean μ i and variance σ i Perform feature sampling, ~ indicates Following this distribution, N(μ,σ 2 ) means the mean is μ and the variance is σ 2 Gaussian distribution Then use the sampled multiple base categories to replay the features For the original feature x′ of the incremental category i′ Reconstruction to obtain Where K is the number of high-similarity categories randomly selected above, i represents the i-th basic category, i′ represents the i′th incremental category, and j represents the j-th reconstructed feature under this category. is the feature mean of this category; 6. The method according to claim 1, characterized in that In step S23, the feature mean after incremental category feature reconstruction is calculated as the cluster center parameter after fine-tuning, where c i′ represents the cluster center of the i′th incremental category, represents the jth reconstruction feature of the i′th incremental category, N represents the number of reconstructed samples of the i′th incremental category, and uses the features y and c of the test sample i′ The distance between them determines the category to which they belong.

7. The method according to claim 1, characterized in that In step S31, a decomposable low-rank matrix is ​​used to approximate the parameter change when the model parameters are updated, and the parameter initialization process is performed using Xavier initialization and zero initialization respectively. ΔW is the parameter to be updated, and B and A are the two matrices after low-rank decomposition. ΔW=BA=0.

8. The method according to claim 1, characterized in that In step S32, the model structure initialized in S31 is used as the branch structure of the backbone network for parameter fine-tuning, where W is the original parameter and remains unchanged, and W′ is the parameter after the update. W′=W+ΔW.

9. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images