Small sample class incremental learning method based on robustness few-parameter fine-tuning

Abstract

In the field of image recognition, small sample class incremental learning of deep learning model is a challenging technical problem, which iteratively updates parameters based on pre-trained model, so that the model can continuously learn the knowledge of new classes only with a limited number of labeled samples, while retaining the knowledge of original classes. Parameter updating under extremely limited samples will cause the model to deviate, and a large number of parameter updates will cause the model to forget existing knowledge. In order to solve the above problems, on the basis of nearest neighbor class mean classification method, the application proposes a small sample class incremental learning method based on robustness and few parameters fine-tuning. The method can realize efficient fine-tuning of key parameters, and produce flexible decision region by controlling the number and amplitude of parameter adjustment, thereby avoiding the above problems.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image recognition and image processing (G06V, G06F), and particularly relates to a small sample class incremental learning method based on robustness fine-tuning of few parameters. BACKGROUND

[0002] In the past decade, with the development of hardware devices, the continuous optimization of network structure design [1] and the proposal of more data sets, traditional deep learning algorithms have shown impressive results in many tasks [2,3] , such as image classification [4] , object detection [5] , semantic segmentation [6] , etc. These tasks originally relying on a large amount of artificial priori can now be efficiently completed with the help of deep learning integrated training method, thereby playing an important role in various scenes of people's daily life; in recent years, with the gradual maturity of traditional deep learning algorithms, many researchers try more complex generative, multi-modal or 3D scene tasks, such as cross-modal generation (text-to-image [7] , text-to-video [8] , 3D generation [9] , etc.), and also make certain progress. Whether it is a traditional task with mature solutions or a newly proposed task still under research, their continuous progress is not only due to excellent network design, but more importantly, relies on the support of a large amount of labeled training data. However, in many scenarios, the scene relied on by deep learning algorithms has the demand of continuous knowledge expansion, for example, in the open set face recognition task

[10] , the model is usually required to recognize an increasing number of testers, so such deep learning algorithms usually need to have an updating mechanism to adapt to new data distribution; in addition, in such scenarios, the number of samples used for algorithm updating is usually very small, on the one hand, due to the natural difficulty of data collection (for example, in the face recognition algorithm updating, only a small number of photos can be used), on the other hand, due to the high cost of real-time data labeling, the small number of labeled samples forces people to reduce the number of samples required when training deep learning algorithms for updating themselves. Combining the characteristics of continuous knowledge expansion and small sample size, researchers propose the task of small sample class incremental learning

[11] , exploring the use of only a small number of labeled samples to continuously update the algorithm to recognize new object categories, and has received a certain degree of attention in the development of deep learning algorithms. In small samples, the number of single-class samples is small, for example, less than a specified number threshold.

[0003] The core challenge of small sample class incremental learning technology is to overcome catastrophic forgetting

[12] and overfitting

[13] In the current small sample class incremental learning setting, the data set can be divided into base class data and incremental class data

[14] Wherein each base class contains sufficient training samples for the initialization of the model, in contrast, each incremental class only includes limited training samples to simulate the incremental task, and meanwhile, when the model updates the incremental class, the original training data involved in the pre-task will no longer be available, therefore, if the general deep learning algorithm is used to directly learn the incremental data, it will often lead to catastrophic forgetting of the old class knowledge, and this forgetting phenomenon will continue to strengthen with the increase of the number of updates, eventually leading to complete forgetting of the base class or the early incremental class, which is unbearable for the algorithm which needs to retain the original class knowledge as much as possible. In addition, since only a small number of training samples are available for the incremental class, if the model is updated using a small number of samples, the general updating method will also often lead to insufficient generalization of the model to the incremental class, so that the algorithm cannot learn new class knowledge through updating. Under the influence of catastrophic forgetting and overfitting, both the retention of the original class knowledge and the extension of the incremental class knowledge are challenging.

[0004] Nearest neighbor classification method

[15] Some works have proved that it is a better algorithm to deal with the small sample class increment problem, however, as Figure 1 Indicated, the cohesion degree and the cluster center position have optimization space, on this basis, the application proposes to realize the approximate Gaussian distribution of the features by using one-dimensional batch regularization mapping with adaptive parameters, and proposes a "feature random reconstruction based on similarity" method, generates a large number of pseudo features for the new class, and robustly fine-tunes the original cluster center, thereby relieving the biased problem and the overfitting problem of the cluster center. At the same time, the application introduces a few-parameter fine-tuning structure based on low-rank decomposition, which is used as a branch of the backbone network to realize the robust updating of the model parameters and enhance the cohesion ability. Since only a few key parameters are updated in this process, this parameter fine-tuning method can completely retain the original parameters of the model and has the ability to resist forgetting. SUMMARY

[0005] The purpose of the application is to study a small sample incremental model updating technology with low resource consumption and high universality based on the theories and methods of statistics and deep learning, which can update the model parameters with extremely limited resources, so that the model can learn new knowledge while not forgetting old knowledge, thereby completing the updating of knowledge.

[0006] The application designs a small sample class incremental learning method based on robustness small parameter fine-tuning, which is used for processing image samples for image recognition.

[0007] The application comprises the following steps:

[0008] Step S1: Gaussian distribution feature mapping;

[0009] Step S11: one-dimensional batch regularization is performed on the features output by the backbone network.

[0010] Step S12: the features obtained in S11 are adaptively scaled.

[0011] Step S2: basic class feature information extraction and small sample incremental class feature reconstruction;

[0012] Step S21: the basic class features are extracted by step S1, and the sample mean and sample variance of each feature channel in the obtained basic class features are calculated and recorded.

[0013] Step S22: the incremental class features are extracted by step S1, the similarity between the incremental class features and the sample mean of the basic class features in S21 is calculated, a plurality of high-similarity basic class features are selected, and the small sample incremental class features are reconstructed.

[0014] Step S23: the reconstructed features obtained in S31 are used to update the clustering center, and the new clustering center is used for classification.

[0015] Step S3: backbone network branch structure parameter fine-tuning;

[0016] Step S31: low-rank decomposition-based small parameter fine-tuning structure initialization.

[0017] Step S32: a small amount of samples in the incremental class are used, and the model structure initialized in S31 is used as the branch structure of the backbone network for parameter fine-tuning.

[0018] Based on the analysis of the deficiencies of other small sample class incremental learning technologies, the application proposes a parameter fine-tuning technology with wider application prospect through the two core technologies of clustering center updating and backbone network updating, and the experimental results prove that the application has reached a very advanced level.

[0019] The application has the following advantages:

[0020] 1. The core idea of the present application is based on robust few-parameter fine-tuning, which controls the position, number and degree of parameter update, so that the model parameters can be updated efficiently within a limited range.

[0021] 2. The present application studies the small sample class incremental learning problem of deep learning, which has important research significance and great application value for large models and various AI applications. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The core idea of the present application is shown in the figure.

[0023] Figure 2 The figure shows the extraction of basic category feature information and the reconstruction of incremental category feature in the present application.

[0024] Figure 3 The figure shows the small sample incremental category feature reconstruction method in the present application.

[0025] Figure 4 The figure shows the few-parameter fine-tuning structure in the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer and more intelligible, the present application will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that the embodiments provided below are only for the purpose of fully and completely disclosing the present application, and fully conveying the technical concept of the present application to those skilled in the art, and are not intended to limit the scope of the present application.

[0027] Step S1: Gaussian distribution feature mapping;

[0028] In order to solve the problem of non-Gaussian distribution of feature distribution, the present application proposes to use one-dimensional batch regularization and adaptive scaling method to map the features output by the backbone network to approximate Gaussian distribution as desired by the present application.

[0029] Step S11: One-dimensional batch regularization is performed on the features output by the backbone network.

[0030] First, in small sample class incremental learning, the Resnet network structure is usually used

[16] As a backbone network, in this step, the present application uses the Resnet network model φ to extract features, I is the knowledge (image) involved in the task, and y is the feature obtained by sample extraction.

[0031] y = φ(I)

[0032] Then, the feature y is mapped to Gaussian distribution by one-dimensional batch regularization. Specifically, for an input feature batch {y(1) ,y (2) ,…,y (m) Assuming its dimension is d (i.e., it has d features), in one-dimensional batch regularization, the mean μ and variance σ are calculated. 2 The formula is shown below, where y ij μ represents the feature value of the j-th dimension of the i-th feature in this batch. j and This represents the mean and variance of the j-th dimension of the feature batch.

[0033]

[0034] Then the features are normalized to obtain the normalized features. Where ε is a very small number (e.g., 10). -8 ), used to prevent the denominator from being 0.

[0035] Step S12: Adaptively scale the features obtained in S11.

[0036] Building upon S11, this step introduces parameters γ and β, which can be automatically updated based on the gradient descent algorithm, to modify the original... Perform mapping. The output after mapping is: The present invention aims to map the features of each dimension in this way so that they conform to an approximately independent Gaussian distribution.

[0037] Step S2: Extraction of basic category feature information and reconstruction of incremental category features from small samples;

[0038] In this step, the present invention utilizes the features mapped by the Gaussian distribution in step S1 to record the basic category features and reconstruct the incremental category features. Based on the reconstructed features, robust fine-tuning of the cluster center parameters is achieved. The recording of basic category features and the reconstruction of incremental category features are as follows: Figure 2 As shown.

[0039] Step S21: Extract basic category features using step S1, and calculate and record the sample mean and sample variance of each feature channel in the obtained basic category features.

[0040] In step S1, the present invention maps each feature dimension of the feature to an approximately independent and distinct Gaussian distribution. In this step, the present invention further considers each channel feature value of each base category as being sampled from an independent but non-identically distributed one-dimensional data distribution. Therefore, in the base category, the sample mean and sample variance of each category i and channel j can be calculated, where x... i,j,k This represents the feature value of the k-th sample in the j-th channel of the i-th category. The sample mean. where m is the number of samples.

[0041]

[0042] For the base class samples, the sample mean and sample variance are recorded in the present application, and are approximated as the mean and variance of the feature population, i.e.

[0043]

[0044] where μ i,j and are the mean and variance of the feature population of the i-th class and j-th channel, respectively.

[0045] Step S22: Increment class features are extracted using step S1, the similarity of the increment class features and the sample mean of the base class features in S21 is calculated, and a plurality of base class replay features with high similarity are randomly selected to reconstruct the small sample increment class features.

[0046] In this step, first, the similarity of the base class feature mean recorded in step S21 and the current increment class feature is calculated. The feature extracted by the increment class sample is x', the i-th base class feature mean is μ i , and the similarity of the two is δ i ·, where · represents the inner product, and ||·||2 represents the second norm.

[0047]

[0048] Then δ i is sorted from high to low, the top K' are selected as high similarity classes, K (K'>K) are randomly selected from them, and feature sampling is performed based on the feature mean μ i and variance σ i , thereby realizing feature replay of the base class, ~ indicates obeys the distribution, N(μ,σ 2 ) indicates a Gaussian distribution with mean μ and variance σ 2 , and each dimension is mutually independent (i.e., the covariance matrix is a diagonal matrix).

[0049]

[0050] Then, the plurality of base class replay features sampled are used to reconstruct the increment class original feature x' i′ to obtain where K is the number of high similarity classes randomly selected above, i represents the i-th base class, i' represents the i'-th increment class, j represents the j-th reconstructed feature of the class, is the feature mean of the class, such asFigure 3 as shown.

[0051]

[0052] Step S23: update the cluster center using the incremental class feature reconstructed in S22, and use the new cluster center for classification.

[0053] In this step, the feature mean of the reconstructed incremental class feature is calculated as the fine-tuned cluster center parameter of the incremental class. Since the base class has good clustering characteristics, the cluster center parameter does not need to be fine-tuned.

[0054]

[0055] where c i′ represents the cluster center of the i'th incremental class, represents the j'th reconstructed feature of the i'th incremental class, and N represents the number of reconstructed samples of the i'th incremental class. In testing, the distance between the feature y of the test sample and c i′ is used to determine the class l to which it belongs, where y k and c i′,k are the k'th dimensional feature values of y and c i′ , respectively.

[0056]

[0057] Step S3: fine-tuning of the backbone network branch structure parameters;

[0058] This step builds a branch structure at a specific position of the backbone network after updating the cluster center parameters, and adjusts the branch structure parameters to achieve robust fine-tuning of the backbone network and enhance the cohesion of the model, as shown. Figure 4

[0059] Step S31: initialization of the few-parameter fine-tuning structure based on low-rank decomposition.

[0060] When the model is adjusted for a new task, the changes in its weights can be approximated by a decomposable low-rank matrix, rather than a comprehensive adjustment of all parameters of the entire model. In this step, the feature map output M -2 after the second-to-last level module of the backbone network model is extracted, and the feature map output M -1 after the last level module, and their relationship is M -1 = W -1 M -2 , where W -1 is the parameter weight of the last level module, i.e., the parameter to be updated. Through low-rank decomposition, the change in W -1 after updating is represented as ΔW​-1 = BA, where B and A are two low-rank matrices, whose ranks are much smaller than the dimension of the original weight matrix, in the actual algorithm, they are composed of two 1x1 convolutions, so in the fine-tuning process, only the parameters of the two low-rank matrices B and A need to be updated, and the whole W -1 , thereby greatly reducing the number of trainable parameters.

[0061] And when initializing the two low-rank matrices B and A, A uses Xavier initialization, and B uses zero initialization, to ensure that at the beginning of the update, AW -1 = BA = 0.

[0062] Step S32: using a small number of samples in the incremental category, the model structure initialized in S31 is used as the branch structure of the backbone network for parameter fine-tuning, and the branch structure is added as the basis for not modifying the original network structure. New neurons, in the incremental learning process, the parameters of the backbone network are frozen and only the parameters of the branch structure are updated.

[0063] On the basis of step S31, the two low-rank one-dimensional convolution matrices B and A are used as bypass, that is, W' -1 = W -1 + AW -1 , where W -1 , the last layer parameters of the original backbone network remain unchanged, and AW -1 , a small number of parameters represented by the incremental category samples, that is, the updated parameters W' -1 .

[0064] The model updating method proposed in the application is tested in three public data sets CIFAR100

[17] CUB200

[18] and mini-imagenet

[19] , wherein the CIFAR100 and mini-imagenet data sets are updated for 8 times, and the CUB200 is updated for 10 times, in each round of model multiple updates, the application and other methods are improved. Specifically, compared with another widely used model parameter fine-tuning method MetaFSCIL

[20] , the average accuracy is increased by 2.59%, 0.63% and 5.14% respectively, and in the last learning round, compared with MetaFSCIL, the accuracy of the application on the mini-ImageNet, CIFAR100 and CUB200 data sets is increased by 2.22%, 1.38% and 7.03% respectively.

[0065] Table 1 mini-Imagenet experimental index (unit: %)

[0066]

[0067] Table 2 cifar100 experimental index (unit: %)

[0068]

[0069] Table 3 cub200 experimental index (unit: %)

[0070]

[0071]

[0072] In summary, the method proposes a small sample class incremental learning method based on robustness and few parameters fine-tuning, and experiments show that this technology has better efficiency and performance.

[0073] The above is only a specific embodiment of the present application, any feature disclosed in the specification can be replaced by other equivalent or similar purpose alternative features unless specifically described, all features disclosed, or steps in all methods or processes can be combined in any way except mutually exclusive features and / or steps.

[0074] [1] Yin Xiaohang, Wang Yongcai, Li Deying. Review of Medical Image Segmentation Technology Based on Improved U-Net Structure[J]. Journal of Software, 2021, 32(02): 519-550. DOI:10.13328 / j.cnki.jos.006104.

[0075] [2] Guo Hua. Deep Learning and Its Significance[J]. Curriculum, Teaching Material, Teaching Method, 2016, 36(11): 25-32. DOI:10.19877 / j.cnki.kcjcjf.2016.11.005.

[0076] [3] Yu Kai, Jia Lei, Chen Yuqiang, et al. Yesterday, Today and Tomorrow of Deep Learning[J]. Computer Research and Development, 2013, 50(09): 1799-1804.

[0077] [4] Zheng Yuapan, Li Guangyang, Li Ye. Review of Deep Learning Application in Image Recognition[J]. Computer Engineering and Applications, 2019, 55(12): 17.

[0078] [5] Review of Vision-based Object Detection and Tracking[J]. Journal of Automation, 2016, 42(10): 24.

[0079] [6] Tian X, Wang L, Ding Q. Survey of image semantic segmentation methods based on deep learning[J]. Journal of Software, 2019, 30(2): 29.

[0080] [7] Zhang ZY, Wang TJ, Guo XR, et al. Survey of AI painting research[J]. Journal of Computer Science and Technology, 2024, 18(06): 1404-1420.

[0081] [8] Zhu GH, Wang XW. Core technology, operation mechanism and future scenarios of artificial intelligence text-to-video large model Sora[J]. Journal of Xinjiang Normal University (Philosophy and Social Sciences), 2024, 45(04): 149-156. DOI:10.14100 / j.cnki.65-1039 / g4.20240320.001.

[0082] [9] Long X X, Cheng X J, Zhu H, et al. Advances in 3D vision[J]. Journal of Image and Graphics, 2021, 26(06): 1389-1428.

[0083]

[10] Xu JY, Sun Y. Open scene face recognition system and its application[J]. Command Information System & Technology, 2014, 5(04): 39-42+47. DOI:10.15908 / j.cnki.cist.2014.04.015.

[0084]

[11] Tao X, Hong X, Chang X, et al. Few-shot class-incremental learning[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 12183-12192.

[0085]

[12] I J Goodfellow, M Mirza, D Xiao, et al. An empirical investigation of catastrophic forgetting in gradient-based neural networks [J]. arXiv preprint arXiv:13126211, 2013.

[13] Y Zou, S Zhang, Y Li, et al. Margin-based few-shot class-incremental learning with class-level overfitting mitigation [J]. Advances in neural information processing systems, 2022, 35: 27267-27279.

[0086]

[14] Chen W Y, Liu Y C, Kira Z, et al. A closer look at few-shot classification [J]. arXiv preprint arXiv:1904.04232, 2019.

[0087]

[15] Shi G, Chen J, Zhang W, et al. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima [J]. Advances in neural information processing systems, 2021, 34: 6747-6761.

[0088]

[16] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.

[0089]

[17] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks [C] / / Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010: 249-256.

[0090]

[18] Wah C, Branson S, Welinder P, et al. The caltech-ucsd birds-200-2011 dataset [J]. 2011.

[0091]

[19] Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge [J]. International journal of computer vision, 2015, 115: 211-252.

[0092]

[20] Chi Z, Gu L, Liu H, et al. MetaFSCIL: A meta-learning approach for few-shot class incremental learning [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 14166-14175.

Claims

1. A small sample class incremental learning method based on robustness few-parameter fine-tuning, used for processing image samples for image recognition, characterized in that, The method comprises the following steps: Step S1: Gaussian distribution feature mapping; Step S11: one-dimensional batch normalization is performed on the features output by the backbone network; Step S12: the features obtained in S11 are adaptively scaled; Step S2: base class feature information extraction and incremental class feature reconstruction; Step S21: base class features are extracted by using step S1, and the sample mean and sample variance of each feature channel in the obtained base class features are calculated and recorded; Step S22: incremental class features are extracted by using step S1, the similarity between the incremental class features and the sample mean of the base class features in S21 is calculated, a plurality of high-similarity base class replay features are selected, and the incremental class features are reconstructed; Step S23: the reconstructed features obtained in S31 are used to update the clustering center, and the new clustering center is used for classification; Step S3: backbone network branch structure parameter fine-tuning; Step S31: low-rank decomposition-based few-parameter fine-tuning structure initialization; Step S32: the model structure initialized in S31 is used as the branch structure of the backbone network for parameter fine-tuning, wherein In step S11, the one-dimensional batch regularization method is used to map the arbitrary distribution of the feature output by the backbone network to a Gaussian distribution with statistical significance, φ is the backbone network parameter, I is the sample input into the backbone network, y is the extracted feature, μ j and are the sample mean and variance of the jth dimension of the one-dimensional batch regularization, respectively, y ij and are the sample features before and after mapping of the jth dimension of the ith sample, respectively. y = φ (I) Where ε is a very small number, used to prevent the denominator from being 0.

2. The method of claim 1, wherein, In step S12, the obtained features are adaptively scaled using learnable parameters γ and β, so that each feature dimension is mapped to a nearly independent and differently distributed Gaussian distribution, x ij are the mapped features, 3. The method of claim 1, wherein, In step S21, the sample mean and sample variance of each feature channel in the obtained base class features are calculated and recorded, where x i,j,k represents the feature value of the i-th category, j-th channel and k-th sample, is the sample mean, is the corresponding sample variance, m is the sample number, μ i,j and are the mean and variance of the feature population of the i-th category and j-th channel, respectively 4. The method of claim 1, wherein, In step S22, the similarity δ between the incremental class feature and the mean value of the base class feature sample in S21 is calculated i Then δ i Sort from high to low, select the first K' as the high similarity class, randomly select K from it to replay the base class feature, K'>K, and reconstruct the incremental class feature, first the mean value μ i The similarity δ between the current incremental class feature x' and the base class feature i · represents the inner product, and ||·||2 represents the second order norm Based on the feature mean μ i and variance σ i Perform feature sampling, ~ denotes obeys this distribution, N(μ,σ 2 ) denotes a Gaussian distribution with mean μ and variance σ 2 ​ Then the multiple base category replay features after sampling are utilized to the incremental category original feature x' i′ to obtain the reconstruction wherein K is the number of high similarity categories randomly selected above, i represents the i-th base category, i' represents the i'-th incremental category, and j represents the j-th reconstruction feature under the category, is the category feature mean, represents the j-th reconstruction feature of the i'-th incremental category; 5. The method of claim 1, wherein, In step S23, the feature mean value of the reconstructed features of the incremental class is calculated as the fine-tuned cluster center parameter, where c i′ represents the cluster center of the i'th incremental class, represents the j'th reconstructed feature of the i'th incremental class, N represents the number of reconstructed samples of the i'th incremental class, and the distance between the feature y of the test sample and c i′ is used to determine the class l to which it belongs. where y k and c i′,k are the eigenvalues of y and c i′ in the kth dimension, respectively.

6. The method of claim 1, wherein, In step S31, a decomposable low-rank matrix is used to approximate the parameter change amount during model parameter updating, and Xavier initialization and zero initialization are used for parameter initialization process, ΔW is the parameter to be updated, B and A are two matrices after low-rank decomposition ΔW = BA = 0.

7. The method of claim 1, wherein, In step S32, the model structure initialized in S31 is used as the branch structure of the backbone network for parameter fine-tuning, wherein W is the original parameter unchanged, W' is the updated parameter, W' = W + ΔW.

8. An information processing apparatus comprising a storage, a processor, and a program stored on the storage and executable on the processor, characterized by The processor implements the method according to any one of claims 1-7 when executing the program.

Images