Zero parameter updating image incremental learning method based on pre-training model

Abstract

The invention discloses a pre-training model-based zero-parameter updating image incremental learning method, which comprises the following steps of: dividing a data set into disjoint incremental task sets, and ensuring that categories are not overlapped; secondly, adopting a pre-training model as a feature extractor, and extracting multi-scale features of the image by using the strong feature representation capability of the pre-training model; thirdly, constructing a prototype library, and obtaining representative prototypes of all categories through feature clustering; in a test stage, a feature matching technology is utilized to classify a test sample to a category to which the most similar prototype belongs. The key point of the method is that parameter updating is not carried out on the pre-training model in the whole process, so that high efficiency and flexibility are kept, and meanwhile, through multi-prototype matching and a multi-scale feature fusion strategy, the fusion capability of the model on new and old knowledge is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a zero-parameter update image incremental learning method based on a pre-training model. Background Art

[0002] As people pay more and more attention to real-time information flow, incremental learning tasks have received more and more attention. Traditional deep learning methods assume that all data are independent and identically distributed, and that models trained on old data can also be used on new data. However, in the real world, the ideal situation of independent and identical distribution is often difficult to achieve. If a model is trained on new data with a different distribution from the old data, it may overfit the new data and forget the distribution of the old data, causing the model to perform poorly on the old data. This situation is called catastrophic forgetting.

[0003] Incremental learning methods focus on further absorbing the knowledge of new data after the model has learned the distribution of old data, and integrating old knowledge with new knowledge in multiple rounds of incremental training to solve the problem of catastrophic forgetting. However, in some application areas faced by incremental learning, the model is usually required to respond quickly to data changes, and existing incremental learning methods usually require additional incremental training to adapt to new data when faced with changing data. This is often unfavorable for the application of incremental learning. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, an object of the present invention is to provide a zero-parameter update image incremental learning method based on a pre-trained model.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A zero-parameter update image incremental learning method based on a pre-trained model, comprising:

[0007] Construct a data set: divide the data set into N groups of sub-data sets;

[0008] Build a feature extractor: Select a pre-trained model as a feature extractor to convert image features in the sub-dataset into high-dimensional vector features;

[0009] Extract image features: The input image is scaled according to multiple scaling factors, and the feature extractor is used to obtain high-dimensional vector features of multiple scales of the same input image. The high-dimensional vector features are fused and output as image representation;

[0010] Constructing a prototype library: extract all image representations from each set of sub-datasets, cluster the representations by category, obtain multiple cluster centers for each category as the prototype of the category, and further obtain the prototypes of all sub-datasets to form a prototype library;

[0011] Feature matching: extract the image representation of the test sample, and perform similarity matching with all prototypes in the prototype library. The category of the prototype with the highest similarity to the test sample is taken as the category of the test sample;

[0012] Repeat the steps of building the prototype library and feature matching to complete all incremental tasks.

[0013] Furthermore, the data set is divided into N groups of sub-data sets, specifically:

[0014] A dataset is divided into N groups of non-overlapping sub-datasets. Each group of sub-datasets is an incremental task. The incremental tasks are executed sequentially. Each group of incremental tasks includes M category labels. Each category label includes several images. The category labels of any two groups of incremental tasks are not repeated.

[0015] Furthermore, the pre-trained model is ViT-B / 16.

[0016] further,

[0017] If the scaled image is larger than the input of the feature extractor, it is cropped to the input size by center cropping;

[0018] If the scaled image is smaller than the input of the feature extractor, white is filled around the image to meet the input size.

[0019] Furthermore, the multiple scaling factors are 2-4.

[0020] Furthermore, the multiple scaling factors are specifically r﹦[0.8, 1.0, 1.2, 1.4].

[0021] Furthermore, the pre-trained model includes a backbone network.

[0022] Furthermore, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the zero-parameter update image incremental learning method when executing the computer program.

[0023] Furthermore, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the zero-parameter update image incremental learning method is implemented.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] The present invention does not require parameter updates for the pre-trained model, which improves the efficiency and flexibility of dealing with incremental learning tasks. In particular, in application scenarios where sample categories are rapidly updated, new data can be quickly adapted without additional training, effectively solving the problems of high resource consumption and long training time in traditional methods.

[0026] The present invention integrates multi-scale features and multi-prototypes for matching, which enhances the model's understanding of image diversity. It not only improves the generalization ability of the model, but also ensures the seamless integration of new and old knowledge by building a prototype library and implementing a feature matching mechanism, thereby alleviating catastrophic forgetting and enabling the model to retain the memory of old category information when learning new categories.

[0027] The model selection of the present invention is highly flexible and can adapt to a variety of pre-trained models. With the advancement of pre-training technology, the performance can be improved by replacing the pre-trained model without modifying the core algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0029] Figure 2 It is a schematic diagram of the feature extraction process of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0031] Example

[0032] like Figure 1 As shown, a zero-parameter update image incremental learning method based on a pre-trained model includes the following steps:

[0033] Step 1: Build a dataset: Divide a complete dataset into multiple sub-datasets. There is no overlap between sub-datasets and the categories are disjoint. Each sub-dataset is an incremental task. Each sub-dataset is divided into a training set and a test set. The training set includes images and the category labels corresponding to the images. The images in the test set are called test samples, which are used to test the accuracy of this method.

[0034] Specifically:

[0035] Divide a dataset D into N groups of disjoint incremental tasks T = [t1, t2, ..., t10], and execute incremental tasks T in the order of the N groups without repetition. Each task ti includes the corresponding category label Where M is the number of categories contained in ti, here M = 10.

[0036] In this embodiment, the data set uses the CIFAR100 data set. The CIFAR100 data set contains a total of 100 categories, each category contains 600 32x32 color images, of which 500 are used as training sets and 100 are used as test sets.

[0037] Step 2: Build a feature extractor: Select a pre-trained model as the feature extractor to convert the image features in the sub-dataset into high-dimensional vector features.

[0038] In this embodiment, the pre-trained model uses ViT-B / 16, which is used as a feature extractor after training on ImageNet-21k. This model is a representative of non-convolutional image feature extractors, and the model design is reasonable. In addition, the pre-trained model is pre-trained on the large-scale dataset ImageNet-21k and has sufficient image feature extraction capabilities.

[0039] The present invention is a zero parameter update method, and no parameter update is performed in subsequent steps. Therefore, a large-scale pre-training model with more parameters can be selected to make the extracted high-dimensional feature vector more discriminative.

[0040] The pre-training model in the present invention only includes the backbone network.

[0041] Other models such as VGG and ResNet can also be used.

[0042] Step 3: Extract image features: Figure 2 As shown, the input image is scaled according to multiple scaling factors, and a feature extractor is used to obtain high-dimensional vector features of multiple scales of the same input image, and the high-dimensional vector features are fused and output as image representation.

[0043] Specifically:

[0044] According to the scaling factor r = [r1, r2, ..., r k ] The input image I is scaled to obtain image I′. In this embodiment, r=[0.8,1.0,1.2,1.4]

[0045] h′=h*r,w′=w*r

[0046] If the scaled image I′ is larger than the input of the feature extractor F, it is cropped to the appropriate size through the center. If it is smaller, white is filled around to meet the input size. After this step, 4 images of the same photo with different scales that meet the input size I″=[I″1,I″2,I″3,I″4] are obtained.

[0047] Use feature extractor F to extract features X = [x1, x2, x3, x4] from the four images I″, and take the average of X to get the image representation of the input image I.

[0048]

[0049] Furthermore, the scaling factor is generally selected from 2 to 4 different values.

[0050] Step 4: Construct a prototype library: extract the representations of all images in each group of sub-datasets, cluster the representations according to categories, obtain multiple cluster centers for each category, and use the multiple cluster centers as prototypes of the category. Further, obtain the prototypes of all sub-datasets to form a prototype library.

[0051] Specifically:

[0052] Repeat step 3 to extract the final representation of all training samples in task ti Where L is the task t i The number of training samples: In this embodiment, the CIFAR100 dataset contains 500 training samples for each category, so L is 500.

[0053] right Classify by label and cluster the features with the same label. Use clustering algorithm to cluster the features of the same category and get multiple cluster centers μ1,μ2,...,μ T As the prototype of this category, T is the number of clusters in the clustering algorithm, and in this embodiment, T is 100.

[0054] After this stage, all prototypes are stored and a prototype library is built.

[0055] Furthermore, the clustering algorithm may adopt a K-MEANS clustering algorithm, a mean shift clustering algorithm, a DBSCAN clustering algorithm, and the like.

[0056] Step 5: Feature matching: extract the image representation of the test sample, and perform similarity matching with all prototypes in the prototype library. The category of the prototype with the highest similarity to the test sample is taken as the category of the test sample.

[0057] Repeat steps 4 and 5 to complete all incremental tasks.

[0058] The incremental process in this embodiment is specifically as follows: the data set is divided into 10 sub-data sets, that is, there are ten incremental tasks, each incremental task has 10 category labels, each sub-data set builds a prototype library according to this method, and then the image extraction representation of the test set is compared with the prototype in the prototype library to obtain the category and the accuracy of this method. When the data in the next batch of sub-data sets is input, the steps of the previous sub-data set are repeated until all batches of data are completed. The accuracy test of the last batch of data is the incremental accuracy result of this method for the data set. Because the last test includes all categories of the previous increments, the last test will include all 100 categories.

[0059] To verify the effectiveness of the present invention, 500 images from the CIFAR100 dataset are used for training and 100 images are used for testing. The buffer size refers to how many training images of previous tasks can be retained for playback in subsequent training processes to prevent the model from forgetting old knowledge. This is a common method for combating catastrophic forgetting in incremental learning, but this method is often difficult to implement in practical applications.

[0060] The evaluation index used in this embodiment is the average accuracy. The specific index results are shown in Table 1. From the results, it can be seen that the performance of the present invention is still very outstanding in the case of no buffer and no parameter update. And no buffer and no parameter update can be better applied to actual scenarios.

[0061] Table 1 Split CIFAR-100

[0062]

[0063] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the zero-parameter update image incremental learning method when executing the computer program.

[0064] This embodiment also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the zero-parameter update image incremental learning method is implemented.

[0065] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A zero-parameter update image incremental learning method based on a pre-trained model, characterized in that: include: Construct a data set: divide the data set into N groups of sub-data sets; Build a feature extractor: Select a pre-trained model as a feature extractor to convert image features in the sub-dataset into high-dimensional vector features; Extract image features: The input image is scaled according to multiple scaling factors, and the feature extractor is used to obtain high-dimensional vector features of multiple scales of the same input image. The high-dimensional vector features are fused and output as image representation; Constructing a prototype library: extract all image representations from each set of sub-datasets, cluster the representations by category, obtain multiple cluster centers for each category as the prototype of the category, and further obtain the prototypes of all sub-datasets to form a prototype library; Feature matching: extract the image representation of the test sample, and perform similarity matching with all prototypes in the prototype library. The category of the prototype with the highest similarity to the test sample is taken as the category of the test sample; Repeat the steps of building the prototype library and feature matching to complete all incremental tasks.

2. The zero-parameter update image incremental learning method according to claim 1, characterized in that: The data set is divided into N groups of sub-data sets, specifically: A dataset is divided into N groups of non-overlapping sub-datasets. Each group of sub-datasets is an incremental task. The incremental tasks are executed sequentially. Each group of incremental tasks includes M category labels. Each category label includes several images. The category labels of any two groups of incremental tasks are not repeated.

3. The zero-parameter update image incremental learning method according to claim 1, characterized in that: The pre-trained model is ViT-B / 16.

4. The zero-parameter update image incremental learning method according to claim 1, characterized in that: If the scaled image is larger than the input of the feature extractor, it is cropped to the input size by center cropping; If the scaled image is smaller than the input of the feature extractor, white is filled around the image to meet the input size.

5. The zero-parameter update image incremental learning method according to claim 1, characterized in that: The multiple scaling factors are 2-4.

6. The zero-parameter update image incremental learning method according to claim 5, characterized in that: The multiple scaling factors are specifically r = [0.8, 1.0, 1.2, 1.4].

7. The zero-parameter update image incremental learning method according to any one of claims 1 to 6, characterized in that: The pre-trained model includes a backbone network.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor runs the computer program, the zero-parameter update image incremental learning method as described in any one of claims 1-7 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the zero-parameter update image incremental learning method as described in any one of claims 1 to 7 is implemented.

Images