Small sample image classification migration method based on low-frequency enhancement

Abstract

The invention discloses a small sample image classification migration method based on low-frequency enhancement, and the method comprises the steps: enabling an existing image classification model to be divided into an image feature encoder and a classifier, setting a convolution kernel adapter for each convolution kernel in the image feature encoder, taking the parameters of the convolution kernel adapters and the classifier as learnable parameters, and carrying out the recognition of the learnable parameters. Other parameters are fixed as parameter values obtained by pre-training of the image classification model, then multiple rounds of training are carried out, and in each round, sampling is carried out from a small sample set of each image classification task to obtain a support set and a query set for training; and after the training is finished, selecting the model parameter with the maximum average prediction precision as the final image classification model parameter according to the average prediction precision of each round. According to the method, the learning speed of small sample image classification migration can be effectively improved, the video memory consumption is reduced, and the learning robustness is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of small sample image classification, and more specifically, relates to a small sample image classification migration method based on low-frequency enhancement. Background Art

[0002] At present, deep neural networks have shown excellent performance in the field of small sample image classification. However, the key to the effectiveness of deep learning is often sufficient data volume, so that the neural network can estimate the data distribution parameters as unbiased as possible. From the perspective of parameter estimation, small sample learning is a trade-off between sample size and parameter estimation efficiency. However, when higher animals such as humans learn new knowledge, they can often quickly master the knowledge based on a few sample examples, while neural networks cannot do this well. In order to solve this problem, small sample learning (Few-shot Learning) was proposed. Few-shot learning is based on a very small number of sample examples to quickly learn the generalization of new knowledge, making machine learning gradually approach general intelligence. In recent years, many effective small sample learning methods have emerged, which can be conceptually divided into the following types: small sample learning methods based on meta-learning, small sample learning methods based on transfer learning, and contextual cue learning based on large models.

[0003] Although existing methods have achieved remarkable success, the small sample problem is still far from being solved. In the field of small sample learning based on transfer learning, existing small sample learning methods usually require fine-tuning of a large number of parameters in the model, resulting in large-scale video memory and time consumption; in addition, existing small sample learning methods are limited by the sample size, prone to overfitting to noise, and perform poorly on new data sets; and existing small sample learning methods lack understanding support and interpretability. Due to the above problems, further improvements are needed to small sample image classification transfer methods. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a small sample image classification migration method based on low-frequency enhancement. Starting from the characteristics of the small sample image classification problem itself, taking into account the optimization problems brought about by low signal-to-noise ratio data, based on the simplicity and frequency principles of neural network learning, and based on the theory of Fourier transform, the low-frequency part of the convolution kernel parameters is selectively fine-tuned, thereby improving the learning speed, reducing video memory consumption and improving learning robustness.

[0005] In order to achieve the above-mentioned object of the invention, the small sample image classification migration method based on low-frequency enhancement of the present invention comprises the following steps:

[0006] S1: Obtain the support set A of the image classification task that needs to be migrated according to the actual situation. S and query set A Q, where the support set A S contains samples of D categories, each category contains N image samples and their labels, and the query set A Q Contains several samples of each category and their labels;

[0007] S2: Determine the existing image classification model based on actual conditions, and then divide it into an image feature encoder and a classifier; set a convolution kernel adapter for each convolution kernel in the image feature encoder Where i = 1, 2, ..., K, K represents the number of convolution kernels in the image feature encoder, L i Represents the length of the i-th convolution kernel channel;

[0008] S3: Initialize each convolution kernel adapter C i All elements in are 0, and the convolution kernel parameters W are set. i ′=W i +C i , W i Represents the convolution kernel parameters obtained by pre-training the image classification model; the convolution kernel adapter C i The parameters of the classifier and the classifier are used as learnable parameters, and the other parameters are fixed to the parameters obtained by pre-training of the image classification model;

[0009] S4: Set training round t = 1, initialize average prediction accuracy ACC0 = 0;

[0010] S5: Using the support set A of the image classification task S Fine-tune the image classification model;

[0011] S6: The query set A of the image classification task Q Each sample in the fine-tuned image classification model is input and the prediction accuracy is calculated, and then the average prediction accuracy ACC of each category in this round is calculated. t ;

[0012] S7: Determine whether the average prediction accuracy is ACC t -ACC t-1 >ε, where ε represents a preset threshold. If yes, go to step S8, otherwise go to step S9;

[0013] S8: Set t=t+1, and return to step S5;

[0014] S9: Use the current model parameters as the final image classification model parameters.

[0015] The present invention is based on a small sample image classification migration method for low-frequency enhancement. The existing image classification model is divided into an image feature encoder and a classifier. A convolution kernel adapter is set for each convolution kernel in the image feature encoder. The parameters of the convolution kernel adapter and the classifier are used as learnable parameters, and other parameters are fixed to parameter values ​​obtained by pre-training the image classification model. Then, multiple rounds of training are performed. In each round, a support set and a query set are sampled from a small sample set of each image classification task for training. After the training is completed, the model parameters with the largest average prediction accuracy are selected as the final image classification model parameters according to the average prediction accuracy of each round.

[0016] The present invention has the following beneficial effects:

[0017] 1) Based on the performance of the simplicity principle of deep learning in frequency, the present invention uses Fourier transform theory to perform frequency domain transformation on the convolution kernel parameters, and learns a set of specific adapter parameters for each small sample classification task sampled from the data set. Considering the structural characteristics of the encoder, a single-value increment is learned for each channel of each convolution kernel in the encoder and applied to each element of the channel; the fine-tuning time and video memory overhead are significantly reduced, the learning speed is accelerated, and the hardware requirements for learning are reduced;

[0018] 2) The present invention can mathematically ensure that the model only learns the low-frequency information that has the greatest effect on migration enhancement, thereby suppressing the high-frequency noise in the low signal-to-noise ratio data that has no effect on migration, so that learning can be done from the low-frequency part of the data that contains the most effective information, ensuring the robustness and generalization of learning;

[0019] 3) According to the Fourier transform theory, the convolution kernel parameter value is transformed into the superposition of amplitudes of different frequencies, and the single-value increment in the frequency domain is equivalent to the amplitude increment of the low-frequency component in the frequency domain. Therefore, mathematically, the present invention is equivalent to fine-tuning the low-frequency component of the convolution kernel parameter value, thereby improving the interpretability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of a specific implementation of the small sample image classification migration method based on low-frequency enhancement of the present invention;

[0021] Figure 2 It is a structural and learning diagram of the convolution kernel adapter in the present invention;

[0022] Figure 3 4 is a diagram comparing the time and video memory overhead between the present invention and the full fine-tuning method in this embodiment. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0024] In order to better illustrate the technical solution of the present invention, the technical principle on which the present invention is based is first briefly described.

[0025] In the present invention, the image classification model is divided into an image feature encoder and a classifier. A convolution kernel adapter is set for each convolution kernel in the image feature encoder. The low-frequency part in the model parameters is adjusted by learning the parameters of the convolution kernel adapter and adding them to the original convolution kernel parameters. This adjustment can be regarded as an operation on the low-frequency component in the frequency domain.

[0026] Fourier transform is a mathematical tool for frequency analysis of signals. In the present invention, the model parameters are regarded as signals, so that they can be converted into a combination of sine and cosine wave signals of different frequencies through Fourier transform.

[0027] The two-dimensional Fourier transform converts a two-dimensional signal s(x,y) into a frequency domain signal S(u,v), and its mathematical expression is:

[0028]

[0029] Where u and v represent the frequencies in two dimensions.

[0030] By inverse Fourier transform, the frequency domain signal S(u,v) can be converted back to the spatial domain signal s(x,y):

[0031]

[0032] Assume that θ is the convolution kernel matrix of the image classification model. By learning a vector v and adding it to the convolution kernel matrix channel by channel, the adjusted parameter is θ′=θ+v. According to Fourier transform theory, since the addition operation adds a unique value to all elements in each channel of the convolution kernel, the increment is equivalent to the zero-frequency DC component in the frequency domain. Specifically, the above operation can be mathematically equivalent to the following steps:

[0033] 1) Fourier transform. Perform Fourier transform on the convolution kernel parameter θ channel by channel to obtain the frequency domain graph s(θ) = FFT(θ);

[0034] 2) Low-frequency migration. Add the zero-frequency component in the frequency domain graph s(θ) to the learnable vector v to obtain the migrated frequency domain graph s′(θ):

[0035]

[0036] 3) Inverse Fourier transform: The adjusted frequency domain image s′(θ) is transformed back to the spatial domain through inverse Fourier transform to obtain the adjusted convolution kernel parameter θ=iFFT(s′(θ)).

[0037] Based on the above theory, it can be proved that the method of setting the convolution kernel adapter of the present invention is equivalent to fine-tuning the low-frequency part of the convolution kernel parameters. In theory, it can effectively suppress high-frequency noise that has no gain to model learning, making the learning process more robust, thereby improving the performance of the migrated image classification model.

[0038] Figure 1 This is a flow chart of a specific implementation method of the small sample image classification migration method based on low-frequency enhancement of the present invention. Figure 1 As shown, the specific steps of the small sample image classification migration method based on low-frequency enhancement of the present invention include:

[0039] S101: Obtain training sample set:

[0040] Obtain the support set A of the image classification task that needs to be migrated according to the actual situation S and query set A Q , where the support set A S contains samples of D categories, each category contains N image samples and their labels, and the query set A Q Contains several examples for each category and their labels.

[0041] S102: Add convolution kernel adapter:

[0042] Determine the existing image classification model based on actual conditions, and then divide it into an image feature encoder and a classifier. Set a convolution kernel adapter for each convolution kernel in the image feature encoder. Where i = 1, 2, ..., K, K represents the number of convolution kernels in the image feature encoder, L i Represents the channel length of the i-th convolution kernel.

[0043] S103: Model parameter configuration:

[0044] Initialize each convolution kernel adapter C i All elements in are 0, and the convolution kernel parameters W are set. i ′=W i +C i , W i Represents the convolution kernel parameters obtained by pre-training the image classification model. iThe parameters of the image feature encoder and the classifier are used as learnable parameters, and the other parameters are fixed to the parameters obtained by pre-training the image classification model. In other words, the required_grad attribute of the convolution kernel adapter is set to True, and its is_leaf attribute is confirmed to be True; the required_grad of other parameters in the image feature encoder is set to False, and there is no need to check other attributes; the required_grad attribute of the classifier is set to True, and its is_leaf attribute is confirmed to be True.

[0045] Figure 2 Schematic diagram of the structure and learning of the convolution kernel adapter in the present invention. Figure 2 As shown, in the present invention, adding a convolution kernel adapter to the convolution kernel parameters in the image feature extractor is equivalent to adding a learnable low-frequency component to the zero-frequency component. The present invention has found through research that by learning the convolution kernel adapter, fine-tuning the low-frequency component can achieve the effect of fine-tuning all parameters. Compared with complete fine-tuning, the number of adjusted parameters is greatly reduced, and the migration cost is effectively reduced.

[0046] S104: Set training round t=1, and initialize average prediction accuracy ACC0=0.

[0047] S105: Support set fine-tuning learning:

[0048] Use the support set A of the image classification task S Fine-tune the image classification model. In this embodiment, Adam is selected as the optimizer when fine-tuning the learning, the cosine annealing learning rate scheduler is used, and the loss function uses the cross entropy loss function. Since other parameters in the image classification model are fixed, only the parameters of the convolution kernel adapter and the classifier are adjusted at this time.

[0049] S106: Query set prediction:

[0050] The query set A of the image classification task Q Each sample in the fine-tuned image classification model is input and the prediction accuracy is calculated, and then the average prediction accuracy ACC of each category in this round is calculated. t .

[0051] S107: Determine whether the average prediction accuracy is ACC t -ACC t-1 >ε, ε represents a preset threshold, if yes, go to step S108, otherwise go to step S109;

[0052] S108: Let t=t+1, and return to step S105.

[0053] S109: Determine image classification model parameters:

[0054] Use the current model parameters as the final image classification model parameters.

[0055] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention.

[0056] Example 1

[0057] In this embodiment, the experimental conditions are set as follows: system: Ubuntu 20.04, software: Python 3.6, processor: Intel(R) Xeon(R) CPU E5-2678 v3@2.50GHz×2, memory: 256GB.

[0058] In this embodiment, different data sets are used for experimental verification, including the Omniglot data set, the Aircraft data set, the CUB data set, the DTD data set, the QuickD data set, the VGG data set and the MSCOCO data set.

[0059] Based on the small sample image classification task, the existing parameter efficient small sample fine-tuning method is used as a comparison method to compare with the method of the present invention, and the comparison method includes a zero fine-tuning method (w / o FT), a task specific adapter method (TSA) and a full fine-tuning method (Full FT). Table 1 is a comparison table of the classification accuracy of the present invention and the comparison method under different data sets in this embodiment.

[0060]

[0061] Table 1

[0062] As shown in Table 1, the classification accuracy of the proposed method is slightly better than the comparative task-specific adapter method TSA, and is very close to the full fine-tuning method, while also significantly better than the zero fine-tuning method.

[0063] This embodiment also experiments on the time and video memory overhead of the present invention. Figure 3 is a comparison chart of the time and memory overhead between the present invention and the full fine-tuning method in this embodiment. Figure 3 As shown in the figure, the present invention is slightly faster than the full fine-tuning method in terms of time. This is because the number of parameters that need to be updated in the model is greatly reduced, which significantly reduces the time cost of parameter update. The most significant optimization of the present invention is the reduction of video memory cost. Compared with full fine-tuning, the present invention reduces the video memory requirement by about 1 / 3, which provides conditions for engineering applications of small sample fine-tuning.

[0064] Next, this embodiment also studies the performance change with the increase of the number of shots (the number of samples in each category). Table 2 is a comparison table of the classification accuracy of the present invention and the full fine-tuning method under different numbers of shots in this embodiment.

[0065]

[0066] Table 2

[0067] As shown in Table 2, when small sample fine-tuning is getting closer and closer to ordinary fine-tuning, the present invention can still maintain a fairly good performance. Even under the 32-shot condition, the low-frequency enhancement method adopted by the present invention can still learn most of the knowledge.

[0068] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A small sample image classification migration method based on low-frequency enhancement, characterized in that: The following steps are involved: S1: Obtain the support set A of the image classification task that needs to be migrated according to the actual situation. S and query set A Q , where the support set A S contains samples of D categories, each category contains N image samples and their labels, and the query set A Q Contains several samples of each category and their labels; S2: Determine the existing image classification model based on actual conditions, and then divide it into an image feature encoder and a classifier; set a convolution kernel adapter for each convolution kernel in the image feature encoder Where i = 1, 2, ..., K, K represents the number of convolution kernels in the image feature encoder, L i Represents the length of the i-th convolution kernel channel; S3: Initialize each convolution kernel adapter C i All elements in are 0, and the convolution kernel parameters W are set. i ′=W i +C i , W i Represents the convolution kernel parameters obtained by pre-training the image classification model; the convolution kernel adapter C i The parameters of the classifier and the classifier are used as learnable parameters, and the other parameters are fixed to the parameters obtained by pre-training of the image classification model; S4: Set training round t = 1, and initialize average prediction accuracy ACC0 = 0; S5: Using the support set A of the image classification task S Fine-tune the image classification model; S6: The query set A of the image classification task Q Each sample in the fine-tuned image classification model is input and the prediction accuracy is calculated, and then the average prediction accuracy ACC of each category in this round is calculated. t ; S7: Determine whether the average prediction accuracy is ACC t -ACC t-1 >ε, where ε represents a preset threshold value. If yes, go to step S8, otherwise go to step S9; S8: Set t=t+1, and return to step S5; S9: Use the current model parameters as the final image classification model parameters.

2. The small sample image classification migration method according to claim 1, characterized in that: In the step S5, Adam is selected as the optimizer in the fine-tuning learning, a cosine annealing learning rate scheduler is used, and a cross entropy loss function is used as the loss function.

Images