An Image Classification Method and Device Based on a Feature Diffusion Model

Abstract

The present invention discloses an image classification method and device based on a feature diffusion model, relating to the technical field of image classification. By introducing the technologies of feature migration and diffusion model, the deep convolutional features of samples are used as conditions, and denoising is performed on the deep features on this basis, so that the denoised features have a high similarity with the features of samples of the same category. A better feature extractor is learned under the base class data by means of feature diffusion, high-recognizability visual features are extracted, and more discriminative features are generated. At the same time, compared with the original diffusion model algorithm, the required amount of data is small, it is easy to implement, and a large amount of time and computational resource costs are saved. In the test stage, the cosine distance is used to measure the cosine similarity between the query sample features and the denoised features, and then the query samples of the new class are classified, thereby improving the performance of few-shot image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to an image classification method and device based on a feature diffusion model. Background Art

[0002] In recent years, with the development of deep learning, the recognition performance of machines has surpassed that of humans in many large-sample image classification tasks. However, when the sample size is relatively small, there is still a large gap between the recognition level of machines and humans. Therefore, image classification with a small number of training samples, especially small-sample image classification with only one or a few labeled samples per class (Few-shot Image Classification), has received widespread attention from researchers in the past two years. Few-shot classification belongs to the category of few-shot learning (Few-shot Learning), and often contains two types of data that do not intersect in the category space, namely base class data and new class data. Few-shot classification aims to use the knowledge learned from base class data and a small number of labeled samples (support samples) of new class data to learn classification rules and accurately predict the categories of unlabeled samples (query samples) in new class tasks. Its framework is as follows: Figure 1 shown.

[0003] Small sample image classification is a research problem that needs to be solved urgently in the current field of computer vision and artificial intelligence. Existing and relatively successful large sample image classification methods rely heavily on the number of samples, while the sample size of things in the real world follows a long-tail distribution, that is, the sample size of a large number of things is seriously insufficient. For example, in the military, medical, industrial, astronomical and other fields, sample collection requires a lot of manpower, material resources, time and economic costs, and it is difficult to collect large-scale image samples. Therefore, conducting research on small sample image classification is of great value to the widespread application of image classification technology.

[0004] For large sample image classification, existing deep learning technology can learn highly recognizable image features by increasing model elasticity and sample size. However, for small sample classification tasks with very few labeled samples, existing deep learning technology is not applicable, which greatly limits the practical application of small sample image classification technology. Therefore, how to learn highly recognizable feature representations based on base class data and new class data with very few labeled samples is a problem worth exploring.

[0005] The current small sample image classification methods based on metric learning usually adopt scene-based training, which cannot sample all possible classification scenes and has insufficient fitting ability for base class data. At the same time, since the samples under the new class are never seen by the model, the model trained on the base class data has insufficient representation ability of the extracted visual features of the new class. Summary of the invention

[0006] To solve the problem that the current method has insufficient fitting ability under the base-class data and cannot well identify new-class samples, the present invention proposes an image classification method and device based on a feature diffusion model. During the model training stage, a diffusion model is used to train a better feature extractor to extract highly distinguishable visual features, and based on a small number of labeled samples under the new class, query samples are identified, so as to obtain better classification performance under the new-class data.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides an image classification method based on a feature diffusion model, including the following steps:

[0009] S1. Data preparation. The image data includes a training set and a test set. Among them, the training set is used as the base-class data to train the model, and the test set is used as the new-class data to evaluate the small-sample image classification performance of the model. The new-class data is organized into a C-way K-shot classification task;

[0010] S2. Use the base-class data to train the feature diffusion model and obtain the optimal image classification model; the image classification model consists of an embedding module f θ , a feature diffusion module, and a distance metric module. Among them, the embedding module f θ is used to extract features from the input image data, and θ is a learnable parameter; the feature diffusion module takes the feature of an input sample as the input, and through multiple iterations of denoising, generates the target feature of the same class; the feature diffusion module includes a denoising network as a learnable parameter; the distance metric module is used to calculate the cosine distance between the generated sample feature and the query sample feature, and classify the query sample according to the proximity of the cosine distance;

[0011] S3. Use the optimal image classification model to classify the test images.

[0012] On the other hand, the present invention also provides an image classification device based on a feature diffusion model, including the following modules to implement the image classification method based on a feature diffusion model described in any one of the above:

[0013] The embedding module f θ , which is used to extract features from the input image data, and θ is a learnable parameter;

[0014] The feature diffusion module takes the feature of an input sample as the input, and through multiple iterations of denoising, generates the target feature of the same class; the feature diffusion module includes a denoising network as a learnable parameter;

[0015] A distance metric module, which is used to calculate the cosine distance between the generated sample features and the query sample features, and classify the query sample according to the proximity of the cosine distance.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] The image classification method and device based on the feature diffusion model proposed by the present invention introduce the technologies of feature migration and diffusion model. Taking the deep convolutional features of the sample as the condition, denoising is performed on the basis of the deep features, so that the denoised features have a high similarity with the features of the same-class samples. A better feature extractor is learned under the base class data by using the feature diffusion method, high-recognizability visual features are extracted, and more discriminative features are generated. At the same time, compared with the original diffusion model algorithm, the required amount of data is small, it is easy to implement, and a large amount of time and computing resource costs are saved. At the test stage, the cosine distance is used to measure the cosine similarity between the query sample features and the denoised features, and then the query samples of the new class are classified, thereby improving the performance of few-shot image classification. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

[0019] Figure 1 It is a framework diagram for few-shot classification.

[0020] Figure 2 It is a flowchart of the image classification method based on the feature diffusion model provided by the embodiment of the present invention.

[0021] Figure 3 It is a flowchart of the training stage of the image classification method based on the feature diffusion model provided by the embodiment of the present invention.

[0022] Figure 4 It is a flowchart of the test stage of the image classification method based on the feature diffusion model provided by the embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of the functional modules of the image classification device based on the feature diffusion model provided by the embodiment of the present invention. Detailed Embodiments

[0024] To better understand the present technical solution, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described examples are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on this application belong to the scope of protection of the present invention.

[0025] The image classification method based on the feature diffusion model proposed by the present invention, as Figure 2 shown, includes the following steps:

[0026] S1. Data preparation. The image data includes a training set and a test set. Among them, the training set is used as the base class data to train the model, and the test set is used as the new class data to evaluate the small-sample image classification performance of the model. The new class data is organized into a C-way K-shot classification task. The base class data is not organized into a C-way K-shot classification task.

[0027] Specifically, step S1 divides the data into a training set and a test set in two parts. Among them, L train and L test are the categories of the training set and the test set respectively, and the category spaces of these two parts are mutually exclusive, that is N train is the total number of samples in the training set, N test is the total number of samples in the test set, is the training set image sample, y n is the category corresponding to the training set sample, n is the index of the training set sample, is the test set image sample, is the category corresponding to the test set sample, is the index of the test set sample, H is the height of the original image sample, and W is the width of the image sample.

[0028] For the test set D test , randomly select C categories from it, and randomly select M + K samples from each category. Among them, K samples are used as support samples S e , and M samples are used as query samples Q e . The support samples S e and the query samples Q e constitute a task T e , where e = 1, 2,..., E is the index of the task, and E is the total number of tasks for upsampling the base class data; for the task T e ={S e , Q e}, where Denote the k-th support sample of the c-th class, Denote the j-th query sample, c ∈ [1, 2, …, C], k ∈ [1, 2, …, K], j ∈ [1, 2, …, C×M]; H represents the length of the image sample, and W represents the width of the image sample; Denote the one-hot label of the k-th support sample of the c-th class. The one-hot label is a 1×C vector. If the sample belongs to the c-th class, the value at the c-th position is 1, and the values at the other positions are 0.

[0029] S2. Use the base-class data to train the feature diffusion model and obtain the optimal image classification model. The image classification model consists of an embedding module f θ 、a feature diffusion module, and a distance metric module. Among them, the embedding module f θ is used to extract features from the input image data, and θ is a learnable parameter; the feature diffusion module takes the features of an input sample as input, and through multiple iterations of denoising, generates the target features of the same class; the feature diffusion module contains a denoising network as a learnable parameter; generally speaking, the generated features are significantly distinguishable in terms of class, so the generated features can be used to measure the features of the query samples subsequently, and then classify the query samples; the distance metric module is used to calculate the cosine distance between the generated sample features and the query sample features, and classify the query sample according to the proximity of the cosine distance.

[0030] Specifically, the training stage of the few-shot image classification model based on the feature diffusion model is as Figure 3 shown. Step S2 includes the following sub-steps:

[0031] Step S21, extract the features of the positive sample pair

[0032] For a positive sample pair obtained by downsampling the base-class data Send the input sample and the target sample into the embedding module f with shared parameters θ to extract features, and obtain the input sample features and the query sample features where d, h, w are the dimension, length, and width of the feature map respectively; the embedding module f θ adopts a common convolutional neural network, which specifically includes four convolutional blocks. Each convolutional block includes the following parts: a convolutional layer with a 3×3 convolutional kernel with an output dimension of 64, a stride of 1, and an input padding of 1; a 2×2 max-pooling layer; a batch normalization layer (BatchNorm) and a non-linear activation function ReLU;

[0033] Step S22, optimize the feature diffusion model

[0034] Given a series of hyperparameters Let T represent the total number of iterations, and β t is a linear interpolation of T from 0.0001 to 0.02. Let α t = 1 - β t , denotes the cumulative product of the first t times, that is For the i-th target sample feature obtained in step S21 perform the update of the Gaussian noise network and the denoising of the input feature to obtain the finally denoised output feature

[0035] Among them, the update of the Gaussian noise network and the denoising process of the input feature include the following sub-steps:

[0036] Step S221, first sample t from a uniform distribution, that is, t ~ Uniform({1,..., T}), then sample a noise ε ~ N(0, I) from the standard Gaussian distribution, and update the Gaussian noise network through the following formula

[0037]

[0038] where η1 is the learning rate, initially set to 0.01, is the gradient of, is the optimized result, and the Gaussian noise network is a multi-layer perceptron (MLP);

[0039] Step S222, repeat step S221 until the Gaussian noise network converges to obtain the converged Gaussian noise model as the network parameters after convergence;

[0040] Step S223, then relabel the input sample feature as representing the input sample feature at the T-th iteration; set the hyperparameter σ t , t gradually decreases from T. For each t-th time, if t > 1, sample the Gaussian noise π ~ N(0, I), otherwise z = 0; the denoising process uses the following formula:

[0041]

[0042] to obtain the finally denoised output feature

[0043] Step S23: Calculate the loss and optimize the embedding network

[0044] Use the mean squared error loss function MSELoss to calculate the features of the i-th sample after denoising and the target features The error between them, and the specific calculation formula is: Freeze the parameters of the Gaussian noise module and use the SGD optimization algorithm to only optimize the parameters θ of the embedding module f θ The specific calculation formula of the SGD optimization algorithm is: where η2 is the learning rate, initially set to 0.01, is the gradient of θ, and θ * is the optimized result; then the positive sample pair index i = i + 1, and resample the positive sample pair X from the base class data i and perform steps S22 - S23 in the same way until i = N pairs where N pairs is the total number of sampled sample pairs;

[0045] Step S24: Output the optimal model

[0046] In the iteration of step S23, use the embedding module with the minimum loss as the optimal embedding module and output it, where θ * is the optimal model parameter.

[0047] Step S3: Classify the test images using the optimal image classification model

[0048] Specifically, the test stage of the few-shot image classification model based on the feature diffusion model is as Figure 4 shown. Step S3 includes the following sub-steps:

[0049] Step S31: New class task construction

[0050] For a task T e ={S e ,Q e} of the new class data, send the support samples and the query samples into the optimal embedding module to extract features, and obtain the support sample features and the query sample features where d, h, w are the dimension, length, and width of the feature map respectively;

[0051] Step S32: Classify the query samples in the current new class task

[0052] Take each support sample feature in Step S31 as the input sample in Step S21 After Steps S21 and S22, obtain the feature of each denoised sample, marked as

[0053] Next, calculate the mean of the denoised sample features of the same class. The specific calculation is as follows:

[0054]

[0055] Calculate the query sample feature and the cosine distance between them. The calculation formula is:

[0056]

[0057] Obtain the distance of the j-th query sample to the c-th class Take the class corresponding to the maximum value of the distances between the j-th query sample and all C classes as the predicted class of this query sample, that is Classify all query samples for all j ∈ [1, 2, …, C×M] All are classified;

[0058] Step S33: Classify the query samples in the next task

[0059] Then the task number e = e + 1, and repeat Steps S31 and S32 until e = E.

[0060] On the other hand, the present invention also provides an image classification device based on a feature diffusion model, as Figure 5 shown, including the following modules to implement the image classification method based on a feature diffusion model described in any one of the above:

[0061] Embedding module f θ , which is used to extract features from the input image data, and θ is a learnable parameter;

[0062] Feature diffusion module, which takes the feature of an input sample as the input and generates the target feature of the same class after multiple iterations of denoising; the feature diffusion module includes a denoising network is a learnable parameter;

[0063] Distance metric module, which is used to calculate the cosine distance between the generated sample feature and the query sample feature, and classify the query sample according to the proximity of the cosine distance.

[0064] Advantages of the present invention:

[0065] 1. Existing diffusion models all operate on the original image. Due to the large amount of data in the original image, it is difficult to implement the diffusion model algorithm, which requires a large amount of calculation and iterative solution. The present invention creatively uses the algorithm of the diffusion model for denoising and restoring image features. By adopting the diffusion model on the features of the base class data, denoising and restoring the features of the same type of image samples, a better feature extractor is learned in this way. Compared with the original diffusion model algorithm, it requires less data, is easy to implement, and saves a large amount of time and computational resource costs.

[0066] 2. In current methods for small-sample image classification, high-identifiability feature extraction is a key issue. The present invention introduces the idea of the diffusion model into the small-sample image classification task, and uses the method of feature diffusion to generate more discriminative features, thereby improving the performance of small-sample image classification.

[0067] 3. Existing diffusion models use denoising to restore the original image from random noise. The present invention creatively uses the features of the image samples as conditions to restore the features of the same category of samples, enabling the network to learn more identifiable features, and applying this method to the restoration of query sample features from support sample features, thereby realizing the image classification problem under the condition of a small number of labeled samples.

[0068] The above is only a detailed description of the preferred embodiments and principles of the present invention, and is not intended to limit the protection scope of the present invention. For those of ordinary skill in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be regarded as within the protection scope of the present invention.

Claims

1. An image classification method based on a feature diffusion model, characterized in that, It includes the following steps: S1. Data preparation: The image data includes a training set and a test set. Among them, the training set is used as the base class data to train the model, and the test set is used as the new class data to evaluate the few-shot image classification performance of the model. The new class data is organized into a C-way K-shot classification task; S2. Train a feature diffusion model using the base class data and obtain the optimal image classification model; the image classification model consists of an embedding module f θ , a feature diffusion module, and a distance metric module. Among them, the embedding module f θ is used to extract features from the input image data, and θ is a learnable parameter; the feature diffusion module takes the features of an input sample as input and generates the target features of the same class after multiple iterations of denoising; the feature diffusion module contains a denoising network is a learnable parameter; the distance metric module is used to calculate the cosine distance between the generated sample features and the query sample features, and classify the query sample according to the proximity of the cosine distance; S3. Use the optimal image classification model to classify the test images; Step S2 includes the following sub-steps: Step S21. Extract the positive sample pair features A positive sample pair for downsampling base-class data Input the input sample and the target sample into the embedding module f with shared parameters respectively θ to extract features, obtaining the input sample features and the query sample features where d, h, and w are the dimension, length, and width of the feature map respectively; the embedding module f θ employs a common convolutional neural network, specifically consisting of four convolutional blocks. Each convolutional block includes the following parts: a convolutional layer with a 3×3 convolutional kernel of output dimension 64, a stride of 1, and an input padding of 1; a 2×2 max pooling layer; a batch normalization layer; and a non-linear activation function ReLU; Step S22. Optimize the feature diffusion model Given a series of hyperparameters Let T represent the total number of iterations, and β t is a linear interpolation of T from 0.0001 to 0.

02. Let α t = 1 - β t , represents the cumulative product of the first t times, that is For the i-th target sample feature obtained in step S21 perform the update of the Gaussian noise network and the denoising of the input feature to obtain the finally denoised output feature Step S23. Calculate the loss and optimize the embedding network Calculate the features after denoising of the $i$-th sample using the mean squared error loss function MSELoss and the target features The error between them, and the specific calculation formula is: Freeze the parameters of the Gaussian noise module and only optimize the parameters $\theta$ of the embedding module $f$ using the SGD optimization algorithm θ The specific calculation formula of the SGD optimization algorithm is: where $\eta$ is the learning rate, initially set to 0.01, is the gradient of $\theta$, and $\theta$ * is the optimized result; then the positive sample pair index $i = i + 1$, and resample the positive sample pair $X$ in the base class data i and perform the same steps S22 - S23 until $i = N$ pairs where $N$ pairs is the total number of sampled sample pairs; Step S24. Output the optimal model In the iteration of step 23, the embedding module with the minimum loss is taken as the optimal embedding module and output, where θ * is the optimal model parameter.

2. The image classification method based on a feature diffusion model according to claim 1, wherein Step S1 divides the data into a training set and a test set in two parts. Among them, L train and L test are the categories of the training set and the test set respectively, and the category spaces of these two parts are mutually exclusive, that is N train is the total number of samples in the training set, N test is the total number of samples in the test set, is the image sample of the training set, y n is the category corresponding to the training set sample, n is the index of the training set sample, is the image sample of the test set, is the category corresponding to the test set sample, is the index of the test set sample, H is the height of the original image sample, and W is the width of the image sample.

3. The image classification method based on the feature diffusion model according to claim 2, wherein In step S1, for the test set D test , randomly select C categories from it, and randomly select M + K samples from each category, where K samples are used as support samples S e , and M samples are used as query samples Q e , support samples S e and query samples Q e constitute a task T e , where e = 1, 2, …, E is the index of the new class task, and E is the total number of tasks for upsampling the base class data; for task T e = {S e , Q e}, where represents the k-th support sample of the c-th class in the new class data, represents the j-th query sample in the new class data, c ∈ [1, 2, …, C], k ∈ [1, 2, …, K], j ∈ [1, 2, …, C × M]; H represents the height of the image sample, and W represents the width of the image sample; represents the one-hot label of the k-th support sample of the c-th class in the new class data. The one-hot label is a 1 × C vector. If the sample belongs to the c-th class, the value at the c-th position is 1, and the rest of the positions are 0.

4. The image classification method based on a feature diffusion model according to claim 1, wherein Step S22 Gaussian noise network update and the input features denoising process includes the following sub-steps: Step S221, first, sample t from a uniform distribution, i.e., t ∼ Uniform({1, …, T}), and then, sample a noise ε ∼ N(0, I) from a standard Gaussian distribution, and update the Gaussian noise network through the following formula where η1 is the learning rate, initially set to 0.01, is the gradient of the optimized result, and the Gaussian noise network is a multi-layer perceptron; Step S222. Repeat Step S221 until the Gaussian noise network converges to obtain a converged Gaussian noise model which is the network parameter after convergence; Step S223, then for the input sample features relabel them as the input sample features representing the T-th iteration; set the hyperparameter σ t , t gradually decreases from T. For each t-th time, if t > 1, sample Gaussian noise π ∼ N(0, I), otherwise π = 0; the denoising process uses the following formula: Obtain the final denoised output features 5. The image classification method based on the feature diffusion model according to claim 1, characterized in that, Step S3 includes the following sub-steps: Step S31. New class task construction For a task T of new class data e ={S e ,Q e}, the support samples and query samples are respectively fed into the optimal embedding module to extract features, obtaining the support sample features and query sample features where d, h, w are respectively the dimension, length and width of the feature map; Step S32. Classify the query samples in the current new class task Each support sample feature in step S31 is used as the input sample in step S21 After steps S21 and S22, each denoised sample feature is obtained and labeled as Next, calculate the mean of the denoised sample features of the same class. The specific calculation is as follows: Calculate the features of the query sample and The cosine distance between them is calculated by the following formula: Obtain the distance of the j-th query sample to the c-th class Take the class corresponding to the maximum value among the distances of the j-th query sample to all C classes as the predicted class of this query sample, that is For all query samples with all j ∈ [1, 2, …, C × M] Perform classification Step S33. Classify the query samples in the next task Next, the task number e = e + 1, and repeat steps S31 and S32 until e = E, where e = 1, 2, …, E is the index of the new class task, and E is the total number of upsampled tasks on the base class data.

6. An image classification device based on a feature diffusion model, characterized in that, It includes the following modules to implement the image classification method based on the feature diffusion model according to any one of claims 1-5: Embedding module f θ , which is used to extract features from the input image data, where θ is a learnable parameter; Feature diffusion module: Take the features of an input sample as input, and generate the target features of the same class after multiple iterations of denoising; Feature The diffusion module contains a denoising network are learnable parameters; Distance metric module: Used to calculate the cosine distance between the generated sample features and the query sample features, and classify the query sample according to the proximity of the cosine distance.

Images