An image classification method and device based on category-related feature reconstruction

Abstract

The application discloses an image classification method and device based on category-related feature reconstruction, and relates to the technical field of image classification. The application adopts a feature reconstruction mode, first filters irrelevant features and background features through a support feature filtering module, retains category-related key features, and uses the key features to reconstruct features of the same category, so that the model has the ability to extract category-related key features. By constructing a category-related feature reconstruction network, high-recognition features are learned, so as to improve the small sample image classification capability, and solve the problems that the current small sample image classification method based on metric learning directly measures sample features, the visual features are not aligned, irrelevant features and background features interfere, and the like, so as to ensure the accuracy of the measurement result.

Description

An image classification method and apparatus based on category-related feature reconstruction Technical Field

[0001] This invention relates to the field of image classification technology, and in particular to an image classification method and apparatus based on category-related feature reconstruction. Background Technology

[0002] In recent years, with the development of deep learning, machine recognition performance has surpassed human capabilities in many large-sample image classification tasks. However, existing and relatively successful large-sample image classification methods heavily rely on the number of samples. In the real world, the sample size of objects follows a long-tailed distribution, meaning that the sample size for a large number of objects is severely insufficient. For example, in fields such as military, medical, industrial, and astronomical fields, sample collection requires significant human, material, time, and economic resources, making it difficult to collect large-scale image samples. Therefore, research on small-sample image classification is of great value for the widespread application of image classification technology.

[0003] Few-shot image classification is a pressing research problem in computer vision and artificial intelligence. When the sample size is small, machine recognition performance still lags significantly behind human capabilities. Image classification with a limited number of training samples, especially few-shot image classification where each class has only one or a few labeled samples, has received widespread attention from researchers in recent years. Few-shot classification falls under the category of few-shot learning and often involves two classes of data with disjoint class spaces: base class data and new class data. The aim of few-shot classification is to learn classification rules using knowledge learned from the base class data and a small number of labeled samples (support samples) from the new class data, accurately predicting the class of unlabeled samples (query samples) in the new class task. Its framework is shown in Figure 1.

[0004] For large-sample image classification, existing deep learning techniques can learn highly discriminative image features by increasing model flexibility and sample size. However, for small-sample classification tasks with very few labeled samples, existing deep learning techniques are not suitable, which greatly limits the practical application of small-sample image classification techniques. Therefore, how to learn highly discriminative feature representations based on base class data and new class data with very few labeled samples is a problem worth exploring.

[0005] Current few-sample image classification methods based on metric learning typically involve direct measurement of sample features. However, these methods suffer from problems such as visual feature misalignment, interference from irrelevant features and background features, resulting in inaccurate measurement results. Summary of the Invention

[0006] This invention addresses the problems of visual feature misalignment, irrelevant features, and background feature interference in current metric learning-based few-sample image classification methods. It proposes an image classification method and apparatus based on category-related feature reconstruction. By employing feature reconstruction, some irrelevant and background features are first filtered out, while key category-related features are retained. These key features are then used to reconstruct features of the same category, enabling the model to extract key category-related features and improving the performance of few-sample image classification under new class data, while ensuring the accuracy of the metric results.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] On the one hand, the present invention provides an image classification method based on category-related feature reconstruction, comprising the following steps:

[0009] S 1. Data preparation: The data includes training set and test set. The class spaces of the training set and test set are mutually exclusive. The training set is used as base class data to train the model, and the test set is used as new class data to evaluate the small sample image classification performance of the model.

[0010] S2. Base class data is used to train category-related features. These features are then used to reconstruct a few-sample image classification model, and the model parameters are optimized to obtain the optimal classification model. The few-sample image classification model reconstructed based on category-related features consists of four parts: an embedding module, a support feature filtering module, a feature reconstruction module, and a distance metric module. The embedding module extracts features from the input image data. The support feature filtering module removes irrelevant or background interference features based on the importance of local features of the support samples, retaining only category-related features. The feature reconstruction module reconstructs the query sample features using the filtered category-related features, obtaining the reconstructed query sample features. The distance metric module calculates the cosine distance between the reconstructed sample features and the query sample features, and classifies the query sample based on the proximity of the cosine distance.

[0011] S3. Classify the test images using the optimal model.

[0012] On the other hand, the present invention also provides an image classification apparatus based on category-related feature reconstruction, comprising the following modules to implement the image classification method based on category-related feature reconstruction described above:

[0013] The embedding module is used to extract features from the input image data, where θ is a learnable parameter;

[0014] It supports a feature filtering module that removes irrelevant or background interference features based on the importance of local features of the supporting samples, retaining only the features related to the category.

[0015] The feature reconstruction module reconstructs the features of the query samples using the filtered category-related features. These are learnable parameters;

[0016] The distance metric module is used to calculate the cosine distance between the reconstructed sample features and the query sample features, and to classify the query sample based on the magnitude of the cosine distance.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] This invention proposes an image classification method and apparatus based on category-related feature reconstruction, employing a metric-based approach in both the irrelevant local feature filtering and cosine similarity measurement stages. In the irrelevant local feature filtering stage, the importance of each local feature in the same category of support samples is measured, and unimportant local features are removed. When measuring importance, this invention calculates the cosine distance between each local feature and the global average feature of all local features. The importance of each local feature is measured based on the magnitude of the cosine distance. This method is simple and efficient, obtaining highly discriminative support sample features after filtering, and reducing the negative impact of irrelevant features or background interference features on image feature reconstruction. In the cosine distance measurement stage, the cosine similarity between the query sample features and the reconstructed features is measured, thereby classifying the query samples. In the feature reconstruction stage, a Transformer structure is used. The filtered key support sample features are taken as input, and the Transformer network outputs reconstructed features. By constraining the similarity between the query sample features and the reconstructed features in the same category through loss constraints, the network has the ability to generate similar features. This invention eliminates the need for interaction with query samples during reconstruction. It only constrains the reconstructed features to be as close as possible to features of the same class in the final loss, thus ensuring the effectiveness of the reconstruction. Compared to parameter-free feature reconstruction methods that typically require iterative solutions and complex reconstruction weight calculations, this invention uses a network to learn how to reconstruct, simplifying the complex reconstruction process and resulting in more flexible reconstructed features. Furthermore, the method of this invention is an end-to-end classification model that does not require any pre-training or multi-stage training. The training phase employs a meta-training approach matched to the new class task, effectively reducing the time and resource overhead required for training. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 is a framework diagram of few-shot classification.

[0021] Figure 2 is a flowchart of the image classification method based on category-related feature reconstruction provided in an embodiment of the present invention.

[0022] Figure 3 is a diagram of an image classification model based on category-related feature reconstruction provided in an embodiment of the present invention.

[0023] Figure 4 is a schematic diagram of the functional modules of the image classification device based on category-related feature reconstruction provided in an embodiment of the present invention. Detailed Implementation

[0024] To better understand this technical solution, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described examples are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0025] The image classification method based on category-related feature reconstruction proposed in this invention, as shown in Figures 2 and 3, includes the following steps:

[0026] S1. Data preparation: Image data includes training set and test set.

[0027] The image data is divided into training sets. and test set Two parts, of which L train and L test These are the categories of the training set and the test set, respectively, and the category spaces of these two parts are mutually exclusive. N train N is the total number of samples in the training set. test The total number of samples in the test set. These are image samples from the training set. It is the category corresponding to the training set sample. This is the index of the training set samples. These are image samples from the test set. It is the category corresponding to the test set sample. This is the index of the test set sample. H is the height of the original image, and W is the width of the original image.

[0028] In this invention, we will use the training set D train The test set D is used as base class data to train the model. test The base class data and the new class data are used to evaluate the model's small sample image classification performance. Furthermore, the base class data and the new class data are organized into a C-way K-shot classification task.

[0029] Specifically, for training set D train We train the set D from it. train C categories are randomly selected from the dataset. M+K samples are randomly selected from each category, of which K samples are used as support samples S. e M samples are used as query samples Q e S e and Q e Constitute a task T e , where e = 1, 2, ..., E is the index of the task, and E is the total number of tasks sampled from the base class data.

[0030] Similarly, for test set D test C categories are randomly selected from the list, and M+K samples are randomly selected from each category, of which K samples are used as support samples. M samples are used as query samples Supporting samples and query samples Constitute a task in It is an index of the task. The total number of tasks for upsampling the new data type.

[0031] For a task T of base class data e ={S e Q e},in This represents the k-th supporting sample of class c in the base class data. This represents the j-th query sample in the base class data. c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M], This represents the one-hot label of the k-th supporting sample in class c of the base class data. If the sample belongs to class c, then the value at position c is 1, and the values ​​at the other positions are 0; This represents the one-hot label of the j-th query sample in the base class data. If the sample belongs to class c, then the value at position c is 1, and the values ​​at the other positions are 0.

[0032] S2. Establish an image classification model: Use base class data to train category-related features to reconstruct a small sample image classification model, and optimize the model parameters to obtain the optimal classification model.

[0033] The few-sample image classification model based on category-related feature reconstruction consists of four parts: the embedding module f θIt supports feature filtering and feature reconstruction modules. And a distance metric module. Among them, the embedding module f... θ This module is used to extract features from the input image data, where θ is a learnable parameter; it supports the feature filtering module to remove irrelevant or background features based on the importance of local features of supporting samples, retaining only category-related features; the feature reconstruction module... Reconstruct the features of the query samples using the filtered category-related features. These are learnable parameters. Generally, support samples and query samples belonging to the same class have more common features, so class-related support features are more likely to reconstruct query sample features of the same class. The distance metric module is used to calculate the cosine distance between the reconstructed sample features and the query sample features, and classifies the query sample according to the proximity of the cosine distance.

[0034] Specifically, step S2 includes the following sub-steps:

[0035] Step S21: Extract sample features.

[0036] Support samples S from the training set e And query sample Q e They are respectively sent to the embedded module f θ Extract features to obtain supporting sample features and query sample features in d, h, and w are the dimension, length, and width of the feature, respectively. We call a d×1×1 vector a local feature, so each sample has a total of h×w local features.

[0037] Feature extraction module f θ It is a commonly used convolutional neural network, specifically containing four convolutional blocks. Each convolutional block contains the following parts: a convolutional layer with a 3×3 kernel and an output dimension of 64, a stride of 1, and input padding of 1; a 2×2 max pooling layer; a batch normalization layer (BatchNorm); and a non-linear activation function ReLU.

[0038] Step S22: Filter irrelevant local features from the local features of supporting samples.

[0039] For the support sample features obtained in step S21 We combine all the local features of each class c to form a feature pool for the class c, denoted as: Then, the average of hwk local features in each c-class feature pool is calculated to obtain the global average feature of the c-th class. Then calculate the feature pool p of class c. c Each local feature in Compared with global average features cosine distance i∈[1,2,…,hwk] represents the index of each local feature. The cosine distance is specifically represented as: And these distances form a set Where |||2 represents the L2 norm. For this set... The cosine distances are sorted from largest to smallest, and the first hw values ​​are selected and their corresponding indices i are recorded. The local features corresponding to these i indices are then... This forms a set, which is then concatenated to obtain the filtered local feature vector.

[0040] Step S23, Feature Reconstruction.

[0041] The filtered local feature vector obtained in step S22 Input Feature Reconstruction Module The query sample features are reconstructed to obtain the reconstructed query sample features. Feature Reconstruction Module Specifically, it's a Transformer network where the computation includes an Attention mechanism, a Layer Regularization (LN), and a Multilayer Perceptron (MLP). The Transformer is a neural network architecture based on a self-attention mechanism, initially proposed as a model for machine translation tasks. Unlike traditional recurrent neural networks or convolutional neural networks, the Transformer can not only process the sequential information of the input sequence but also establish direct connections between different positions, and it doesn't require step-by-step computation during training like recurrent neural networks. This makes the Transformer perform better when handling long text sequences and relationships within sequences.

[0042] In the Transformer, self-attention is a key component. It enables the model to apply attention weights to all positions in the input to capture global information and local relationships. Self-attention is achieved by calculating the similarity between queries, keys, and values, and then averaging the values. Furthermore, the Transformer uses techniques such as residual connections and layer normalization to improve model performance and training speed. This invention employs a Transformer structure in the feature reconstruction stage, using filtered key support sample features as input and utilizing the Transformer network to output reconstructed features. By constraining the similarity between similar query sample features and the reconstructed features through loss constraints, the network gains the ability to generate similar features.

[0043] The specific calculation of Attention in this invention is as follows:

[0044]

[0045] in, These represent the query, key, and value used in Attention computation, respectively. It is the transpose of K.

[0046] Therefore, the filtered local feature vector The feature q is obtained by calculating the input attention. c :

[0047]

[0048] in, and denoted as d×d, respectively, which are learnable weight parameters.

[0049] Next, we will use the q obtained in the previous step. c Input LN and MLP and then perform calculations to obtain the reconstructed query sample features:

[0050]

[0051] Step S24: Calculate the cosine distance.

[0052] The features of the query sample obtained in step S21 are calculated using the cosine distance formula. and the reconstructed query sample features obtained in step S23 The cosine similarity between them is calculated as follows: This represents the cosine similarity between the j-th query sample and the c-th class.

[0053] Step S25: Calculate the classification loss and optimize the model.

[0054] Use the cosine similarity obtained in step S24 and the true labels of the query samples. The cross-entropy loss (CELoss) is calculated as follows:

[0055]

[0056] Where CELOSs is the cross-entropy loss function.

[0057] The SGD optimization algorithm is used to optimize the embedded module f. θ and feature reconstruction module The parameters θ and The SGD optimization algorithm is specifically calculated as follows: Where η is the learning rate, initially set to 0.01. For θ and gradient, This is the optimized result.

[0058] Next, the number of tasks e = e + 1, and tasks T are resampled in the base class data. e Then, perform the same steps S21-S25 until e = E.

[0059] Step S26: Output the optimal class-related feature reconstruction model. In the iteration of step S25, the model with the smallest loss is selected as the optimal model.

[0060] S3. Classify the test images using the optimal model.

[0061] Step S3 includes the following sub-steps:

[0062] Step S31, construct the new task class.

[0063] A task for new types of data in This represents the k-th supporting sample in class c of the new data. This represents the j-th query sample in the new class of data. c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M]. This represents the one-hot label of the k-th supporting sample in class c of the new data. If the sample belongs to class c, then the value at position c is 1, and the values ​​at the other positions are 0.

[0064] Step S32: Classify the query samples in the current new task category.

[0065] The supporting samples in step S31 and query samples After steps S21-S24, the distance of the j-th query sample to the c-th class is obtained. The category corresponding to the maximum distance between the j-th query sample and all C categories is taken as the predicted category of the query sample. Query samples for all j∈[1,2,…,C×M] of the new class They are all categorized.

[0066] Step S33: Classify the query samples in the next task.

[0067] Next task number Repeat steps S31 and S32 until... in The total number of tasks for upsampling the new data type.

[0068] On the other hand, the present invention also provides an image classification apparatus based on category-related feature reconstruction, as shown in FIG4, comprising the following modules to implement the image classification method based on category-related feature reconstruction described above:

[0069] Embedded module f θ , used to extract features from input image data, where θ is a learnable parameter;

[0070] It supports a feature filtering module that removes irrelevant or background interference features based on the importance of local features of the supporting samples, retaining only the features related to the category.

[0071] Feature Reconstruction Module Reconstruct the features of the query samples using the filtered category-related features. These are learnable parameters;

[0072] The distance metric module is used to calculate the cosine distance between the reconstructed sample features and the query sample features, and to classify the query sample based on the magnitude of the cosine distance.

[0073] This invention employs a feature reconstruction approach. First, a support feature filtering module filters irrelevant and background features, retaining key category-related features. These key features are then used to reconstruct features within the same category, enabling the model to extract key category-related features. By constructing a category-related feature reconstruction network, highly discriminative features are learned, thereby improving the ability to classify few-sample images. This addresses the problems of visual feature misalignment, interference from irrelevant and background features, and other issues inherent in current metric-based few-sample image classification methods that directly measure sample features, ensuring the accuracy of the measurement results.

[0074] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention, based on the ideas provided by the present invention, should be considered within the scope of protection of the present invention.

Claims

1. An image classification method based on category-related feature reconstruction, characterized in that, The process includes the following steps: S1, Data preparation, which includes training and test sets. The class spaces of the training and test sets are mutually exclusive. The training set is used as base class data to train the model, and the test set is used as new class data to evaluate the model's small sample image classification performance; S2, Training class-related features on base class data, reconstructing the small sample image classification model using class-related features, and optimizing the model parameters to obtain the optimal classification model; The small sample image classification model reconstructed based on class-related features consists of four parts: an embedding module, a support feature filtering module, a feature reconstruction module, and a distance metric module; The embedding module is used to extract features from the input image data; The support feature filtering module removes irrelevant features or background and other interfering features based on the importance of the local features of the support samples, retaining only the features related to the category; The feature reconstruction module uses the filtered class-related features to reconstruct the query sample features, obtaining the reconstructed query sample features; The distance metric module calculates the cosine distance between the reconstructed sample features and the query sample features, and classifies the query sample based on the proximity of the cosine distance; Step S2 includes the following sub-steps: Step S21, Extracting sample features is a task for the base class data. ,in ， ， Represents the first in the base class data The class of One supporting sample, Represents the first in the base class data One query sample, Represents the first in the base class data The class of One-hot labels for each supporting sample. If the sample belongs to the first Class, then the first One position has a value of 1, and the rest have a value of 0; Represents the first in the base class data One-hot labels for each query sample. If the sample belongs to the first Class, then the first One position has a value of 1, and the rest have a value of 0; This sets the training set support samples... and query samples They are sent to the embedded module respectively. Extract features to obtain supporting sample features and query sample features , in These are the dimension, length, and width of the feature, respectively. The vector is called a local feature, and each sample has a total of Local features; Feature extraction module Using convolutional neural networks, The parameters are learnable, specifically comprising four convolutional blocks, each containing the following: an output dimension of 64... A convolutional layer with a stride of 1 and input padding of 1; one A max pooling layer; a batch normalization layer and a non-linear activation function ReLU; Step S22: Filter irrelevant local features from the local features of the support samples obtained in step S21. , each All the local features of a class are put together to form the first A feature pool of a class, represented as Then for each In the feature pool The average of the local features is used to obtain the i-th local feature. Global average features of a class Then calculate the first... Feature pool of classes Each local feature in Compared with global average features cosine distance Let i∈[1,2,…,hwk] represent the index of each local feature, and the cosine distance is specifically represented as: And form a set of all cosine distances. in Denotes the 2-norm; for a set Sort the mid-cosine distances from largest to smallest, and select the first... Each record corresponds to an index. , will this Local features corresponding to each index This forms a set, which is then concatenated to obtain the filtered local feature vector. Step S23: Feature Reconstruction. The filtered local feature vectors obtained in step S22 are... Input Feature Reconstruction Module The query sample features are reconstructed to obtain the reconstructed query sample features. Feature Reconstruction Module Specifically, it is a Transformer network. The parameters are learnable, where the Transformer computation includes an Attention layer, a layer of regularized LN, and a multilayer perceptron (MLP). The specific computation of the Attention is as follows: ,in, These represent the query, key, and value used in Attention computation, respectively. yes The transpose of the filtered local feature vector. Features are obtained by inputting Attention. ： ,in, ， and They are respectively Learnable weight parameters; the weights obtained in the previous step Input LN and MLP and then perform calculations to obtain the reconstructed query sample features: Step S24: Calculate the cosine distance using the cosine distance formula to calculate the query sample features obtained in step S21. and the reconstructed query sample features obtained in step S23 The cosine similarity between them, the cosine distance formula is: ， Indicates the first The query sample and the first Cosine similarity of classes; Step S25: Calculate classification loss and optimize the model using the cosine similarity obtained in step S24 and the true labels of the query samples. The cross-entropy loss is calculated using the following formula: ,in, The cross-entropy loss function is used; the embedding module is optimized using the SGD optimization algorithm. and feature reconstruction module parameters and The specific calculation formula for the SGD optimization algorithm is as follows: ,in The learning rate is initially set to 0.

01. for and gradient, The result is the optimized result; then the number of tasks. Resampling task in base class data Repeat steps S21-S25 until... Step S26: Output the optimal category-related feature reconstruction model; In the iteration of step S25, The smallest model is taken as the optimal model; S3, use the optimal model to classify the test images.

2. The image classification method based on category-related feature reconstruction according to claim 1, characterized in that, Step S1 divides the data into training sets. and test set Two parts, of which, and These are the image samples and their corresponding categories from the training set. The index of the image samples in the training set. and These are the image samples and their corresponding categories in the test set. The index of the image samples in the test set; and These are the categories of the training set and the test set, respectively, and the category spaces of these two parts are mutually exclusive. ， This represents the total number of samples in the training set. This represents the total number of samples in the test set.

3. The image classification method based on category-related feature reconstruction according to claim 2, characterized in that, Step S1 organizes the base class data and the new class data into -way -Shot classification task, for training set Randomly select from them There are 10 categories, and each category is randomly selected. 1 sample, of which One sample as supporting samples ， One sample was used as the query sample. Supporting samples and query samples To constitute a task Where e = 1, 2, ..., E is the index of the task. The total number of tasks for upsampling the base class data; for the test set Randomly select from them There are 10 categories, and each category is randomly selected. 1 sample, of which One sample as supporting samples ， One sample was used as the query sample. Supporting samples and query samples To constitute a task ,in It is the index of the task. The total number of tasks for upsampling the new data type.

4. The image classification method based on category-related feature reconstruction according to claim 1, characterized in that, Step S3 includes the following sub-steps: Step S31, New class task construction: a task for the new class of data. ,in ， ， Indicates the first in the new class of data The class of One supporting sample, Indicates the first in the new class of data One query sample, Indicates the first in the new class of data The class of One-hot labels for each supporting sample. If the sample belongs to the first Class, then the first One position has a value of 1, and the rest have values ​​of 0; Step S32: Classify the query samples in the current new type of task and the supporting samples from step S31. and query samples After steps S21-S24 respectively, the new class of data is obtained. For the query sample of the _th Distance between classes ; will the first One query sample and all The category corresponding to the maximum distance between the categories is taken as the predicted category of the query sample. ;Query samples of new classes for all j∈[1,2,…,C×M] All are classified; Step S33: Classify the query samples in the next task, then continue with the task number. Repeat steps S31 and S32 until... 。 5. An image classification device based on category-related feature reconstruction, characterized in that, The image classification method based on category-related feature reconstruction as described in any one of claims 1-4 includes the following modules: an embedding module for extracting features from the input image data. These are learnable parameters; It supports a feature filtering module that removes irrelevant or background interference features based on the importance of local features of the supporting samples, retaining only the features related to the category. The feature reconstruction module reconstructs the features of the query samples using the filtered category-related features. These are learnable parameters; the distance metric module is used to calculate the cosine distance between the reconstructed sample features and the query sample features, and to classify the query sample based on the magnitude of the cosine distance.

Images