Small sample image classification method and device based on multilevel feature extraction and prototype updating

Abstract

The invention discloses a small sample image classification method and device based on multi-level feature extraction and prototype updating. The method comprises the steps of obtaining a to-be-classified image; the to-be-classified image is processed based on the multi-level feature extraction network, a classification result of the to-be-classified image is obtained, and each Transform encoder of the multi-level feature extraction network corresponds to each convolution block embedded into the neural network; the method for processing the to-be-classified image based on the multi-level feature extraction network comprises the steps that all layers of all Transform encoders based on the Vision Transform network capture all feature representations in the to-be-classified image, and spectrum feature dimension reduction pooling operation is carried out on all the feature representations extracted from the last four layers of all the encoders once every two layers, each feature representation after dimensionality reduction and noise suppression is obtained; and outputting a classification result, so that the classification performance and robustness of the model under the condition of a small amount of data are improved.

Description

Technical Field

[0001] The present application relates to the field of small sample image classification, and in particular to a small sample image classification method and device based on multi-level feature extraction and prototype updating. Background Art

[0002] In recent years, the rapid development of deep learning technology has made it achieve remarkable results in the field of image classification. Deep learning can extract effective image features from large-scale data through neural networks, significantly improving the accuracy and robustness of classification models. However, in practical applications, especially in the fields of medicine, military and finance, model training often faces the problem of insufficient data due to privacy protection, high data collection costs and scarcity of labeled samples. This data scarcity limits the promotion of traditional deep learning technology and forces researchers to explore new methods to achieve efficient learning under limited sample conditions.

[0003] Few-shot learning came into being, dedicated to completing classification tasks efficiently with limited samples. Unlike traditional deep learning that relies on massive data, few-shot learning often builds a cognitive foundation on the existing rich basic category data, and then quickly migrates and adapts in the target category of a small number of samples, thereby significantly reducing the dependence on large-scale labeled data. In few-shot learning, a key goal is to design a feature space that can significantly distinguish between inter-class features while achieving compact expression of intra-class features.

[0004] Although existing methods have made some progress in the field of small sample learning, they still face significant challenges in dealing with highly complex distributions, blurred boundaries between categories, and scenes with more noise interference. For example, in medical image analysis, the morphology and distribution differences of different lesions are often very subtle, and there may be multiple subtypes or variants under the same category. If the embedding space cannot accurately distinguish these subtle differences, it may lead to overlapping features of different categories. For example, in the fine-grained classification task of species identification, the appearance differences of birds of different categories may only be reflected in local feathers or markings. The category boundaries themselves are not clear enough, and slight noise or angle changes will make it difficult for the model to distinguish. At this point, even if metric learning or attention mechanisms have been adopted, once the quality of the embedding space decreases, it is difficult for the model to maintain sufficient discrimination. Therefore, how to obtain local or subtle features more accurately and maintain the robustness of the model to interference under small sample conditions is still an urgent problem to be solved in the industry.

[0005] In addition, although various methods represented by prototype networks can quickly generate a category center to complete small sample recognition, once encountering new categories, their internal differences or external background complexity are high, it is difficult to form an accurate category representation based on the initial small number of samples, and additional samples or subsequent iterative updates are often required to correct the prototype. At the same time, if the new categories are already highly similar or in a transitional state, the prototype method will be even more inadequate in its portrayal of the category. This deficiency is particularly prominent in complex scenarios, such as the blurred edges of tissue segmentation areas in medical images or the drift of category features caused by shooting angles and lighting changes in field scenes, which will further increase the difficulty of model discrimination. Summary of the invention

[0006] The main purpose of this application is to provide a small sample image classification method and device based on multi-level feature extraction and prototype updating, aiming to improve the classification accuracy and classification robustness in small sample scenarios.

[0007] To achieve the above-mentioned purpose, the present application provides a small sample image classification method based on multi-level feature extraction and prototype updating, including: obtaining an image to be classified; processing the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each Transformer encoder of the improved network has each convolution block; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the VisionTransformer network captures each feature representation in the image to be classified respectively, and for each feature representation extracted in the last four layers of each encoder, performs a spectral feature dimensionality reduction pooling operation every two layers to obtain a tokens matrix after dimensionality reduction and noise suppression, wherein the feature representation refers to a feature vector with both local information and global structural information, and the convolution block is a plurality of convolution kernels with residual connection; and outputs the classification result.

[0008] Optionally, each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, including: using the multi-head self-attention module of each encoder to capture the global features of different regions in the image to be classified; using each convolution block to capture the local features of the local region in the image to be classified; and performing weighted fusion of the local features and the global features to obtain the feature representation output by each layer of each encoder.

[0009] Optionally, for each feature representation extracted from the last four layers of each encoder, a spectral feature dimensionality reduction pooling operation is performed every two layers to obtain a tokens matrix after dimensionality reduction and noise suppression, including: performing graph clustering and singular value decomposition on each feature representation to obtain dimensionality reduction features; clustering the dimensionality reduction features based on a clustering algorithm to obtain a tokens matrix after dimensionality reduction and noise suppression.

[0010] Optionally, the tokens represented by each feature are converted into a graph structure, and an adjacency matrix is ​​obtained based on the graph structure; a normalized Laplace matrix is ​​calculated based on the adjacency matrix; singular value decomposition is performed on the normalized Laplace matrix, and multiple eigenvalues ​​are selected to select the corresponding main feature components to obtain a eigenvector belonging to a low-dimensional space.

[0011] Optionally, clustering the reduced-dimensional features based on a clustering algorithm to obtain a tokens matrix after dimensionality reduction and noise suppression includes: using a K-Means clustering algorithm to perform semantic similarity aggregation on the feature vectors belonging to the low-dimensional space to obtain a clustered tokens matrix; and performing an averaging operation on the clustered tokens matrix to obtain a new tokens matrix after dimensionality reduction and noise suppression.

[0012] Optionally, the method also includes: in the training stage, initializing a prototype vector for each category in the support set, wherein each prototype vector corresponds to the center position of the category to which it belongs in the feature space; inputting the query sample into the multi-level feature extraction network to obtain a high-dimensional embedding vector of the query sample; using the multi-level feature extraction network to calculate the similarity between the high-dimensional embedding vector and each prototype vector, and updating the prototype vector based on the high-dimensional embedding vector with the greatest similarity to obtain the updated prototype vector; using the updated prototype vector to train the multi-level feature extraction network to obtain a feature extraction network with updated parameters.

[0013] Optionally, before calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, the method further includes: performing exponentially weighted smoothing processing on the high-dimensional embedding vector.

[0014] Optionally, the training of the multi-level feature extraction network includes a meta-training stage and a fine-tuning stage. Before using the multi-level feature extraction network to calculate the similarity between the high-dimensional embedding vector and each of the prototype vectors, the method also includes: in the meta-training stage, the multi-level feature extraction network is trained as a whole so that the multi-level feature extraction network jointly learns the features required for the task to be processed; in the fine-tuning stage, the main parameters of the multi-level feature extraction network are frozen, and only the convolution blocks embedded in each of the encoders are trained.

[0015] Optionally, the similarity calculation formula includes: one of a cosine similarity calculation formula, a Euclidean distance calculation formula and a Mahalanobis distance calculation formula.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a small sample image classification method based on multi-level feature extraction and prototype updating, including: an acquisition module, used to acquire an image to be classified; a processing module, used to process the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each Transformer encoder of the multi-level feature extraction network has each convolution block; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified respectively, and for each feature representation extracted from the last four layers of each encoder, performs a spectral feature dimensionality reduction pooling operation every two layers to obtain a tokens matrix after dimensionality reduction and noise suppression, wherein the feature representation refers to a feature vector with both local information and global structural information, and the convolution block is a plurality of convolution kernels with residual connection; an output module, used to output the classification result.

[0017] The embodiment of the present application proposes a small sample image classification method and device based on multi-level feature extraction and prototype updating, which obtains an image to be classified; processes the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each Transformer encoder of the multi-level feature extraction network corresponds to each convolution block embedded in the neural network; processing the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified respectively, and for each feature representation extracted in the last four layers of each encoder, performs a spectral feature dimensionality reduction pooling operation every two layers to obtain each feature representation after dimensionality reduction and noise suppression, wherein the feature representation refers to feature information that has both local information and global structural information; outputs the classification result, thereby improving the classification performance and robustness of the model in the case of small amounts of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart diagram of an embodiment of a small sample image classification method based on multi-level feature extraction and prototype updating provided by the present application;

[0019] Figure 2 A schematic diagram of an embodiment of a small sample image classification method based on multi-level feature extraction and prototype updating provided by the present application;

[0020] Figure 3 A structural block diagram of an embodiment of a small sample image classification device based on multi-level feature extraction and prototype updating provided in this application.

[0021] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] In the existing technology, due to the limitation of sample number in small sample classification tasks, the training and generalization ability of the model is often seriously affected. Especially in the traditional convolutional neural network and ViT framework, the extracted features may have problems such as overlap between categories and uneven distribution, thus reducing the accuracy and stability of classification.

[0024] Current research attempts to alleviate the above problems by enhancing feature expression capabilities and optimizing classification modules. For example, CNN is widely used for its powerful local feature extraction capabilities, but its global modeling capabilities are insufficient; ViT can capture global features, but lacks in capturing local information details. In addition, prototype learning methods, as a meta-learning-based strategy, can effectively reduce the distribution overlap between categories by aggregating sample features into category prototypes, but the dynamic update mechanism of prototypes still needs to be improved in complex scenarios.

[0025] In some existing technologies, a convolution module is added after each Transformer layer to enhance the ability to extract local features. This method solves the shortcomings of pure Transformer in processing local details to a certain extent, but its feature fusion method is relatively simple and lacks the ability to express multi-scale and multi-level features;

[0026] In other existing technologies, a query-support Transformer framework is proposed for small sample learning tasks. This framework realizes the effective fusion and metric learning of global and local features of the support set and query set by introducing cross-scale interactive feature extractors, sampleFormer and patchFormer. However, its much more complex module design leads to high computational complexity, especially when processing high-resolution images and a large number of block-level features, the computing resource requirements increase significantly; secondly, this technology lacks effective feature dimensionality reduction and redundancy control mechanisms, which makes it difficult to operate efficiently in resource-constrained environments;

[0027] In other existing technologies, query samples are used to generate query prototypes and merge them with support set prototypes, which effectively improves the dynamic adaptability of category prototypes. However, its update strategy requires comprehensive interaction between all query samples and category prototypes, which has high computational complexity and insufficient robustness to noise samples or outliers, which may cause the prototype to be biased. In addition, the embedded features are not explicitly optimized before the prototype operation, and the redundancy and noise problems of high-dimensional features may further affect the accuracy and efficiency of the update.

[0028] In the existing technologies for small sample classification tasks, there are the following main shortcomings: First, the feature representation ability of existing models is limited. Models represented by the Vision Transformer (ViT) network, although they have strong global feature modeling capabilities, are relatively weak in capturing small-sized targets or micro-textures due to the lack of convolution operations. On the contrary, the classic convolutional neural network (CNN, such as the ResNet series) is more sensitive to local details, but is slightly insufficient in large-scale context modeling. If only one of the structures is relied on, it is difficult for the model to simultaneously obtain rich global context and fine local texture expression under small sample conditions, resulting in limited classification effect. Secondly, in a small sample environment, due to the lack of massive samples to compensate or learn more robust features, the negative impact of noise interference on model discrimination is very prominent, resulting in many feature extraction networks often not considering the impact of background noise on feature representation during feature extraction, resulting in low classification accuracy. Prototype-based learning methods perform well in small sample classification. For categories with complex distribution or multimodal situations, this fixed prototype is difficult to adapt to the morphological changes or noise shocks that occur in the query stage in a timely manner. Once the distribution of categories drifts in actual scenarios, the accuracy of model recognition will drop significantly.

[0029] In view of the above shortcomings, the small sample image classification method based on multi-level feature extraction and prototype updating provided by the present application aims to: propose a feature extraction module that combines the advantages of ViT and convolution. The global context is obtained through multi-head self-attention, and the convolution block is introduced to capture local details, and the two are fused, so as to take into account both global associations and local high-resolution textures, and improve the classification accuracy in small sample scenarios. For complex environments or task scenarios, the adaptability of the model to uncertainty conditions is enhanced by the method of spectral clustering and pooling. Using the idea of ​​spectral clustering, high-dimensional redundant and noisy parts are effectively compressed and aggregated to help maintain a relatively compact and highly recognizable feature space in the case of small samples. A prototype update mechanism based on distance weighted interpolation in the inference stage is designed to replace the traditional prototype network with a fixed prototype method that only uses a simple mean. This dynamic update enables the category prototype to absorb the information of the query sample, reduce overlap between categories, and thus improve the robustness of the classification.

[0030] In small sample image classification, given a dataset D = (x i ,y i ), x i ∈R H×W×3 is the i-th image, y i ∈Y is its category label. D is divided into basic data sets D b And the new dataset D n Different from traditional image recognition tasks, small sample learning focuses on the classification of new categories after the training phase. Specifically, the label set Y is divided into the basic label set Y b and the new label set Y n ,satisfy In D n Create small sample tasks on the support set, each task contains N categories, and each new category has only K labeled samples. and queryset The model is composed of , M is the number of test samples. The model is trained on the support set s to build a classifier and evaluated on the query set q.

[0031] Reference Figure 1 The first embodiment of the present application provides a small sample image classification method based on multi-level feature extraction and prototype updating. The method can be executed by a processor of a terminal or a server. The small sample image classification method based on multi-level feature extraction and prototype updating may include:

[0032] S10, obtaining an image to be classified;

[0033] Before this step, it also includes preprocessing the classified images to meet the subsequent training and feature extraction requirements. The data preprocessing stage of this application is mainly to enhance the data. Due to the limited amount of data faced by small sample learning, the data enhancement technology in the data preprocessing stage is mainly through rotating, flipping, cropping, scaling and other operations on the original image. The data enhancement technology in the data preprocessing stage mainly includes random rotation (±15 degrees), horizontal and vertical flipping, random cropping (cropping ratio is 80%), and scaling (scaling factor is 0.8). This not only expands the scale of the data set, but also enhances the generalization ability of the model, enabling it to better cope with the changing situations in practical applications.

[0034] S20, processing the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each encoder of the multi-level feature extraction network has each convolution block; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, and performs a spectral feature dimension reduction pooling operation every two layers on each feature representation extracted from the last four layers of each Transformer encoder to obtain a tokens matrix after dimension reduction and noise suppression, wherein the feature representation refers to a feature vector having both local information and global structural information, and the convolution block is a plurality of convolution kernels with residual connections;

[0035] In this step, the global context is obtained through multi-head self-attention, and convolution blocks are introduced to capture local details, and the two are fused to take into account global correlation and local high-resolution texture, thereby improving the classification accuracy in small sample scenarios. This application also enhances the adaptability of the multi-level feature extraction network to uncertainty conditions for complex environments or task scenarios through spectral clustering and pooling methods. Using the idea of ​​spectral clustering, high-dimensional redundant and noisy parts are effectively compressed and aggregated to help maintain a relatively compact and highly recognizable feature space in small sample conditions. It should be noted that the multi-level feature extraction network is built on the basis of the improved ViT network.

[0036] S30: Output the classification result.

[0037] In summary, the embodiment of the small sample image classification method based on multi-level feature extraction and prototype updating proposed in this application obtains the classification result of the image to be classified by processing the image to be classified based on the multi-level feature extraction network, wherein each Transformer encoder of the multi-level feature extraction network corresponds to each convolution block embedded in the neural network; processing the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, and performs a spectral feature dimensionality reduction pooling operation every two layers for each feature representation extracted from the last four layers of each encoder, and obtains each feature representation after dimensionality reduction and noise suppression, wherein the feature representation refers to feature information with both local information and global structural information; outputting the classification result, combining the global attention mechanism with the local convolution path, which can not only capture the long-range dependency relationship between different regions in the image, but also retain the edge, texture and other detail information, and more effectively mine the image information in the small sample scenario. Secondly, the dimensionality reduction through spectral feature pooling further compresses the feature space dimension, reduces noise interference, and reduces the amount of calculation during classification. Compared with the traditional method that simply relies on global feature extraction, this application has a more flexible and hierarchical feature expression.

[0038] In an embodiment of the present application, each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, including:

[0039] S201, using the multi-head self-attention module of each encoder to capture the global features of different regions in the image to be classified;

[0040] In this step, the processor uses Vision Transformer (ViT) as the base network to learn global attention of the image. ViT divides the image into several patches and treats them as a sequence input to the Transformer encoder to capture the long-range dependencies between the patches of the image. Since small sample data usually contains key information distributed in local areas, ViT can effectively integrate the global context scattered between different patches to obtain global features.

[0041] S202, using each of the convolution blocks to capture local features of a local area in the image to be classified;

[0042] In this step, unlike pure ViT, a convolution block is added after each layer of Transformer encoder to embed the local feature enhancement path to capture the input feature F l-1Local details The convolution block is composed of multiple small convolution kernels: 1×1 convolution, 3×3 convolution, and 1×1 convolution. The 1×1 convolution is used to reduce the input channel dimension and reduce the amount of calculation. The 3×3 convolution is the core part of the convolution block and is used to capture the visual information of the local area. The following 1×1 convolution is used to restore the channel dimension and return the features to the original channel size. High-resolution features such as the edges and textures of objects can be extracted from local areas of space.

[0043] S203, performing weighted fusion on the local features and the global features to obtain feature representations output by each layer of each encoder.

[0044] Then the local features Global features with ViT Weighted Fusion α is a learnable parameter used to adjust the ratio between local features and global features, so as to achieve multi-scale and multi-level feature expression. This can preserve ViT's ability to integrate global information while taking into account the local fine structure.

[0045] By executing the above steps S201-S203, the processor realizes the use of the joint architecture of the convolutional neural network and the Vision Transformer network to fully combine local features and global features, and extracts richer and more distinctive feature representations. In the process of obtaining the feature representation, first, the processor uses the Vision Transformer network to extract global features of the input image, and captures the long-distance dependencies between different regions in the image through the multi-head self-attention mechanism of the Vision Transformer network, thereby generating a feature representation with rich global context. Secondly, the processor embeds convolution blocks in each layer of the encoder to capture the detailed information and local features in the image. Finally, the processor weightedly fuses the global features and local features to form a rich and diverse feature representation. This joint architecture can not only capture the detailed texture in the image, but also effectively understand the global structural information of the image, thereby significantly improving the classification performance in the case of small samples.

[0046] In an embodiment of the present application, for each of the feature representations extracted from the last four layers of each of the encoders, a spectral feature dimension reduction pooling operation is performed every two layers to obtain a tokens matrix after dimension reduction and noise suppression, including:

[0047] S204, performing graph clustering and singular value decomposition on each of the feature representations to obtain dimensionality reduction features;

[0048] In this step, the tokens matrix T∈R is input after feature extraction. N×DWith the adjacency weight matrix S0∈R N×N Define the Laplace matrix L = D-S0, where D is a diagonal matrix, Compute the normalized Laplace: To enhance numerical stability;

[0049] S205, clustering the dimension reduction features based on a clustering algorithm to obtain a tokens matrix after dimension reduction and noise suppression;

[0050] In this step, the normalized Laplace matrix L is decomposed using singular value decomposition. norm Perform dimensionality reduction and extract the main feature components. Select the first N0 singular vectors The subspace basis vectors are composed to reflect the low-dimensional embedding structure of the graph. The first N0 eigenvectors are stacked to obtain the matrix Each row v of the matrix U i That is, the representation of tokeni in low-dimensional space. The representation of each token in low-dimensional space is All v i Aggregate them and use K-means to cluster them into N0 clusters. Each cluster will be merged into a new token later. Average the features of tokens in the same cluster to get a new token: Among them C k Represents the index of all original tokens in the kth cluster. Finally, N0 new tokens are obtained to form a new token matrix

[0051] It should be noted that, in this application, features are represented as feature tokens.

[0052] In the above steps, since high-dimensional features often contain a lot of redundant information and noise, the processor performs a spectral feature dimensionality reduction pooling operation on the extracted feature tokens every two layers in the last four layers of each encoder to reduce the redundancy between high-dimensional features and effectively suppress noise. The core idea of ​​the spectral feature dimensionality reduction pooling operation is to use the graph clustering method combined with the singular value decomposition technology to reduce the dimensionality of high-dimensional features, thereby optimizing feature representation, reducing redundant features, and improving computational efficiency and classification performance.

[0053] Specifically, step S204 may include the following execution process:

[0054] S2041. Convert each of the feature representations into a graph structure based on an adjacency matrix and a Laplacian matrix;

[0055] S2042: Perform singular value decomposition on the feature matrix in the graph structure to obtain a plurality of main feature components, and generate dimensionality reduction features based on the plurality of main feature components.

[0056] The processor implements the dimensionality reduction of feature tokens by executing the above steps S2041 and S2042. The above may include the following execution process:

[0057] First, the processor converts the fused high-dimensional feature representation into a graph structure by constructing an adjacency matrix and a Laplacian matrix. The adjacency matrix is ​​used to describe the connection relationship between features, while the Laplacian matrix further reveals the topological structure and feature distribution of the graph.

[0058] After the graph structure is established, the processor performs spectral dimensionality reduction. In this stage, the processor decomposes the feature matrix of the graph structure through the singular value decomposition method, extracts the main feature components, and maps the high-dimensional features to the low-dimensional space. The processor obtains a set of the most discriminative feature vectors through singular value decomposition, thereby retaining the main structural information of the data while effectively filtering out noise and redundant information. Compared with directly using feature dimensionality reduction, singular value decomposition can further improve the stability of dimensionality reduction, reduce redundancy, and optimize the computational efficiency of the multi-level feature extraction network in subsequent tasks. This dimensionality reduction method can reduce the dimension of the feature space while maintaining global semantic information, thereby reducing the complexity of subsequent calculations and making the feature representation more compact and discriminative.

[0059] Specifically, step S205 may include the following execution process:

[0060] Using the K-Means clustering algorithm to perform semantic similarity aggregation on the feature vectors belonging to the low-dimensional space to obtain a clustering tokens matrix;

[0061] An average operation is performed on the clustered tokens matrix to obtain a new tokens matrix after dimension reduction and noise suppression.

[0062] This step further aggregates the features after dimensionality reduction through KMeans clustering, classifies semantically similar feature tokens into the same category, and forms more representative feature clusters. Through this clustering operation, the multi-level feature extraction network can more effectively integrate similar information, reduce the redundancy of input data, and improve the ability to distinguish different categories in the final classification process, thereby improving the accuracy of classification.

[0063] On this basis, the small sample image classification method based on multi-level feature extraction and prototype updating provided by this application also includes:

[0064] S401, in the training phase, initializing a prototype vector for each category in the support set, wherein each prototype vector corresponds to the center position of the category to which it belongs in the feature space;

[0065] S402, inputting the query sample into the multi-level feature extraction network to obtain a high-dimensional embedding vector of the query sample;

[0066] This application first calculates the feature vectors of all supporting samples {f1,f2,…,f m} Perform simple mean or weighted mean calculation to obtain the initial prototype vector of category c

[0067]

[0068] Where m is the number of samples of category c in the support set, f i Represents the feature vector obtained after the i-th sample passes through the feature extraction network.

[0069] S403, using the multi-level feature extraction network to calculate the similarity between the high-dimensional embedding vector and each of the prototype vectors, and fine-tuning the prototype vector based on the high-dimensional embedding vector with the greatest similarity to obtain the fine-tuned prototype vector;

[0070] Calculate the distance between the query sample and the prototype. When a new query sample x is input, first obtain the embedding vector f through the above feature extraction process x , respectively, with all current category prototypes Calculate the distance and select the category prototype with the greatest similarity for subsequent updates;

[0071] (3.3) The prototype is updated using the basic interpolation formula to obtain the fine-tuned prototype vector:

[0072]

[0073] Among them: α∈(0,1) is the update rate, balancing the old prototype and new information.

[0074] Represents a weighting factor based on distance. The smaller the distance, the greater the weight and the stronger the update. When the distance threshold is too large, it is regarded as an outlier and the update is skipped to avoid the prototype from being pulled away and protect the prototype from being disturbed by extreme samples.

[0075] S404: Use the fine-tuned prototype vector to train the multi-level feature extraction network to obtain a fine-tuned Vision Transformer network.

[0076] By executing the above steps S401-S404, the processor continuously corrects the category prototype in the query phase through distance-based weighted interpolation, which greatly enhances the adaptive ability of the multi-level feature extraction network to the new sample distribution. Most traditional small sample classification methods have fixed category prototypes and are difficult to adapt to morphological changes or distribution drifts in query samples. The prototype update mechanism proposed in this application can make the category representation of the multi-level feature extraction network more in line with the actual environment while ensuring that the computational overhead is controllable.

[0077] Exemplarily, the process of secondary updating of the category prototype may include the following execution process:

[0078] In the training phase, the processor initializes a prototype vector for each category in the support set. Each prototype vector represents the center position of the category in the feature space. Secondly, the processor inputs the query sample into the multi-level feature extraction network and obtains its high-dimensional embedding vector through the feature extraction module. This embedding vector contains the global and local feature information of the query sample and is the key basis for subsequent matching and classification.

[0079] Next, the processor uses the Euclidean distance to calculate the similarity between the embedding vector of the query sample and all the class prototype vectors. This process determines which class the query sample is most likely to belong to. Based on the preliminary match, the query sample is assigned to the class with the highest similarity. The processor then updates the prototype vector of the class to better reflect the information of the new query sample. This update process fine-tunes the prototype vector by weighted fusion of the embedding vector of the query sample with the current class prototype vector. In this way, the class prototype can gradually adapt to the feature changes of the new sample and enhance the ability to capture the diversity within the class. At the same time, in order to prevent the class prototype from changing dramatically due to the influence of individual query samples, a smooth update strategy is used for the secondary update of the class prototype. This strategy ensures a smooth update process of the prototype vector by introducing an exponentially weighted average method to avoid offsets caused by noise or abnormal samples, thereby improving the overall stability of the multi-level feature extraction network.

[0080] Using cosine similarity as the similarity metric, the feature vector of the query sample is compared with the prototype vector of each category one by one, and the corresponding similarity score is calculated. The system can make classification decisions based on the calculated similarity score. The cosine similarity scores of the query sample and all category prototypes are sorted to identify the category with the highest similarity. The category with the highest similarity score is selected as the predicted category of the query sample. This selection process ensures high accuracy of the classification results, because the category with the highest score represents the category that is closest to the query sample's features. Finally, the system outputs the predicted category to which the query sample belongs, completing a complete classification process.

[0081] It is worth noting that the cosine similarity used to calculate the matching between known category classification and classification prototype can also be replaced by other distance measurement methods such as Euclidean distance and Mahalanobis distance.

[0082] In another alternative embodiment of the present application, the category prototype is continuously modified during the query phase based on distance-based weighted interpolation. It is also possible to consider using soft labels to assign query sample features to various categories and then update the classification prototype evenly.

[0083] In another alternative embodiment of the present application, the weight value of the category prototype during the updating process may be changed from a fixed weight to an adaptive weight to obtain a more effective classification prototype updating ratio.

[0084] The training of the multi-level feature extraction network includes a meta-training phase and a fine-tuning phase. Before step S403, the method further includes:

[0085] In the meta-training stage, the multi-level feature extraction network is trained as a whole so that the multi-level feature extraction network can jointly learn the features required for the task to be processed.

[0086] Since the supervisory information in small sample data is limited, if the entire network is fine-tuned in the subsequent fine-tuning stage, it is very likely to cause overfitting due to too many learnable parameters, and it will also significantly increase the training overhead. For this reason, the present application trains the model as a whole in the meta-training stage so that the entire model can learn the features required for the task together; in the fine-tuning stage, the strategy of freezing the ViT main body parameters and only training the convolution blocks embedded in each of the encoders is adopted. On the one hand, the overall training in the meta-training stage ensures that the model can make necessary adjustments for small sample tasks; on the other hand, the fine-tuning stage only updates a small number of parameters of the newly added modules, which can reduce the risk of overfitting, reduce the amount of calculation, and allow the model to more stably improve the classification performance on limited sample data.

[0087] This application freezes some parameters and only fine-tunes the local convolution path and classification head. It not only makes full use of the common features of large-scale pre-training, but also significantly reduces the risk of overfitting under small samples. Compared with full fine-tuning, this strategy can reduce training time and computing power while ensuring accuracy.

[0088] The match between the query sample and the category prototype is calculated by cosine similarity. Specifically, cosine similarity is used as a metric to measure the directional consistency between the query sample features and the category prototype features. The calculation formula for cosine similarity is:

[0089]

[0090] Where: f x Represents the feature vector of the query sample. Represents the prototype vector of category c. Respectively represent the vector f x and The Euclidean norm of ;

[0091] Through this formula, the similarity score between the query sample and each category prototype can be calculated.

[0092] The best category is selected as the classification result based on the score. After calculating the cosine similarity between the query sample and all category prototypes, the category with the highest matching degree is selected as the classification result based on the similarity score.

[0093] The formula for classification decision is:

[0094]

[0095] in: represents the final classification result, C represents the set of all categories. Sim(q,c) represents the query sample f x With class prototype The cosine similarity score of ;

[0096] Using the common cross entropy loss function as the classification loss, first map the similarity to logits and then calculate the cross entropy loss:

[0097]

[0098] During the training or fine-tuning phase, backpropagation is performed on model parameters that are not frozen, and frozen layer parameters always remain unchanged.

[0099] The following is a further description of the technical effects of the present invention in conjunction with experiments:

[0100] The present invention is simulated by using pycharm software and python language on an Intel(R) Xeon(R) Gold 5222CPU@3.80GHz central processing unit, 125G memory, and Ubuntu 20.04.2 operating system.

[0101] Simulation experiment 1: The present invention and the existing small sample image classification model are used to classify and compare the general small sample benchmark test. The results are shown in Table 1. In this experiment, the general small sample benchmark test uses two datasets, mini-ImageNet and tiered-ImageNet.

[0102] The mini-ImageNet dataset is a subset of the ImageNet dataset, containing 100 categories, each with 600 images, and the image size is uniformly 84x84 pixels. In this experiment, these 100 categories are further divided into 64 classes for training, 16 for validation, and 20 for testing.

[0103] Tiered-ImageNet is also based on a larger subset of ImageNet, containing 34 superclasses, a total of 608 categories, and different numbers of images per category. To ensure significant domain differences between training and evaluation, 351 categories are used for training, 97 categories are used for validation, and 160 categories are used for testing. Due to its large number of categories and wide distribution, tiered-ImageNet is suitable for testing the performance of models in more challenging multi-class small sample learning scenarios. Most research methods use 84×84 pixel images.

[0104] Table 1: Classification performance of mini-ImageNet and tiered-ImageNet

[0105]

[0106] According to the experimental results in the table, our model demonstrates excellent performance in common small-sample benchmarks (mini-ImageNet and tiered-ImageNet), outperforming other models in both 1-shot and 5-shot settings. Specifically:

[0107] On the mini-ImageNet dataset: In the 1-shot classification, our model achieved an accuracy of 69.45%, which surpassed all reference models and demonstrated its excellent classification ability in the case of very few samples. In the 5-shot classification, the accuracy of the model increased to 86.34%, continuing to maintain its leading position, proving its good generalization ability. On the tiered-ImageNet dataset: In the 1-shot classification, the model leads with an accuracy of 74.09%, showing its strong adaptability when dealing with more complex datasets. In the 5-shot classification, the accuracy further increased to 88.03%, continuing to lead all comparison models, indicating the advantage of our model in deeper data analysis.

[0108] The experimental results not only demonstrate the high efficiency of our model in handling image classification tasks of various scales and types, but also show its practicality and application potential in the field of small sample learning.

[0109] Simulation experiment 2: The fine-grained small sample benchmark test is classified and compared using the present invention and the existing small sample image classification model. The results are shown in Table 2. In this experiment, the fine-grained small sample benchmark test uses the cropped CUB dataset and Aircraft dataset.

[0110] The CUB dataset consists of approximately 11,788 images distributed in 200 bird categories, with an average of approximately 60 images per category. Each image in the CUB dataset is equipped with rich annotation information, covering bounding boxes, part locations, and attribute labels, providing rich visual and semantic information for fine-grained image analysis. The categories of the CUB dataset are randomly divided into 100 training classes, 50 validation classes, and 50 evaluation classes. We use a dataset that crops each image based on the manually annotated bounding boxes.

[0111] The Aircraft dataset consists of approximately 10,000 images, covering 100 different aircraft models, with approximately 100 images for each model. Consistent with the experimental settings of the CUB dataset, the Aircraft dataset is processed according to the same class division ratio, and the 100 aircraft models are randomly divided into 50 training classes, 25 validation classes, and 25 test classes. In addition, similar to the processing method of the CUB dataset, the images in the Aircraft dataset are cropped to ensure that the input image is evenly focused on the aircraft model within the bounding box provided by the dataset.

[0112] Table 2: CUB-Cropped and Aircraft classification performance

[0113]

[0114] According to the experimental results in the table, our model performs well in two fine-grained small sample benchmarks (CUB-Cropped and Aircraft), and shows excellent performance in both 1-shot and 5-shot test settings. These results are significantly better than traditional models such as MatchNet and ProtoNet, and even exceed several current advanced models, including DeepEMD, DeepBDC and FRN. This performance highlights the significant advantages of our model in handling fine-grained small sample classification problems and its great potential for practical applications.

[0115] refer to Figure 3 Based on the above method embodiment, the present application also provides a small sample image classification device based on multi-level feature extraction and prototype updating. The small sample image classification device based on multi-level feature extraction and prototype updating may include an acquisition module 101, a processing module 102 and an output module 103, wherein:

[0116] The acquisition module 101 is used to acquire the image to be classified;

[0117] The processing module 102 is used to process the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each encoder of the multi-level feature extraction network has each convolution block; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, and performs a spectral feature dimension reduction pooling operation every two layers on each feature representation extracted from the last four layers of each encoder to obtain a tokens matrix after dimension reduction and noise suppression, wherein the feature representation refers to a feature vector having both local information and global structural information;

[0118] The output module 103 is used to output the classification result.

[0119] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A small sample image classification method based on multi-level feature extraction and prototype updating, characterized in that: include: Get the image to be classified; Processing the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each Transformer encoder of the multi-level feature extraction network corresponds to each convolution block embedded in the neural network; Processing the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, and for each feature representation extracted in the last four layers of each encoder, performing a spectral feature dimension reduction pooling operation every two layers to obtain a tokens matrix after dimension reduction and noise suppression, wherein the feature representation refers to a feature vector with both local information and global structural information, and the convolution block is a plurality of convolution kernels with residual connection; The classification result is output.

2. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 1, characterized in that: Each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, including: Using the multi-head self-attention module of each encoder to capture the global features of different regions in the image to be classified; Using each of the convolution blocks to capture local features of a local area in the image to be classified; The local features and the global features are weightedly fused to obtain feature representations of outputs of each layer of each encoder.

3. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 1, characterized in that: For each of the feature representations extracted from the last four layers of each encoder, a spectral feature dimension reduction pooling operation is performed every two layers to obtain a tokens matrix after dimension reduction and noise suppression, including: Performing graph clustering and singular value decomposition on each of the feature representations to obtain dimensionality reduction features; The dimension reduction features are clustered based on a clustering algorithm to obtain a tokens matrix after dimension reduction and noise suppression.

4. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 3, characterized in that: The performing of graph clustering and singular value decomposition on each of the feature representations to obtain dimension reduction features includes: Convert the tokens represented by each of the features into a graph structure, and obtain an adjacency matrix based on the graph structure; Calculating a normalized Laplacian matrix based on the adjacency matrix; The normalized Laplace matrix is ​​subjected to singular value decomposition, and a plurality of eigenvalues ​​are selected to select the corresponding main eigencomponents to obtain eigenvectors belonging to the low-dimensional space.

5. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 3, characterized in that: The clustering algorithm is used to cluster the dimension reduction features to obtain a tokens matrix after dimension reduction and noise suppression, including: Using the K-Means clustering algorithm to perform semantic similarity aggregation on the feature vectors belonging to the low-dimensional space to obtain a clustering tokens matrix; An average operation is performed on the clustered tokens matrix to obtain a new tokens matrix after dimension reduction and noise suppression.

6. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 1, characterized in that: The method further comprises: In the training phase, a prototype vector is initialized for each category in the support set, wherein each prototype vector corresponds to the center position of the category to which it belongs in the feature space; Inputting the query sample into the multi-level feature extraction network to obtain a high-dimensional embedding vector of the query sample; Calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, and updating the prototype vector based on the high-dimensional embedding vector with the greatest similarity to obtain the updated prototype vector; The multi-level feature extraction network is trained using the updated prototype vector to obtain a fine-tuned Vision Transformer network.

7. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 6, characterized in that: Before calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, the method further includes: The high-dimensional embedding vector is smoothed based on exponential weighting.

8. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 6, characterized in that: The training of the multi-level feature extraction network includes a meta-training stage and a fine-tuning stage. Before calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, the method further includes: In the meta-training stage, the multi-level feature extraction network is trained as a whole so that the multi-level feature extraction network can jointly learn the features required for the task to be processed; In the fine-tuning stage, the main parameters of the multi-level feature extraction network are frozen, and only the convolution blocks embedded in each encoder are trained.

9. The small sample image classification method based on multi-level feature extraction and prototype updating as claimed in claim 6, characterized in that: The calculation formula of the similarity includes: One of the cosine similarity calculation formula, the Euclidean distance calculation formula, and the Mahalanobis distance calculation formula.

10. A small sample image classification device based on multi-level feature extraction and prototype updating, characterized in that: include: An acquisition module, used for acquiring images to be classified; A processing module, used for processing the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each Transformer encoder of the multi-level feature extraction network corresponds to each convolution block embedded in the neural network; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the VisionTransformer network captures each feature representation in the image to be classified, and performs a spectral feature dimension reduction pooling operation every two layers on each feature representation extracted from the last four layers of each encoder to obtain a tokens matrix after dimension reduction and noise suppression, wherein the feature representation refers to a feature vector having both local information and global structural information, and the convolution block is a plurality of convolution kernels connected by residuals; An output module is used to output the classification result.

Images