A small sample image classification method and device based on multi-level feature extraction and prototype updating

Abstract

This application discloses a few-sample image classification method and apparatus based on multi-level feature extraction and prototype update. The method involves acquiring an image to be classified; processing the image using a multi-level feature extraction network to obtain a classification result; where each Transformer encoder of the multi-level feature extraction network corresponds to a convolutional block embedded in the neural network; and processing the image using the multi-level feature extraction network includes: each layer of the Transformer encoder based on the Vision Transformer network captures a feature representation in the image to be classified, and performing spectral feature dimensionality reduction pooling on the feature representations extracted from the last four layers of each encoder every two layers to obtain dimensionality-reduced and noise-suppressed feature representations; and outputting the classification result. This improves the classification performance and robustness of the model under limited data conditions.

Description

Technical Field

[0001] This application relates to the field of few-sample image classification, and in particular to a few-sample image classification method and apparatus based on multi-level feature extraction and prototype update. Background Technology

[0002] In recent years, the rapid development of deep learning technology has led to significant achievements in image classification. Deep learning, through neural networks, can extract effective image features from large-scale datasets, significantly improving the accuracy and robustness of classification models. However, in practical applications, especially in fields such as medicine, military, and finance, model training often faces the problem of insufficient data due to privacy concerns, high data acquisition costs, and the scarcity of labeled samples. This data scarcity limits the widespread adoption of traditional deep learning techniques, forcing researchers to explore new methods for achieving efficient learning under limited sample conditions.

[0003] Few-shot learning emerged to efficiently complete classification tasks with limited sample sizes. Unlike traditional deep learning, which relies on massive amounts of data, few-shot learning typically builds its cognitive foundation on existing rich datasets of basic categories, and then rapidly transfers and adapts its knowledge to the target category using a small number of samples, thus significantly reducing its dependence on large-scale labeled data. A key objective of few-shot learning is to design a feature space that can significantly distinguish between inter-class features while achieving a compact representation of intra-class features.

[0004] While existing methods have made some progress in few-shot learning, they still face significant challenges when dealing with highly complex distributions, blurred class boundaries, and scenarios with significant noise interference. For example, in medical image analysis, the morphological and distributional differences between different lesions are often very subtle, and multiple subtypes or variations may exist within the same category. If the embedding space cannot accurately distinguish these subtle differences, it may lead to feature overlap between different categories. Similarly, in fine-grained classification tasks for species identification, the appearance differences between different bird categories may only be reflected in local feathers or markings, and the class boundaries themselves are not clear enough. Slight noise or changes in angle can make it difficult for the model to distinguish them. In this case, even with metric learning or attention mechanisms, once the quality of the embedding space deteriorates, the model struggles to maintain sufficient discriminative power. Therefore, how to more accurately acquire local or subtle features and maintain the model's robustness to interference under few-shot conditions remains a pressing problem for the industry.

[0005] Furthermore, while various methods, such as prototype networks, can quickly generate a category center for small-sample recognition, they struggle to form accurate category representations with only a small number of initial samples when encountering new categories, internal differences, or complex external backgrounds. Additional samples or subsequent iterative updates are often required to refine the prototype. Moreover, if new categories are already highly similar or in a transitional state, the prototype method's ability to characterize categories becomes even more inadequate. This deficiency is particularly pronounced in complex scenarios, such as blurred edges in tissue segmentation regions in medical images or category feature drift caused by changes in shooting angle and lighting in outdoor scenes, further increasing the difficulty of model discrimination. Summary of the Invention

[0006] The main purpose of this application is to provide a few-sample image classification method and apparatus based on multi-level feature extraction and prototype update, which aims to improve the classification accuracy and robustness in few-sample scenarios.

[0007] To achieve the above objectives, this application provides a few-sample image classification method based on multi-level feature extraction and prototype update, comprising: acquiring an image to be classified; processing the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each Transformer encoder of the improved network has a convolutional block; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder of the VisionTransformer network captures a feature representation in the image to be classified, and performing a spectral feature dimensionality reduction pooling operation every two layers on each feature representation extracted from the last four layers of each encoder to obtain a dimensionality reduction and noise-suppressed tokens matrix, wherein the feature representation refers to a feature vector that has both local and global structural information, and the convolutional block is multiple convolutional kernels with residual connections; and outputting the classification result.

[0008] Optionally, each layer of the Transformer encoder based on the Vision Transformer network captures the feature representations in the image to be classified, including: capturing global features of different regions in the image to be classified using the multi-head self-attention module of each encoder; capturing local features of local regions in the image to be classified using each convolutional block; and weightedly fusing the local features and the global features to obtain the feature representations output by each layer of each encoder.

[0009] Optionally, the step of performing a spectral feature dimensionality reduction pooling operation every two layers on each of the feature representations extracted from the last four layers of each encoder to obtain a dimensionality-reduced and denoised tokens matrix includes: performing spectral clustering and singular value decomposition on each feature representation to obtain dimensionality-reduced features; and clustering the dimensionality-reduced features based on a clustering algorithm to obtain a dimensionality-reduced and denoised tokens matrix.

[0010] Optionally, the tokens representing each feature are converted into a graph structure, and an adjacency matrix is ​​obtained based on the graph structure; a normalized Laplacian matrix is ​​calculated based on the adjacency matrix; singular value decomposition is performed on the normalized Laplacian matrix, and multiple eigenvalues ​​are selected to select the corresponding principal feature components to obtain feature vectors belonging to the low-dimensional space.

[0011] Optionally, the step of clustering the dimensionality-reduced features based on a clustering algorithm to obtain a dimensionality-reduced and noise-suppressed tokens matrix includes: using the K-Means clustering algorithm to perform semantic similarity aggregation on the feature vectors belonging to the low-dimensional space to obtain a clustered tokens matrix; and performing an averaging operation on the clustered tokens matrix to obtain a new dimensionality-reduced and noise-suppressed tokens matrix.

[0012] Optionally, the method further includes: during the training phase, initializing a prototype vector for each category in the support set, wherein each prototype vector corresponds to the center position of its category in the feature space; inputting a query sample into the multi-level feature extraction network to obtain a high-dimensional embedding vector of the query sample; calculating the similarity between the high-dimensional embedding vector and each prototype vector using the multi-level feature extraction network, and updating the prototype vector based on the high-dimensional embedding vector with the highest similarity to obtain the updated prototype vector; and training the multi-level feature extraction network using the updated prototype vector to obtain a parameter-updated feature extraction network.

[0013] Optionally, before calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, the method further includes: performing an exponentially weighted smoothing process on the high-dimensional embedding vector.

[0014] Optionally, training the multi-level feature extraction network includes a meta-training phase and a fine-tuning phase. Before calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, the method further includes: in the meta-training phase, training the multi-level feature extraction network as a whole so that the multi-level feature extraction network jointly learns the features required for the task to be processed; in the fine-tuning phase, freezing the main parameters of the multi-level feature extraction network and training only the convolutional blocks embedded in each of the encoders.

[0015] Optionally, the similarity calculation formula includes one of the following: cosine similarity calculation formula, Euclidean distance calculation formula, and Mahalanobis distance calculation formula.

[0016] Furthermore, to achieve the above objectives, this application also provides a few-sample image classification method based on multi-level feature extraction and prototype update, comprising: an acquisition module for acquiring an image to be classified; a processing module for processing the image to be classified based on a multi-level feature extraction network to obtain a classification result of the image to be classified, wherein each Transformer encoder of the multi-level feature extraction network has a convolutional block; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures a feature representation in the image to be classified, and performs a spectral feature dimensionality reduction pooling operation every two layers on each feature representation extracted from the last four layers of each encoder to obtain a dimensionality reduction and noise-suppressed tokens matrix, wherein the feature representation refers to a feature vector that has both local and global structural information, and the convolutional block is multiple convolutional kernels with residual connections; and an output module for outputting the classification result.

[0017] This application proposes a few-sample image classification method and apparatus based on multi-level feature extraction and prototype update. The method involves acquiring an image to be classified; processing the image using a multi-level feature extraction network to obtain a classification result; wherein each Transformer encoder in the multi-level feature extraction network corresponds to a convolutional block embedded in the neural network; and processing the image using the multi-level feature extraction network includes: each layer of each Transformer encoder in the Vision Transformer network captures a feature representation in the image to be classified, and performing spectral feature dimensionality reduction pooling on the feature representations extracted from the last four layers of each encoder every two layers to obtain dimensionality-reduced and noise-suppressed feature representations. Here, a feature representation refers to feature information that combines local and global structural information. The method outputs a classification result, thereby improving the classification performance and robustness of the model under conditions of limited data. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an embodiment of the few-sample image classification method based on multi-level feature extraction and prototype updating in this application.

[0019] Figure 2 This is a schematic diagram of an embodiment of a few-sample image classification method based on multi-level feature extraction and prototype update in this application;

[0020] Figure 3 This is a structural block diagram of an embodiment of a small sample image classification device based on multi-level feature extraction and prototype updating in this application.

[0021] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0023] In existing technologies, the training and generalization capabilities of models are often severely affected by the limited number of samples in small-sample classification tasks. Especially under traditional convolutional neural networks and ViT frameworks, the extracted features may have problems such as inter-class overlap and uneven distribution, thereby reducing the accuracy and stability of classification.

[0024] Current research attempts to alleviate these problems by enhancing feature representation capabilities and optimizing classification modules. For example, CNNs are widely used due to their powerful local feature extraction capabilities, but their global modeling capabilities are insufficient; ViT can capture global features, but it lacks in capturing local information details. In addition, prototype learning methods, as a meta-learning-based strategy, can effectively reduce the distribution overlap between categories by aggregating sample features into category prototypes, but the dynamic update mechanism of prototypes still needs improvement in complex scenarios.

[0025] In some existing technologies, convolutional modules are added after each Transformer layer to enhance the ability to extract local features. This method addresses the shortcomings of pure Transformers in processing local details to some extent, but its feature fusion method is relatively simple and lacks multi-scale and multi-level feature representation capabilities.

[0026] In other existing technologies, a query-support Transformer framework has been proposed for few-shot learning tasks. This framework achieves efficient fusion and metric learning of global and local features of the support and query sets by introducing a cross-scale interactive feature extractor, sampleFormer, and patchFormer. However, its complex modular design leads to high computational complexity, especially when processing high-resolution images and a large number of block-level features, significantly increasing the demand for computational resources. Furthermore, this technique lacks effective feature dimensionality reduction and redundancy control mechanisms, making it difficult to operate efficiently in resource-constrained environments.

[0027] In other existing technologies, query prototypes are generated using query samples and fused with support set prototypes, effectively improving the dynamic adaptability of category prototypes. However, this update strategy requires comprehensive interaction between all query samples and category prototypes, resulting in high computational complexity. Furthermore, it lacks robustness to noisy samples or outliers, potentially leading to prototype skew. Additionally, the lack of explicit optimization of embedded features before prototype operations means that redundancy and noise in high-dimensional features may further impact the accuracy and efficiency of the update.

[0028] Existing techniques for few-shot classification tasks suffer from the following main drawbacks: First, existing models have limited feature representation capabilities. Models like the Vision Transformer (ViT) network, while possessing powerful global feature modeling capabilities, are relatively weak in capturing small-sized targets or microscopic textures due to the lack of convolutional operations. Conversely, classic convolutional neural networks (CNNs, such as the ResNet series) are more sensitive to local details but slightly lacking in large-scale context modeling. Relying solely on one structure makes it difficult for the model to simultaneously obtain rich global context and fine local texture representation under few-shot conditions, resulting in limited classification performance. Second, in few-shot environments, the lack of massive samples to compensate for or learn more robust features makes the negative impact of noise interference on model discrimination very prominent. This leads many feature extraction networks to often fail to consider the impact of background noise on feature representation during feature extraction, resulting in low classification accuracy. Prototype-based learning methods perform well in few-shot classification, but for complex categories or multimodal situations, this fixed prototype is difficult to adapt to morphological changes or noise impacts that occur during the query phase. Once the distribution of categories shifts in a real-world scenario, the accuracy of model recognition will decrease significantly.

[0029] To address the aforementioned shortcomings, this application presents a few-sample image classification method based on multi-level feature extraction and prototype update. The method aims to: propose a feature extraction module that combines the advantages of ViT (Vision-Induced Transformation) and convolution. It acquires global context through multi-head self-attention while introducing convolutional blocks to capture local details, and then fuses the two to balance global correlation and local high-resolution texture, thereby improving classification accuracy in few-sample scenarios. For complex environments or task scenarios, it enhances the model's adaptability to uncertain conditions through spectral clustering and pooling. Utilizing spectral clustering, it effectively compresses and aggregates high-dimensional redundancy and noise, helping to maintain a relatively compact and highly discriminative feature space even with few samples. A prototype update mechanism based on distance-weighted interpolation during the inference stage is designed to replace the fixed prototype approach of traditional prototype networks that only use simple means. This dynamic update allows the category prototype to absorb information from query samples, reducing overlap between categories and thus improving classification robustness.

[0030] In few-sample image classification, given a dataset D = (x i ,y i ), x i ∈R H×W×3 It is the i-th image, y i ∈Y is its class label. D is divided into the base dataset D. b and the new dataset D n Unlike traditional image recognition tasks, few-shot learning focuses on classifying new categories after the training phase. Specifically, the label set Y is divided into the base label set Y0 and the base label set Y0. b and new tag set Y n ,satisfy In D n Create few-sample tasks, each task containing N classes, with only K labeled samples for each new class. Each task is supported by a support set. and query set The model is composed of a support set s, where M is the number of test samples. The model is trained using a classifier built on the support set s and evaluated on the query set q.

[0031] Reference Figure 1 The few-shot image classification method based on multi-level feature extraction and prototype update provided in the first embodiment of this application can be executed by a processor of a terminal or server. The few-shot image classification method based on multi-level feature extraction and prototype update may include:

[0032] S10. Obtain the image to be classified;

[0033] Prior to this step, preprocessing of the images to be classified is included to adapt them for subsequent training and feature extraction. The data preprocessing stage of this application primarily involves data augmentation. Due to the limited amount of data faced by few-shot learning, the data augmentation techniques in the data preprocessing stage mainly involve operations such as rotation, flipping, cropping, and scaling of the original images. These data augmentation techniques mainly include random rotation (±15 degrees), horizontal and vertical flipping, random cropping (cropping ratio of 80%), and scaling (scaling factor of 0.8). This not only expands the scale of the dataset but also enhances the model's generalization ability, enabling it to better cope with the varied situations in real-world applications.

[0034] S20. Process the image to be classified based on a multi-level feature extraction network to obtain the classification result of the image to be classified, wherein each encoder of the multi-level feature extraction network has a convolutional block; the processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, and performs spectral feature dimensionality reduction pooling operation once every two layers on each feature representation extracted from the last four layers of each Transformer encoder to obtain a tokens matrix after dimensionality reduction and noise reduction, wherein the feature representation refers to a feature vector that has both local information and global structural information, and the convolutional block is multiple convolutional kernels with residual connections;

[0035] In this step, global context is acquired through multi-head self-attention, while convolutional blocks are introduced to capture local details, and the two are fused to balance global correlation and local high-resolution texture, thereby improving classification accuracy in small-sample scenarios. This application also enhances the adaptability of the multi-level feature extraction network to uncertain conditions through spectral clustering pooling for complex environments or task scenarios. Utilizing the idea of ​​spectral clustering, high-dimensional redundancy and noise are effectively compressed and aggregated, helping to maintain a relatively compact and highly discriminative feature space even with small samples. It should be noted that the multi-level feature extraction network is built based on an improved ViT network.

[0036] S30. Output the classification results.

[0037] In summary, the few-sample image classification method based on multi-level feature extraction and prototype update proposed in this application processes the image to be classified using a multi-level feature extraction network to obtain the classification result. Each Transformer encoder in the multi-level feature extraction network corresponds to a convolutional block embedded in the neural network. Processing the image to be classified using the multi-level feature extraction network includes: each layer of each Transformer encoder in the Vision Transformer network captures the feature representations in the image to be classified; and for each feature representation extracted from the last four layers of each encoder, a spectral feature dimensionality reduction pooling operation is performed every two layers to obtain dimensionality-reduced and noise-suppressed feature representations. Here, feature representation refers to feature information that combines local and global structural information. The output classification result combines a global attention mechanism with local convolutional paths, which can capture the long-range dependencies between different regions in the image while preserving detailed information such as edges and textures, thus more effectively mining image information in few-sample scenarios. Furthermore, spectral feature pooling dimensionality reduction further compresses the feature space dimension, reduces noise interference, and lowers the computational cost during classification. Compared to traditional methods that rely solely on global feature extraction, this application offers more flexible and hierarchical feature representation.

[0038] In the embodiments of this application, each layer of the Transformer encoder based on the Vision Transformer network captures the feature representations in the image to be classified, including:

[0039] S201. Use the multi-head self-attention module of each encoder to capture global features of different regions in the image to be classified;

[0040] In this step, the processor selects the Vision Transformer (ViT) as the base network to learn global attention for the image. ViT segments the image into several patches and treats them as sequence inputs to the Transformer encoder to capture the long-range dependencies between the patches. Since small sample data often contains key information distributed in local regions, ViT can effectively integrate the global context scattered across different patches to obtain global features.

[0041] S202. Use each of the convolutional blocks to capture local features of local regions in the image to be classified;

[0042] In this step, unlike pure ViT, a convolutional block is added after each Transformer encoder layer, embedding a local feature enhancement path to capture the input features F. l-1Local details This convolutional block consists of multiple small convolutional kernels: 1×1 convolution, 3×3 convolution, and 1×1 convolution. The 1×1 convolution is used to reduce the channel dimension of the input, thus reducing computational cost. The 3×3 convolution is the core part of the convolutional block, used to capture visual information of local regions. The subsequent 1×1 convolution is used to restore the channel dimension, returning the extracted features to their original channel size. High-resolution features, such as object edges and textures, can be extracted from local spatial regions.

[0043] S203. The local features and the global features are weighted and fused to obtain the feature representation of each layer output of each encoder.

[0044] Then local features Global features of ViT Perform weighted fusion α is a learnable parameter used to adjust the ratio between local and global features, enabling multi-scale and multi-level feature representation. This preserves ViT's ability to integrate global information while also taking into account the fine structure of local features.

[0045] By executing steps S201-S203, the processor leverages a joint architecture of convolutional neural networks and VisionTransformer networks to fully integrate local and global features, extracting richer and more discriminative feature representations. In obtaining these feature representations, firstly, the processor uses the VisionTransformer network to extract global features from the input image. Through the multi-head self-attention mechanism of the VisionTransformer network, it captures long-distance dependencies between different regions in the image, thereby generating a feature representation rich in global context. Secondly, the processor embeds convolutional blocks in each layer of the encoder to capture detailed information and local features in the image. Finally, the processor weightedly fuses the global and local features to form rich and diverse feature representations. This joint architecture not only captures detailed textures in the image but also effectively understands the global structural information of the image, thus significantly improving classification performance even with small sample sizes.

[0046] In the embodiments of this application, the step of performing a spectral feature dimensionality reduction pooling operation every two layers on the feature representations extracted from the last four layers of each encoder to obtain a dimensionality-reduced and noise-suppressed tokens matrix includes:

[0047] S204. Perform graph clustering and singular value decomposition on each of the aforementioned feature representations to obtain dimensionality-reduced features;

[0048] In this step, the tokens matrix T∈R is input after feature extraction. N×DWith adjacency weight matrix S0∈R N×N Define the Laplacian matrix L = D - S0, where D is a diagonal matrix. Calculate the normalized Laplace: To enhance numerical stability;

[0049] S205. Cluster the reduced features based on the clustering algorithm to obtain the reduced and denoised tokens matrix;

[0050] In this step, singular value decomposition is used to normalize the Laplace matrix L. norm Dimensionality reduction is performed to extract the main feature components. The top N0 singular vectors are selected. The basis vectors of the subspace are formed, reflecting the low-dimensional embedding structure of the graph. The first N0 eigenvectors are stacked to obtain a matrix. Each row of matrix U, v i That is, the representation of tokens in low-dimensional space. The representation of each token in low-dimensional space is: All v i Aggregate them, and use K-means to cluster them into N0 clusters. Each cluster corresponds to a new token that will be merged later. Average the features of the tokens within the same cluster to obtain the new token: Where C k This represents the index of all original tokens in the k-th cluster. Finally, N0 new tokens are obtained, forming a new tokens matrix.

[0051] It should be noted that in this application, features are represented as feature tokens.

[0052] In the above steps, since high-dimensional features often contain a large amount of redundant information and noise, the processor performs a spectral feature dimensionality reduction pooling operation on the extracted feature tokens every two layers in the last four layers of each encoder. This reduces redundancy among high-dimensional features and effectively suppresses noise. The core idea of ​​the spectral feature dimensionality reduction pooling operation is to use graph clustering combined with singular value decomposition to reduce the dimensionality of high-dimensional features, thereby optimizing feature representation, reducing redundant features, and improving computational efficiency and classification performance.

[0053] Specifically, step S204 may include the following execution process:

[0054] S2041. Convert each of the aforementioned feature representations into a graph structure based on the adjacency matrix and the Laplacian matrix;

[0055] S2042. Perform singular value decomposition on the feature matrix in the graph structure to obtain multiple main feature components, and construct dimensionality reduction features based on the multiple main feature components.

[0056] The processor performs dimensionality reduction of the feature tokens by executing steps S2041-S2042. The above process may include the following steps:

[0057] First, the processor transforms the fused high-dimensional feature representation into a graph structure. This transformation is achieved by constructing an adjacency matrix and a Laplacian matrix. The adjacency matrix describes the connections between features, while the Laplacian matrix further reveals the graph's topology and feature distribution.

[0058] After the graph structure is established, the processor performs spectral dimensionality reduction. In this stage, the processor decomposes the feature matrix of the graph structure using singular value decomposition (SVD), extracts the main feature components, and maps the high-dimensional features to a low-dimensional space. SVD yields a set of the most discriminative feature vectors, thus preserving the main structural information of the data while effectively filtering noise and redundant information. Compared to directly using feature dimensionality reduction, SVD further improves the stability of dimensionality reduction, reduces redundancy, and optimizes the computational efficiency of multi-level feature extraction networks in subsequent tasks. This dimensionality reduction method reduces the dimensionality of the feature space while maintaining global semantic information, thereby reducing the complexity of subsequent calculations and making the feature representation more compact and discriminative.

[0059] Specifically, step S205 may include the following execution process:

[0060] The K-Means clustering algorithm is used to perform semantic similarity aggregation on the feature vectors belonging to the low-dimensional space to obtain the clustering tokens matrix;

[0061] The clustered tokens matrix is ​​averaged to obtain a new tokens matrix after dimensionality reduction and noise suppression.

[0062] This step further aggregates the dimensionality-reduced features using KMeans clustering, grouping semantically similar feature tokens into the same category to form more representative feature clusters. Through this clustering operation, the multi-level feature extraction network can more effectively integrate similar information, reduce input data redundancy, and improve the ability to distinguish between different categories in the final classification process, thereby enhancing classification accuracy.

[0063] Based on this, the few-sample image classification method based on multi-level feature extraction and prototype update provided in this application also includes:

[0064] S401. During the training phase, a prototype vector is initialized for each category in the support set, wherein each prototype vector corresponds to the center position of its category in the feature space.

[0065] S402. Input the query sample into the multi-level feature extraction network to obtain the high-dimensional embedding vector of the query sample;

[0066] This application first analyzes the feature vectors {f1, f2, ..., f...} of all support samples within the same category. m Perform simple mean or weighted mean calculations to obtain the initial prototype vector of category c.

[0067]

[0068] Where m is the number of samples of category c in the support set, and f i This represents the feature vector obtained after the i-th sample passes through the feature extraction network.

[0069] S403. Calculate the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, and fine-tune the prototype vector based on the high-dimensional embedding vector with the highest similarity to obtain the fine-tuned prototype vector.

[0070] Calculate the distance between the query sample and the prototype. When a new query sample x is input, first obtain the embedding vector f through the aforementioned feature extraction process. x Each of the current category prototypes Calculate the distance and select the prototype of the category with the highest similarity for subsequent updates;

[0071] (3.3) The prototype vector is updated using the basic interpolation formula to obtain the fine-tuned prototype vector:

[0072]

[0073] Where α∈(0,1) is the update rate, balancing the old prototype with the new information.

[0074] This represents a distance-based weighting factor; the smaller the distance, the greater the weight and the stronger the update. When the distance threshold is too large, it is considered an outlier and skipped from the update, preventing the prototype from being skewed and protecting it from interference from extreme samples.

[0075] S404. The multi-level feature extraction network is trained using the fine-tuned prototype vector to obtain the fine-tuned Vision Transformer network.

[0076] By executing steps S401-S404, the processor continuously modifies the category prototype during the query phase through distance-based weighted interpolation, greatly enhancing the adaptability of the multi-level feature extraction network to new sample distributions. Traditional few-sample classification methods mostly fix the category prototype, making it difficult to adapt to morphological changes or distribution shifts in query samples. The prototype update mechanism proposed in this application, however, allows the category representation of the multi-level feature extraction network to better fit the actual environment while ensuring controllable computational overhead.

[0077] For example, the process of a secondary update to a category prototype may include the following execution process:

[0078] During the training phase, the processor initializes a prototype vector for each category in the support set. Each prototype vector represents the center position of that category in the feature space. Next, the processor inputs the query samples into a multi-layered feature extraction network, which obtains their high-dimensional embedding vectors through the feature extraction module. This embedding vector contains global and local feature information of the query samples and is crucial for subsequent matching and classification.

[0079] Next, the processor uses Euclidean distance to calculate the similarity between the query sample's embedding vector and all category prototype vectors. This process determines which category the query sample is most likely to belong to. Based on the initial matching score, the query sample is assigned to the category with the highest similarity. Subsequently, the processor updates the prototype vector of that category to better reflect the information of the new query sample. This update process fine-tunes the prototype vector by weightedly fusing the query sample's embedding vector with the current category prototype vector. In this way, the category prototype can gradually adapt to the feature changes of new samples, enhancing its ability to capture diversity within categories. Simultaneously, to prevent drastic changes in the category prototype due to the influence of individual query samples, a smooth update strategy is adopted for the secondary update of the category prototype. This strategy, by introducing an exponentially weighted average method, ensures a smooth update process for the prototype vector, avoiding shifts caused by noise or outliers, thereby improving the overall stability of the multi-level feature extraction network.

[0080] Using cosine similarity as the similarity metric, the feature vector of the query sample is compared one by one with the prototype vector of each category, and the corresponding similarity score is calculated. The system makes a classification decision based on the calculated similarity score. The query sample is sorted according to the cosine similarity scores of all category prototypes, and the category with the highest similarity is identified. The category with the highest similarity score is selected as the predicted category of the query sample. This selection process ensures high accuracy of the classification results because the category with the highest score represents the category most similar to the features of the query sample. Finally, the system outputs the predicted category to which the query sample belongs, completing a complete classification process.

[0081] It is worth noting that using cosine similarity to calculate the matching between known category classifications and classification prototypes can also be replaced by other distance metrics such as Euclidean distance and Mahalanobis distance.

[0082] In another alternative embodiment of this application, distance-based weighted interpolation continuously corrects the category prototype during the query phase. Alternatively, soft labels can be used to assign query sample features to each category, and then the classification prototype is updated in a balanced manner.

[0083] In another alternative embodiment of this application, the weight value of the category prototype during the update process can be changed from a fixed weight to an adaptive weight in order to obtain a more efficient category prototype update ratio.

[0084] Training the multi-level feature extraction network includes a meta-training phase and a fine-tuning phase. Before step S403, the method further includes:

[0085] During the meta-training phase, the multi-level feature extraction network is trained as a whole so that the multi-level feature extraction network can jointly learn the features required for the task to be processed.

[0086] Since the supervised information in small sample data is limited, fine-tuning the entire network in the subsequent fine-tuning stage is highly likely to lead to overfitting due to an excessive number of learnable parameters, and will also significantly increase training overhead. Therefore, this application trains the model as a whole in the meta-training stage, enabling the entire model to learn the features required for the task. In the fine-tuning stage, a strategy of freezing the ViT main parameters and training only the convolutional blocks embedded in each encoder is adopted. On the one hand, the overall training in the meta-training stage ensures that the model can make necessary adjustments for small sample tasks; on the other hand, updating only a small number of parameters in the newly added modules in the fine-tuning stage can reduce the risk of overfitting, reduce computational cost, and allow the model to more stably improve classification performance on limited sample data.

[0087] This application freezes some parameters and fine-tunes only the local convolution path and classification head. This not only makes full use of the general features of large-scale pre-training, but also significantly reduces the risk of overfitting under small samples. Compared with full fine-tuning, this strategy can reduce training time and computing power while ensuring accuracy.

[0088] Cosine similarity is used to calculate the match between a query sample and its class prototype. Specifically, cosine similarity is used as a metric to measure the directional consistency between the features of the query sample and the features of the class prototype. The formula for calculating cosine similarity is:

[0089]

[0090] Where: f x This represents the feature vector of the query sample. This represents the prototype vector of category c. Let f represent vectors respectively. x and The Euclidean norm;

[0091] This formula allows us to calculate the similarity score between the query sample and each category prototype.

[0092] The optimal category is selected as the classification result based on the score. After calculating the cosine similarity between the query sample and the prototypes of all categories, the category with the highest matching degree is selected as the classification result based on the similarity score.

[0093] The formula for classification decision is:

[0094]

[0095] in: Let S represent the final classification result, and C represent the set of all categories. Sim(q,c) represents the query sample f. x With category prototype The cosine similarity score;

[0096] Using the common cross-entropy loss function as the classification loss, we first map the similarity to logits and then calculate the cross-entropy loss:

[0097]

[0098] During the training or fine-tuning phase, backpropagation is performed on the unfrozen model parameters, while the frozen layer parameters remain unchanged.

[0099] The technical effects of the present invention will be further described below with reference to experiments:

[0100] This invention is a simulation performed using PyCharm software and the Python language on a CPU consisting of an Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz, 125GB of memory, and an Ubuntu 20.04.2 operating system.

[0101] Simulation Experiment 1: The present invention and existing small sample image classification models were used to classify and compare general small sample benchmark tests. The results are shown in Table 1. In this experiment, the general small sample benchmark tests used two datasets: mini-ImageNet and tiered-ImageNet.

[0102] The mini-ImageNet dataset is a subset of the ImageNet dataset, containing 100 categories, with 600 images per category, all at a uniform size of 84x84 pixels. In this experiment, these 100 categories are further divided: 64 categories are used for training, 16 for validation, and 20 for testing.

[0103] Tiered-ImageNet is also a larger subset of ImageNet, containing 34 superclasses and a total of 608 classes, with varying numbers of images per class. To ensure significant domain differences between training and evaluation, 351 classes were used for training, 97 for validation, and 160 for testing. Due to its large number and wide distribution of classes, tiered-ImageNet is suitable for testing model performance in more challenging multi-class few-shot learning scenarios, and most research methods use 84×84 pixel images.

[0104] Table 1: Classification Performance of mini-ImageNet and tiered-ImageNet

[0105]

[0106] Based on the experimental results in the table, our model demonstrates superior performance on common small-sample benchmarks (mini-ImageNet and tiered-ImageNet), outperforming other models in both 1-shot and 5-shot settings. Specifically:

[0107] On the mini-ImageNet dataset: In 1-shot classification, our model achieved an accuracy of 69.45%, surpassing all comparison models and demonstrating its excellent classification ability with very few samples. In 5-shot classification, the model's accuracy improved to 86.34%, maintaining its leading position and proving its good generalization ability. On the tiered-ImageNet dataset: In 1-shot classification, the model led with an accuracy of 74.09%, showing its strong adaptability when handling more complex datasets. In 5-shot classification, the accuracy further improved to 88.03%, continuing to outperform all comparison models and demonstrating the advantage of our model in deeper data analysis.

[0108] These experimental results not only demonstrate the efficiency of our model in handling image classification tasks of various sizes and types, but also showcase its practicality and application potential in the field of few-shot learning.

[0109] Simulation Experiment 2: The present invention and existing small sample image classification models were used to classify and compare fine-grained small sample benchmark tests. The results are shown in Table 2. In this experiment, the cropped CUB dataset and Aircraft dataset were used in the fine-grained small sample benchmark tests.

[0110] The CUB dataset consists of approximately 11,788 images distributed across 200 bird categories, with an average of about 60 images per category. Each image in the CUB dataset is accompanied by rich annotation information, covering bounding boxes, part localization, and attribute labels, providing abundant visual and semantic information for fine-grained image analysis. The CUB dataset is randomly divided into 100 training classes, 50 validation classes, and 50 evaluation classes. We used a dataset where each image was cropped based on manually annotated bounding boxes.

[0111] The Aircraft dataset consists of approximately 10,000 images, covering 100 different aircraft models, with approximately 100 images provided for each model. Consistent with the experimental setup of the CUB dataset, the Aircraft dataset was processed using the same class partitioning ratio, randomly assigning the 100 aircraft models to 50 training classes, 25 validation classes, and 25 test classes. Furthermore, similar to the processing method used for the CUB dataset, the images in the Aircraft dataset were cropped to ensure that the input images were uniformly focused on the aircraft models within the bounding boxes provided by the dataset.

[0112] Table 2: Classification Performance of CUB-Cropped and Aircraft

[0113]

[0114] Based on the experimental results in the table, our model performs exceptionally well on two fine-grained few-shot benchmarks (CUB-Cropped and Aircraft), demonstrating superior performance in both 1-shot and 5-shot test settings. These results significantly outperform traditional models such as MatchNet and ProtoNet, and even surpass several current state-of-the-art models, including DeepEMD, DeepBDC, and FRN. This performance highlights the significant advantages of our model in handling fine-grained few-shot classification problems and its enormous potential for practical applications.

[0115] refer to Figure 3 Based on the above method embodiments, this application also provides a few-sample image classification device based on multi-level feature extraction and prototype update. The few-sample image classification device based on multi-level feature extraction and prototype update may include an acquisition module 101, a processing module 102, and an output module 103, wherein...

[0116] The acquisition module 101 is used to acquire the image to be classified;

[0117] The processing module 102 is used to process the image to be classified based on a multi-level feature extraction network to obtain the classification result of the image to be classified. Each encoder of the multi-level feature extraction network has a convolutional block. The processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures each feature representation in the image to be classified, and for each feature representation extracted from the last four layers of each encoder, a spectral feature dimensionality reduction pooling operation is performed every two layers to obtain a dimensionality-reduced and noise-suppressed tokens matrix. The feature representation refers to a feature vector that contains both local and global structural information.

[0118] The output module 103 is used to output the classification result.

[0119] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A few-sample image classification method based on multi-level feature extraction and prototype update, characterized in that, include: Obtain the image to be classified; The image to be classified is processed based on a multi-level feature extraction network to obtain the classification result of the image to be classified. Each Transformer encoder of the multi-level feature extraction network corresponds to a convolutional block embedded in the neural network. The processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the Vision Transformer network captures a feature representation in the image to be classified, and for each feature representation extracted from the last four layers of each encoder, a spectral feature dimensionality reduction pooling operation is performed every two layers to obtain a dimensionality-reduced and noise-suppressed tokens matrix. The feature representation refers to a feature vector that contains both local and global structural information, and the convolutional block is multiple convolutional kernels with residual connections. Output the classification results.

2. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 1, characterized in that, Each layer of the Transformer encoder based on the Vision Transformer network captures the feature representations in the image to be classified, including: The multi-head self-attention modules of each encoder are used to capture global features of different regions in the image to be classified. Each of the convolutional blocks is used to capture local features of local regions in the image to be classified. The local features and the global features are weighted and fused to obtain the feature representations of each layer output by each encoder.

3. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 1, characterized in that, The feature representations extracted from the last four layers of each encoder are subjected to spectral feature dimensionality reduction pooling operation every two layers to obtain a dimensionality-reduced and noise-suppressed tokens matrix, including: Graph clustering and singular value decomposition are performed on each of the aforementioned feature representations to obtain dimensionality-reduced features; The dimensionality-reduced features are clustered using a clustering algorithm to obtain a dimensionality-reduced and noise-suppressed tokens matrix.

4. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 3, characterized in that, The step of performing graph clustering and singular value decomposition on each of the feature representations to obtain dimensionality-reduced features includes: The tokens representing each feature are converted into a graph structure, and an adjacency matrix is ​​obtained based on the graph structure; Calculate the normalized Laplace matrix based on the adjacency matrix; Singular value decomposition is performed on the normalized Laplacian matrix, and multiple eigenvalues ​​are selected to identify the corresponding principal eigencomponents, resulting in eigenvectors belonging to the low-dimensional space.

5. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 3, characterized in that, The clustering of the dimensionality-reduced features using a clustering algorithm to obtain a dimensionality-reduced and noise-suppressed tokens matrix includes: The K-Means clustering algorithm is used to perform semantic similarity aggregation on the feature vectors belonging to the low-dimensional space to obtain the clustering tokens matrix; The clustered tokens matrix is ​​averaged to obtain a new tokens matrix after dimensionality reduction and noise suppression.

6. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 1, characterized in that, The method further includes: During the training phase, a prototype vector is initialized for each category in the support set, wherein each prototype vector corresponds to the center position of its category in the feature space. The query sample is input into the multi-level feature extraction network to obtain the high-dimensional embedding vector of the query sample; The similarity between the high-dimensional embedding vector and each prototype vector is calculated using the multi-level feature extraction network, and the prototype vector is updated based on the high-dimensional embedding vector with the highest similarity to obtain the updated prototype vector. The multi-level feature extraction network is trained using the updated prototype vector to obtain a fine-tuned Vision Transformer network.

7. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 6, characterized in that, Before calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, the method further includes: The high-dimensional embedding vector is then smoothed using an exponentially weighted method.

8. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 6, characterized in that, Training the multi-level feature extraction network includes a meta-training phase and a fine-tuning phase. Before calculating the similarity between the high-dimensional embedding vector and each of the prototype vectors using the multi-level feature extraction network, the method further includes: During the meta-training phase, the multi-level feature extraction network is trained as a whole so that the multi-level feature extraction network can jointly learn the features required for the task to be processed. During the fine-tuning phase, the main parameters of the multi-level feature extraction network are frozen, and only the convolutional blocks embedded in each encoder are trained.

9. The few-sample image classification method based on multi-level feature extraction and prototype update as described in claim 6, characterized in that, The formula for calculating the similarity includes: One of the formulas for calculating cosine similarity, Euclidean distance, and Mahalanobis distance.

10. A few-sample image classification device based on multi-level feature extraction and prototype update, characterized in that, include: The acquisition module is used to acquire the image to be classified. The processing module is used to process the image to be classified based on a multi-level feature extraction network to obtain the classification result of the image to be classified. Each Transformer encoder of the multi-level feature extraction network corresponds to a convolutional block embedded in the neural network. The processing of the image to be classified based on the multi-level feature extraction network includes: each layer of each Transformer encoder based on the VisionTransformer network captures a feature representation in the image to be classified, and for each feature representation extracted from the last four layers of each encoder, a spectral feature dimensionality reduction pooling operation is performed every two layers to obtain a dimensionality-reduced and noise-suppressed tokens matrix. The feature representation refers to a feature vector that contains both local and global structural information, and the convolutional block is multiple convolutional kernels with residual connections. The output module is used to output the classification results.

Images