Small sample image classification method based on parallel expert structure
By using a parallel expert structure and a hybrid expert LSTM module for dynamic feature fusion, the problem of insufficient context adaptability in small sample image classification is solved, achieving higher feature recognition power and classification accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QUANZHOU INST OF EQUIP MFG
- Filing Date
- 2026-05-09
- Publication Date
- 2026-06-05
AI Technical Summary
Existing few-sample image classification methods lack context adaptability and cannot dynamically adjust feature representations, resulting in the loss of effective information. Furthermore, they lack discriminative power in fine-grained tasks and cannot accurately identify key information regions.
A few-sample image classification method based on a parallel expert structure is adopted, which includes a pre-trained Vision Transformer backbone network in the iBOT framework, combined with a hybrid expert LSTM module and cosine similarity attention. The method calculates the difference distance between feature distributions through dynamic routing weights and feature fusion for classification.
It improves the cohesion and feature recognition of the same category, enhances the model's ability to locate and match key information, and improves the accuracy and robustness of small sample image classification.
Smart Images
Figure CN122156833A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of few-shot image classification, and more specifically to a few-shot image classification method based on a parallel expert structure. Background Technology
[0002] Current mainstream few-sample image classification methods use static feature extraction, which lacks context adaptability. This static representation makes it impossible for the model to dynamically adjust and optimize its feature representation based on contextual cues provided by the support set in the current task, resulting in the loss of effective information.
[0003] Metric learning-based few-sample image classification methods rely on pre-trained models that tend to produce general feature representations. While these features help distinguish between highly different categories, they often lack sufficient discriminative power in fine-grained tasks that require discerning subtle differences.
[0004] Even with advanced architectures like Vision Transformer (ViT), existing models still face challenges in effectively modeling sequence dependencies and local context among image feature tokens processed by ViT. These models tend to treat all image regions equally, failing to accurately identify and focus on the most critical information regions for the classification task, resulting in the final feature representation being easily contaminated by irrelevant background noise. Summary of the Invention
[0005] The purpose of this invention is to provide a few-sample image classification method based on a parallel expert structure that improves the cohesion of enhanced categories.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: The few-sample image classification method based on parallel expert architecture includes the following steps performed sequentially: S1: Obtain a small sample dataset, label the small sample dataset, and divide the labeled small sample dataset into a training set, a validation set, and a test set; S2: Based on the classification task, construct the support set and query set; S3: Use the iBOT framework to perform unsupervised pre-training on the Vision Transformer backbone network to obtain the pre-trained ViT backbone network; S4: Input all image samples from the support set and the query set into the pre-trained ViT backbone network for feature extraction to obtain category feature tokens and image feature tokens. Then, fuse the category feature tokens and the image feature tokens to obtain the fused feature tokens of the support set and the query set, respectively. S5: Input the feature tokens obtained by fusing the support set and the query set into the parallel expert Mamba module. The parallel expert Mamba module includes two parallel branches. One branch is a hybrid expert LSTM module. The hybrid expert LSTM module includes two branches. One branch uses a gated network to dynamically calculate the routing weight for each token in the fused feature tokens. The other branch uses an LSTM expert group to extract features from the fused feature tokens and uses the routing weight to perform weighted fusion of the extracted features to obtain the first extracted feature. Another branch is the Mamba state space module, which extracts features from the input fused feature tokens to obtain the second extracted features; The first extracted feature and the second extracted feature are concatenated along the feature dimension to obtain a joint feature. A fusion gating network is used to linearly transform the joint feature, and an adaptive fusion gating coefficient is calculated by combining the activation function. The first extracted feature and the second extracted feature are weighted and fused using the adaptive fusion gating coefficient to obtain a fused feature. The fused feature is then subjected to linear projection, residual connection and layer normalization in sequence to obtain the enhanced features of the support set and the query set, respectively. S6: Divide the enhanced features of the support set into global features of the support set and local features of the support set, and divide the enhanced features of the query set into global features of the query set and local features of the query set. S7: Input the global features of the support set and the local features of the query set into the cosine similarity attention calculation to obtain the attention weights. The local features of the support set and the global features of the query set are input into the cosine similarity attention calculation to obtain the attention weights. ; S8: The enhanced features of the support set, the enhanced features of the query set, and the attention weights. And the attention weight Input the mixed kernel function, calculate the difference distance between feature distributions, and take the category with the smallest difference distance as the predicted category. Preferably, the construction of the support set and query set in step S2 includes the following steps: S2-1: Randomly select N categories from a predefined subset of the dataset, and then randomly select K labeled samples from each of the N categories to form the support set. ; S2-2: Randomly select Q samples from the N categories that are different from the support set to form the query set. .
[0007] Preferably, the feature extraction in step S4 includes the following steps: S4-1: Perform mean pooling, MLP global context enhancement, and normalization on the input image samples in sequence to obtain category feature tokens and image feature tokens; S4-2: Using learnable adaptive weights, the information of the category feature tokens is fused into the image feature tokens to obtain the fused feature tokens. Preferably, the specific processing flow of this hybrid expert LSTM module is as follows: Assume the input features are B is the batch size, N is the sequence length, C is the feature channel dimension, and the total number of experts is... , To share the number of experts, The output of the shared experts is expressed using the following formula, representing the number of routing experts: ; in, This represents the LSTM transform function corresponding to the shared expert. The output of the i-th routing expert is expressed by the following formula: ; in, Indicates the first The LSTM transform function corresponding to each routing expert; The outputs of the shared expert and all routing experts are stacked along the expert dimension, as shown by the following formula: ; in, This indicates stacking.
[0008] Preferably, the specific processing flow of the gating network in step S5 is as follows: The input features are flattened and expressed using the following formula: ; in, Indicates reshaping; The gated linear layer calculates the raw routing score for each token to each expert. : ; in, and These represent the weights and biases of the gating layer, respectively. Original route score Divided into shared expert scores and routing expert scores: ; in, This indicates the score corresponding to the shared expert. This represents the score corresponding to the routing expert; The Softmax function is used to obtain the probability of each token being assigned to each routing expert: ; in, Let represent the probability that the t-th token is assigned to the i-th expert. Let represent the probability distribution vector of the t-th token across all experts. This represents the score matrix corresponding to the routing experts. This indicates that the original score of the i-th routing expert corresponds to the t-th token in the score matrix. This indicates that the original score of the j-th routing expert corresponds to the t-th token in the score matrix; For each token, only retain the one with the highest allocation probability among the routing experts. The set of routing expert indices corresponding to each expert is represented as follows: ; in, Indicates the first The set of Top-K expert indices corresponding to each token, where K represents the number of experts retained. This represents the route probability distribution for the t-th token across various routing experts. This represents the selection operation used to select the top K routing expert indices with the highest probability values; For routing experts that are not selected, their corresponding weights are reset to zero; for selected routing experts, their weights are renormalized to obtain the sparse gating weights corresponding to the routing experts. ; in, This represents the final weight assigned to the i-th expert by the t-th token after sparsification and renormalization. This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Restore the sparse gating weights corresponding to the routing experts to a three-dimensional tensor: ; in, This represents the gated weight tensor used for expert output aggregation. This represents tensor reshaping, used to restore the gated weights to their corresponding 3D tensor structure (B, N, E). r ), Indicates the process Routing selection and sparse gating weights after weight normalization, where B represents the number of samples in the batch and N represents the number of tokens corresponding to each sample. Indicates the number of routing experts; Set the shared expert weight as follows: ; By combining the shared expert weights with the routing expert weights, we obtain the gated weight tensor for all experts: ; By combining the shared expert weight and the routing expert sparse weight with the shared expert features and routing expert features output by the expert network, the output of the hybrid expert long short-term memory network branch is obtained: ; in, Indicates the first extracted feature. Indicates the first The feature vector output by an expert at the nth token position in the b-th sample. This indicates the gating weight assigned to the nth token in the bth sample by the i-th expert.
[0009] Preferably, the calculation steps for cosine similarity attention in S7 are as follows: a learnable linear transformation is used to project the global and local features of the input into Q and K. The mean of the feature dimensions is subtracted from Q and K respectively to apply a centering. The attention weights are calculated using the cosine similarity of the calculated Q and K.
[0010] Preferably, the calculation process of the hybrid kernel function in S8 is as follows: The enhanced features of the support set, the enhanced features of the query set, and a list of hyperparameters (Alphas) for controlling the kernel scale are input into the hybrid kernel function. The squared Euclidean distance between all vector pairs is calculated to obtain the distance relationship matrix. This distance relationship matrix is multiplied by the current Alpha value to obtain a Gaussian matrix. The multiple single-scale kernel matrices obtained by parallel calculation of all Alpha values are added together to obtain the aggregated kernel matrix. This aggregated kernel matrix is then averaged by dividing it by the total number of Alpha values to obtain the final multi-scale Gaussian kernel matrix.
[0011] By adopting the aforementioned design scheme, the beneficial effects of this invention are as follows: This application introduces a dynamic, dual-branch hybrid expert LSTM module to transform general features into highly discriminative features; through cosine similarity attention, features can be dynamically adjusted according to the task context, and attention weights are generated by calculating the cosine similarity between the features of the support set and the query set, thereby more accurately locating and matching key discriminative information; by using a hybrid kernel function for distance calculation, the query vectors of the same category move closer to the prototype vectors of their corresponding categories, resulting in better cohesion of the enhanced categories, which is beneficial for classification based on distance metrics. Attached Figure Description
[0012] Figure 1 This is a flowchart illustrating the implementation of the image classification method of the present invention; Figure 2 This is a flowchart of the parallel expert state space module of the image classification method of the present invention. Figure 3 This is a flowchart of the processing of the hybrid expert LSTM module of the present invention; Figure 4 This is a flowchart of the cosine similarity attention calculation of the present invention; Figure 5 This is a flowchart of the distance calculation process for the hybrid kernel function of the present invention; Figure 6 This is a flowchart of the multi-scale kernel MMD distance calculation process of the present invention; Figure 7 This is a comparison chart of clustering scatter plots between the classification method of this invention and the baseline on the same five small sample classification task. Detailed Implementation
[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0014] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0015] Few-sample image classification methods based on parallel expert structures, such as Figure 1 As shown, the steps are executed sequentially as follows: S1: Obtain a small sample dataset, label the small sample dataset, and divide the labeled small sample dataset into a training set (D). train ), Validation set (D val ) and test set (D test These three subsets contain completely mutually exclusive image categories, ensuring that the model faces entirely new categories that it has never seen before during the testing phase.
[0016] S2: At each step of training or testing, an independent few-sample image classification task is created, and a support set and query set are built based on the classification task. Step S2, which supports the construction of the set and query set, includes the following steps: S2-1: Randomly select N categories from a predefined subset of the dataset, and then randomly select K labeled samples from each of the N categories to form the support set. The preset dataset subset here can be set according to actual needs, such as the training set during the training phase.
[0017] S2-2: Randomly select Q samples from the N categories that are different from the support set to form the query set. .
[0018] S3: The iBOT framework is used to perform unsupervised pre-training of the Vision Transformer backbone network, and the pre-trained ViT backbone network is used for subsequent few-shot classification tasks.
[0019] S4: Input all image samples from the support set and the query set into the pre-trained ViT backbone network for feature extraction to obtain category feature tokens and image feature tokens. Then, fuse the category feature tokens and the image feature tokens to obtain the fused feature tokens of the support set and the query set, respectively. The feature extraction in step S4 includes the following steps: S4-1: Perform mean pooling, MLP global context enhancement, and normalization on the input image samples in sequence to obtain category feature tokens and image feature tokens; S4-2: Using learnable adaptive weights, the information of the category feature tokens is fused into the image feature tokens to obtain the fused feature tokens.
[0020] S5: Input the feature tokens obtained by fusing the support set and the query set into the parallel expert Mamba module, such as... Figure 3 The parallel expert Mamba module shown is used to transform general features into highly discriminative features.
[0021] The parallel expert Mamba module comprises two parallel branches. One branch is a hybrid expert LSTM module, which includes two sub-branches. One sub-branches use a gating network to dynamically calculate routing weights for each token in the fused feature tokens. The other sub-branches use an LSTM expert group to extract features from the fused feature tokens and then use the routing weights to perform weighted fusion of the extracted features to obtain the first extracted features. In this embodiment, the expert network part of the hybrid expert LSTM module integrates a hybrid expert structure, with each expert implemented by an LSTM network. The expert network part includes one shared expert and three routing experts. LSTM expert 1 acts as the shared expert and participates in feature extraction throughout each forward computation. LSTM experts 2, 3, and 4 act as routing experts, dynamically selected by the gating network based on the top-K scores of different tokens. By combining the shared expert and the routing experts, this module can both retain general feature representation capabilities and adaptively select differentiated experts for feature enhancement based on the input token.
[0022] The specific processing flow of this hybrid expert LSTM module is as follows: Assume the input features are B is the batch size, N is the sequence length, C is the feature channel dimension, and the total number of experts is... , To share the number of experts, To determine the number of routing experts, in this embodiment, the total number of experts E=4, of which the number of shared experts is... The number of routing experts is . The output of shared experts is expressed using the following formula: ; in, This represents the LSTM transform function corresponding to the shared expert. The output of the i-th routing expert is expressed by the following formula: ; in, Indicates the first The LSTM transform function corresponding to each routing expert; The outputs of the shared expert and all routing experts are stacked along the expert dimension, as shown by the following formula: ; in, The term "stack" here refers to combining multiple tensors along a new dimension. In this application, it means stacking the outputs of multiple experts (including shared experts and routing experts) along the newly added expert dimension to form a unified expert output tensor.
[0023] The specific processing flow of the gating network in step S5 is as follows: The input features are flattened and expressed using the following formula: ; in, This refers to reshaping a tensor, specifically reconstructing its shape by adjusting the dimensional representation of the tensor without altering the data values or their order. In this application, it refers to reshaping the original input dimensions. Remodeling .
[0024] The gated linear layer calculates the raw routing score for each token to each expert. : ; in, and These represent the weights and biases of the gating layer, respectively. Original route score Divided into shared expert scores and routing expert scores: ; in, This indicates the score corresponding to the shared expert. This represents the score corresponding to the routing expert; The Softmax function is used to obtain the probability of each token being assigned to each routing expert: ; in, Indicates the first The probability that a token is assigned to the i-th expert. Let represent the probability distribution vector of the t-th token across all experts. This represents the score matrix corresponding to the routing experts. This indicates that the original score of the i-th routing expert corresponds to the t-th token in the score matrix. This indicates that the original score of the j-th routing expert corresponds to the t-th token in the score matrix; For each token, only retain the one with the highest allocation probability among the routing experts. The set of routing expert indices corresponding to each expert is represented as follows: ; in, Indicates the first The set of Top-K expert indices corresponding to each token, where K represents the number of experts retained. This represents the route probability distribution for the t-th token across various routing experts. This represents the selection operation used to select the top K routing expert indices with the highest probability values; For routing experts that are not selected, their corresponding weights are reset to zero; for selected routing experts, their weights are renormalized to obtain the sparse gating weights corresponding to the routing experts. ; in, This represents the final weight assigned to the i-th expert by the t-th token after sparsification and renormalization. This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Restore the sparse gating weights corresponding to the routing experts to a three-dimensional tensor: ; in, This represents the gated weight tensor used for expert output aggregation. Indicates the first In the nth sample The token is paired with the first The weighting of each expert. This represents tensor reshaping, used to restore the gated weights to their corresponding 3D tensor structure (B, N, E). r ), Indicates the process Routing selection and sparse gating weights after weight normalization, where B represents the number of samples in the batch and N represents the number of tokens corresponding to each sample. Indicates the number of routing experts; Set the shared expert weight as follows: ; By combining the shared expert weights with the routing expert weights, we obtain the gated weight tensor for all experts: ; The shared expert weight and the routing expert sparse weight are combined with the shared expert features and routing expert features output by the expert network to obtain the first extracted feature: ; in, Indicates the first extracted feature. Indicates the first The feature vector output by an expert at the nth token position in the b-th sample. This indicates the gating weight assigned to the nth token in the bth sample by the i-th expert.
[0025] Another branch of this parallel expert Mamba module is the Mamba state space module, which is used to efficiently model the global context and long-range dependencies of feature sequences. This Mamba state space module extracts features from the input fused feature tokens to obtain a second extracted feature. : ; in, This represents the feature transformation function corresponding to the Mamba state-space model.
[0026] The first extracted feature and the second extracted feature are concatenated along the feature dimension to obtain the joint feature. : ; The proposed fusion gating network in this embodiment is used to linearly transform the joint feature, and the adaptive fusion gating coefficients are calculated by combining the activation function. The process is as follows: ; in, This represents the Sigmoid function. This represents the weight parameters of the fusion gated network. This represents the bias parameters of the fused gating network.
[0027] The first extracted feature and the second extracted feature are weighted and fused using the adaptive fusion gating coefficient to obtain the fused feature. : ; in, This represents element-wise multiplication. 1-A is used to control the contribution ratio of the hybrid expert LSTM module, and 1-A is used to control the contribution ratio of the Mamba state space module.
[0028] The fused features are sequentially subjected to linear projection, residual connection, and layer normalization to obtain the enhanced features for the support set and the query set, respectively, expressed by the following formula: ; in, This represents the final output of the hybrid expert LSTM module. This represents the weight parameters of the output projection layer. This represents the bias parameters of the output projection layer. This indicates Layer Normalization, or layer normalization operation. correspond Figure 2 Enhanced features at the bottom output end.
[0029] S6: Divide the enhanced features of the support set into global features of the support set and local features of the support set, and divide the enhanced features of the query set into global features of the query set and local features of the query set. S7: Input the global features of the support set and the local features of the query set into the cosine similarity attention calculation to obtain the attention weights. The local features of the support set and the global features of the query set are input into the cosine similarity attention calculation to obtain the attention weights. The steps for calculating cosine similarity attention in S7 are as follows: a learnable linear transformation is used to project the global and local features of the input into Q and K. The mean of the feature dimensions is subtracted from Q and K respectively to apply a centering. The attention weights are calculated using the cosine similarity of the calculated Q and K.
[0030] like Figure 4 As shown, in this embodiment, it is assumed that To support the collection of global features, To support local features of the set, To query global features of the set, This is for querying local features of a set. Global features of the set are also supported. and query set local features The attention computation between them follows the Transformer paradigm, using a learnable linear transformation to project these features as Q and K: ; ; To enhance the discriminative power of features and remove global bias, the mean μ of the feature dimension is subtracted. To apply a centering operation: ; ; After aligning the feature dimensions, the attention weights are calculated by computed cosine similarity between the centered Q and K representations: ; The same operation is also applied symmetrically to local features of the support set. and query set global features Attention weights can be obtained between these points. This cosine similarity attention mechanism enhances the model's ability to generalize to new samples, ensures balanced contributions among different samples, and provides robustness to changes in feature magnitude.
[0031] S8: The enhanced features of the support set, the enhanced features of the query set, and the attention weights. And the attention weight Input the mixed kernel function, calculate the difference distance between feature distributions, and take the category with the smallest difference distance as the predicted category.
[0032] In this embodiment, a hybrid kernel function consisting of four Gaussian kernels and one linear kernel is used. The bandwidth of the Gaussian kernels is set to -2 to 2 and initialized with learnable parameters.
[0033] Depending on the selected kernel function type, kernel matrices are calculated for the support set, query set, and between the support and query sets. For linear kernels, this is achieved through the dot product of eigenvectors, yielding... , and .
[0034] like Figure 5 As shown, the calculation process of this hybrid kernel function is as follows: The enhanced features of the support set, the enhanced features of the query set, and a list of hyperparameters (Alphas) controlling the kernel scale are input into the hybrid kernel function. The squared Euclidean distances between all vector pairs are calculated, resulting in a raw distance relation matrix without any scale bias. This distance relation matrix is multiplied by the current Alpha value, which controls the width and scale of the Gaussian kernel and determines the distance weights. The scaled result is then exponentially converted into a Gaussian matrix, thus transforming the distance metric into a similarity metric between 0 and 1. (Flowchart shown) , and Each Gaussian kernel matrix represents a specific region within the support set, a region within the query set, and a region between the support set and the query set. Multiple single-scale kernel matrices obtained by parallel calculation of all Alpha values are summed to fuse similarity information at different scales, forming an aggregated kernel matrix. The aggregated kernel matrix is then averaged by dividing it by the total number of Alpha values, outputting the final multi-scale Gaussian kernel matrix that has been averaged and incorporates information from multiple scales. This multi-scale Gaussian kernel matrix is more robust in capturing complex relationships between features than a single-scale kernel matrix.
[0035] In this embodiment, the multi-scale Gaussian kernel matrix obtained above , and Further with linear kernel matrix , and The corresponding additive combinations yield the hybrid kernel matrix. , and Then, the previously obtained cosine similarity attention weights are... and Combining the above hybrid kernel matrix, we obtain the following formula: ; ; ; The final MMD distance equals the expected kernel function values between support set samples, plus the expected kernel function values between query set samples, minus twice the expected cross kernel function values between support and query set samples. Therefore, the algorithm comprises three main terms: MMDs (expected kernel values within the support set), MMDq (expected kernel values within the query set), and MMDsq (expected cross kernel values between the support and query sets). Finally, these three terms are combined into... . Figure 6 The overall process is shown below.
[0036] Table 1 shows the ablation experiments of the parallel expert Mamba module trained in mini-ImageNet using a 5-way 1-shot method:
[0037] Compared to baseline methods without this module, introducing either the hybrid expert LSTM branch or the Mamba state space branch improved classification accuracy to varying degrees, indicating that the two branches enhance feature representation from different perspectives. Specifically, the hybrid expert LSTM branch is better at extracting differentiated expert features, while the Mamba state space branch is better at modeling contextual dependencies in sequences. When both are combined in parallel and fused through an adaptive gating mechanism, the model achieves optimal performance, demonstrating the good complementarity and effectiveness of the proposed parallel expert Mamba module.
[0038] Table 2 shows the ablation experiments of cosine similarity attention in mini-ImageNet trained using a 5-way 1-shot method:
[0039] Table 2 shows that the cosine similarity attention module has a positive effect on improving small-sample classification performance. This module adaptively highlights more discriminative feature regions and suppresses interference from irrelevant background information by interactively modeling the global and local features between the support set and the query set. Compared with the case without this module, the model's classification accuracy is further improved after introducing cosine similarity attention, indicating that this module can enhance the semantic association between the support set and the query set, contributing to improved feature matching quality and classification discriminative ability.
[0040] Table 3 shows the ablation experiments of the differences in multi-scale kernel MMD distance calculation modules in mini-ImageNet trained in a 5-way 1-shot manner:
[0041] Table 3 shows that the hybrid kernel function and multi-scale kernel MMD distance calculation module can further improve the classification performance of the model. Compared with using only a single-scale kernel function or a simple distance metric, multi-scale kernel MMD can simultaneously characterize the feature distribution relationship at different scales, thus more robustly measuring the distribution difference between the support set and the query set. Especially in small sample scenarios, there are often large intra-class variations and subtle inter-class differences between different categories. The multi-scale kernel can model feature similarity from multiple scales, thus obtaining more stable and accurate distance measurement results. Experimental results show that the proposed hybrid kernel function and multi-scale kernel MMD distance calculation method is effective.
[0042] like Figure 7As shown in the left figure, the feature distribution obtained by the Baseline method under the same five-sample classification task exhibits a certain degree of inter-class aliasing and intra-class dispersion. The separation between feature points of different classes is not obvious enough, and some sample points deviate from the center of their respective classes, indicating that the features extracted by this method still have shortcomings in terms of discriminativeness and compactness. Figure 7 As shown in the figure on the right, compared with the clustering scatter plots of the same five few-sample classification tasks combining parallel expert feature enhancement module, cosine similarity attention, and multi-scale maximum mean difference, the method in this application can generate features that are closer to the support set class, more compact within the class, and more separated between classes, indicating that it effectively improves the accuracy of few-sample image classification.
[0043] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A few-sample image classification method based on parallel expert structure, characterized by: The steps are as follows, performed sequentially: S1: Obtain a small sample dataset, label the small sample dataset, and divide the labeled small sample dataset into a training set, a validation set, and a test set; S2: Based on the classification task, construct the support set and query set; S3: Use the iBOT framework to perform unsupervised pre-training on the Vision Transformer backbone network to obtain the pre-trained ViT backbone network; S4: Input all image samples from the support set and the query set into the pre-trained ViT backbone network for feature extraction to obtain category feature tokens and image feature tokens. Then, fuse the category feature tokens and the image feature tokens to obtain the fused feature tokens of the support set and the query set, respectively. S5: Input the feature tokens obtained by fusing the support set and the query set into the parallel expert Mamba module. The parallel expert Mamba module includes two parallel branches. One branch is a hybrid expert LSTM module. The hybrid expert LSTM module includes two branches. One branch uses a gated network to dynamically calculate the routing weight for each token in the fused feature tokens. The other branch uses an LSTM expert group to extract features from the fused feature tokens and uses the routing weight to perform weighted fusion of the extracted features to obtain the first extracted feature. Another branch is the Mamba state space module, which extracts features from the input fused feature tokens to obtain the second extracted features; The first extracted feature and the second extracted feature are concatenated along the feature dimension to obtain a joint feature. A fusion gating network is used to linearly transform the joint feature, and an adaptive fusion gating coefficient is calculated by combining the activation function. The first extracted feature and the second extracted feature are weighted and fused using the adaptive fusion gating coefficient to obtain a fused feature. The fused feature is then subjected to linear projection, residual connection and layer normalization in sequence to obtain the enhanced features of the support set and the query set, respectively. S6: Divide the enhanced features of the support set into global features of the support set and local features of the support set, and divide the enhanced features of the query set into global features of the query set and local features of the query set. S7: Input the global features of the support set and the local features of the query set into the cosine similarity attention calculation to obtain the attention weights. The local features of the support set and the global features of the query set are input into the cosine similarity attention calculation to obtain the attention weights. ; S8: The enhanced features of the support set, the enhanced features of the query set, and the attention weights. And the attention weight Input the mixed kernel function, calculate the difference distance between feature distributions, and take the category with the smallest difference distance as the predicted category.
2. The few-sample image classification method based on parallel expert structure as described in claim 1, characterized in that: The construction of the support set and query set in step S2 includes the following steps: S2-1: Randomly select N categories from a predefined subset of the dataset, and then randomly select K labeled samples from each of the N categories to form the support set. ; S2-2: Randomly select Q samples from the N categories that are different from the support set to form the query set. .
3. The few-sample image classification method based on parallel expert structure as described in claim 2, characterized in that: The feature extraction in step S4 includes the following steps: S4-1: Perform mean pooling, MLP global context enhancement, and normalization on the input image samples in sequence to obtain category feature tokens and image feature tokens; S4-2: Using learnable adaptive weights, the information of the category feature tokens is fused into the image feature tokens to obtain the fused feature tokens.
4. The few-sample image classification method based on parallel expert structure as described in claim 3, characterized in that: The specific processing flow of this hybrid expert LSTM module is as follows: Assume the input features are B is the batch size, N is the sequence length, C is the feature channel dimension, and the total number of experts is... , To share the number of experts, The output of the shared experts is expressed using the following formula, representing the number of routing experts: ； in, This represents the LSTM transform function corresponding to the shared expert. The output of the i-th routing expert is expressed by the following formula: ； in, Indicates the first The LSTM transform function corresponding to each routing expert; The outputs of the shared expert and all routing experts are stacked along the expert dimension, as shown by the following formula: ； in, This indicates stacking.
5. The few-sample image classification method based on parallel expert structure as described in claim 4, characterized in that: The specific processing flow of the gating network in step S5 is as follows: The input features are flattened and expressed using the following formula: ； in, Indicates reshaping; The gated linear layer calculates the raw routing score for each token to each expert. : ; in, and These represent the weights and biases of the gating layer, respectively. Original route score Divided into shared expert scores and routing expert scores: ； in, This indicates the score corresponding to the shared expert. This represents the score corresponding to the routing expert; The Softmax function is used to obtain the probability of each token being assigned to each routing expert: ； in, Indicates the first The probability that a token is assigned to the i-th expert. Let represent the probability distribution vector of the t-th token across all experts. This represents the score matrix corresponding to the routing experts. This indicates that the original score of the i-th routing expert corresponds to the t-th token in the score matrix. This indicates that the original score of the j-th routing expert corresponds to the t-th token in the score matrix; For each token, only retain the one with the highest allocation probability among the routing experts. The set of routing expert indices corresponding to each expert is represented as follows: ； in, Indicates the first The set of Top-K expert indices corresponding to each token, where K represents the number of experts retained. This represents the route probability distribution for the t-th token across various routing experts. This represents the selection operation used to select the top K routing expert indices with the highest probability values; For routing experts that are not selected, their corresponding weights are reset to zero; for selected routing experts, their weights are renormalized to obtain the sparse gating weights corresponding to the routing experts. ； in, This represents the final weight assigned to the i-th expert by the t-th token after sparsification and renormalization. This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Restore the sparse gating weights corresponding to the routing experts to a three-dimensional tensor: ； in, This represents the gated weight tensor used for expert output aggregation. Indicates the first In the nth sample The token is paired with the first... The weighting of each expert. This represents tensor reshaping, used to restore the gated weights to their corresponding 3D tensor structure (B, N, E). r ), Indicates the process Routing selection and sparse gating weights after weight normalization, where B represents the number of samples in the batch and N represents the number of tokens corresponding to each sample. Indicates the number of routing experts; Set the shared expert weight as follows: ； By combining the shared expert weights with the routing expert weights, we obtain the gated weight tensor for all experts: ； By combining the shared expert weight and the routing expert sparse weight with the shared expert features and routing expert features output by the expert network, the output of the hybrid expert long short-term memory network branch is obtained: ； in, Indicates the first extracted feature. Let represent the feature vector output by the i-th expert at the n-th token position in the b-th sample. This indicates the gating weight assigned to the nth token in the bth sample by the i-th expert.
6. The few-sample image classification method based on parallel expert structure as described in claim 5, characterized in that: The steps for calculating cosine similarity attention in S7 are as follows: a learnable linear transformation is used to project the global and local features of the input into Q and K. The mean of the feature dimensions is subtracted from Q and K respectively to apply a centering. The attention weights are calculated using the cosine similarity of the calculated Q and K.
7. The few-sample image classification method based on parallel expert structure as described in claim 6, characterized in that: The calculation process of the hybrid kernel function in S8 is as follows: The enhanced features of the support set, the enhanced features of the query set, and a list of hyperparameters (Alphas) for controlling the kernel scale are input into the hybrid kernel function. The squared Euclidean distance between all vector pairs is calculated to obtain the distance relationship matrix. This distance relationship matrix is multiplied by the current Alpha value to obtain a Gaussian matrix. The multiple single-scale kernel matrices obtained by parallel calculation of all Alpha values are added together to obtain the aggregated kernel matrix. This aggregated kernel matrix is then averaged by dividing it by the total number of Alpha values to obtain the final multi-scale Gaussian kernel matrix.