Small sample classification method for multi-granularity image-text prototype matching based on task-driven structure

Abstract

The invention relates to the technical field of small sample image classification, in particular to a small sample classification method for multi-granularity image-text prototype matching based on a task-driven structure. The method comprises the following steps: constructing a support set and a query set; inputting the two features into a multi-granularity feature set extraction network, and extracting low-level, middle-level and high-level image features by a multi-level image feature extraction network consisting of an image modal input improved ResNet12 and an embedded task-driven mapper selection module; the text mode calls a large language model (LLM) three times successively to progressively generate word-level, sentence-level and paragraph-level semantic descriptions, and a text editor is combined to form multi-view text features; the image and text feature set is input into a semantic graph interactive fusion network, and multi-level information fusion is realized through node and edge construction, cross-modal attention, graph convolution and a gating mechanism; the matching score of each layer is calculated through the fusion features through a multi-granularity prototype matching network, so that classification prediction is carried out, and the performance of the model is optimized by adopting a multi-granularity composite weighted loss function in a training stage.

Description

Technical Field

[0001] This invention relates to the field of few-sample image classification technology, and in particular to a few-sample classification method based on multi-granularity image-text prototype matching with a task-driven structure. Background Technology

[0002] Few-shot image classification, a key challenge in the field of computer vision, aims to achieve accurate identification of image categories under conditions of extremely scarce labeled data. With the rapid evolution of deep learning technology, this research direction has shown significant value in practical applications such as medical image diagnosis and industrial quality inspection. However, this task faces challenges such as insufficient classification accuracy due to data scarcity, model overfitting, and weak generalization ability. These technical bottlenecks have driven the academic community to seek innovative solutions aimed at improving and breaking through the performance limitations of few-shot classification.

[0003] Chinese patent publication number "CN116863239A" is titled "An Image Classification Method and Apparatus Based on Feature Diffusion Model." This method introduces feature transfer and diffusion, denoising deep features based on deep convolution, and employs feature diffusion to learn a feature extractor on a base class dataset, generating more discriminative features. Compared to the original diffusion algorithm, it requires less data, saves costs, and improves classification performance for small sample images. However, this method's feature extraction focuses only on deep semantics, neglecting the complementary value of multi-level features, and over-relies on a single image modality, failing to effectively fuse multi-modal information to enhance feature representation capabilities. Furthermore, the single matching mechanism based on cosine similarity is difficult to adapt to complex scene requirements, limiting the model's generalization performance. Therefore, how to construct an efficient multi-granularity feature extraction mechanism to enhance representation capabilities, design a robust multi-modal information fusion strategy to improve classification accuracy, and optimize the classification module to alleviate overfitting problems caused by sample scarcity and small inter-class differences, thereby enhancing the model's generalization performance, has become a key technical challenge that urgently needs to be overcome in the field of small sample image classification. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] The main objective of this invention is to propose a few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching. The aim is to design a task-driven structure to achieve powerful feature extraction, while constructing a multi-granularity image-text prototype matching mechanism to effectively solve the problems of insufficient feature representation, difficulty in cross-modal information fusion, and poor model generalization ability caused by data scarcity in few-sample image classification.

[0006] (II) Technical Solution

[0007] The technical solution of this invention to solve the above problems is to provide a few-sample classification method based on multi-granularity image-text prototype matching with a task-driven structure, including the following steps:

[0008] Step 1, Construct the support set and query set: Extract samples from the publicly available small sample image dataset and divide them into the support set and query set according to the proportion;

[0009] Step 2, Multi-granularity feature set extraction network: The support set and query set are input into the multi-granularity feature set extraction network, and the image modality is input into the multi-level image feature extraction network. This network adopts a deep feature encoder based on the improved ResNet12. By embedding a task-driven mapper selection module after each residual block, image features at three granularity levels (low, medium, and high) are extracted step by step to extract the image feature set. The text modality is input into the multi-perspective text feature extraction network. The Large Language Model (LLM) is called three times to progressively generate multi-perspective text semantic descriptions at the word, sentence, and paragraph levels under the influence of multiple perspectives such as logic, fact, and sentiment. Then, the generated text is structurally optimized and semantically enhanced by a text editor to form a three-level text feature set.

[0010] Step 3, Semantic Graph Interaction Fusion Network: Input the image and text feature sets into the semantic graph interaction fusion network. Through the combined effect of node and edge construction, cross-modal attention, graph convolution and gating fusion strategies, the alignment and fusion of multi-level and multi-view features are achieved, generating the final fused feature set.

[0011] Step 4, Multi-granularity prototype matching network: The fused feature set is input into the multi-granularity prototype matching network to generate multi-granularity prototype representations at the low-level (word level), mid-level (sentence level), and high-level (paragraph level) levels, respectively, capturing local details and global semantics. A parallel matching mechanism is used to calculate the matching score between each prototype and the query sample, and the global matching results are combined.

[0012] Step 5, Matching Score Prediction: Based on the fused matching scores, perform a classification prediction task, and train the overall network using a multi-granularity composite weighted loss function, then use the trained model for classification.

[0013] Furthermore, in step 1, the support set and query set are divided in the form of N-way K-shot.

[0014] Furthermore, the multi-granularity feature extraction network in step 2 includes a multi-level image feature extraction network and a multi-view text feature extraction network. First, the image modality is input into the multi-level image feature extraction network constructed by the improved ResNet12 to extract low, medium, and high-level image features. The residual blocks are composed of depthwise separable convolution, group normalization, and switch activation functions. At the same time, a task-driven mapper selection module is embedded after each residual block. This module contains three candidate mappers (empty mapper, lightweight mapper, and high-expressiveness mapper) and a structure selection mechanism. This module can dynamically select one or more adaptable mappers according to the task characteristics, and the structures of each mapper are independent of each other and used for feature extraction at different levels.

[0015] A multi-perspective text feature extraction network is used for text modal input. Based on category labels, a Large Language Model (LLM) is invoked three times to progressively generate multi-perspective text semantic descriptions at the word, sentence, and paragraph levels. The results of the previous generation serve as the semantic guide input for the next generation. A semantic consistency scoring function is used to calculate the semantic relevance between the generated content and the category labels, dynamically adjusting the subsequently generated content to ensure consistency and progression in semantic descriptions across multiple levels. Each level of generation integrates multiple perspectives such as logic, fact, and sentiment. Subsequently, a text editor is introduced to optimize the structure and enhance the semantics of the generated text from each perspective, constructing a high-quality multi-perspective text feature set as input for the subsequent image-text fusion module.

[0016] Furthermore, the semantic graph interaction fusion network in step 3 includes a node and edge construction module, a cross-modal attention calculation module, and a graph convolution fusion module. First, the features of each level of the image and text are mapped to nodes in the graph structure. Then, three types of edges are constructed: image-internal edges, text-internal edges, and cross-modal edges. Subsequently, the cross-modal attention mechanism is used to model the relationship between each pair of image-text nodes, and the connection weights A of each type of edge are dynamically calculated. ij This strengthens the semantic complementarity between images and text. Subsequently, a graph convolutional network is used for multiple rounds of aggregation operations, and a gating mechanism is introduced to regulate the fusion contribution of information from different levels and perspectives, ultimately obtaining a cross-level, multi-view fusion feature set, which serves as input for subsequent matching and classification. The final fusion feature set F fused .

[0017] Furthermore, in step 4, the fused feature set is input into a multi-granularity prototype matching network to generate low-level-word-level, mid-level-sentence-level, and high-level-paragraph-level multi-granularity prototypes, and the matching score is calculated to perform classification prediction.

[0018] By innovatively constructing multi-granularity prototype representations, efficient matching of cross-level features is achieved: First, the feature set after fusing two modalities is input into the multi-granularity prototype matching network, and multi-granularity prototype representations are generated according to semantics at the low-level (word-level), mid-level (sentence-level), and high-level (paragraph-level) levels, respectively capturing local details and global semantics. Then, a parallel matching mechanism is adopted to calculate the independent matching scores between prototypes at each level and query samples. At the same time, the global matching results are combined, and a gating weighting mechanism is used to dynamically fuse the multi-level independent matching scores and the global matching results to output the final classification result.

[0019] Furthermore, in step 5, the matching score prediction is performed by training the network model using a multi-granularity composite weighted loss function, and then using the trained classifier to classify the image. The multi-granularity composite weighted loss function includes dynamic marginal cross-entropy loss, contrast loss, and graph structure loss function.

[0020] (III) Beneficial Effects

[0021] Compared with existing technologies, this invention provides a few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching, which has the following beneficial effects:

[0022] 1. This invention proposes a multimodal, multi-granular feature extraction method suitable for few-sample classification scenarios. For image modalities, a deep feature encoder based on an improved ResNet12 is introduced, incorporating a task-driven mapper selection module to automatically select and extract low-level, mid-level, and high-level image features, enhancing the network's adaptability to different task complexities. For text modalities, a Large Language Model (LLM) is used to generate text semantic descriptions layer by layer at the word, sentence, and paragraph levels. A text editor is then used to optimize the structure and enhance the semantics, forming a semantically complete multi-perspective text feature set. The complementarity between image and text features significantly improves the diversity, discriminativeness, and robustness of feature representation, enhancing the model's anti-interference ability and few-sample learning ability, thereby significantly improving classification accuracy and stability.

[0023] 2. This invention proposes a semantic graph interaction fusion mechanism. By constructing a heterogeneous semantic graph that integrates image hierarchy and text perspective, it introduces intra-image edges, intra-text edges, and cross-modal edges to accurately model semantic relationships between and within modalities. A cross-modal attention mechanism dynamically calculates the connection strength of edges in the graph, effectively improving semantic alignment. The introduced gating mechanism weights and controls information from different levels and perspectives during the graph convolution fusion stage, thereby significantly enhancing the consistency and expressiveness of cross-modal feature fusion and improving generalization ability and semantic integrity in small sample scenarios.

[0024] 3. This invention proposes a multi-granularity prototype matching network. This network constructs multi-granularity graph-text prototypes encompassing low- to high-level semantic information, performs fine-grained hierarchical matching on each, and calculates the matching score between each level prototype and the query sample through a parallel matching mechanism. Simultaneously, a gating weighting mechanism is designed to dynamically fuse multi-level scores with the global matching result, significantly reducing the dependence on single-level features and improving the model's classification performance for complex samples, samples with high inter-class similarity, or samples with large intra-class differences. Regarding the optimization objective, the proposed multi-granularity composite weighted loss function combines dynamic marginal cross-entropy loss, contrastive loss, and graph structure loss, forming a multi-dimensional optimization synergy effect, effectively improving training stability and enhancing model robustness, especially suitable for practical applications with extremely small sample sizes. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the small sample classification method based on task-driven structure multi-granularity image-text prototype matching according to the present invention;

[0027] Figure 2 This is a network diagram of the few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching according to the present invention.

[0028] Figure 3 This is a structural diagram of the multi-level image feature extraction network described in this invention;

[0029] Figure 4 This is a structural diagram of the task-driven mapper selection module described in this invention;

[0030] Figure 5 This is a structural diagram of the multi-view image feature extraction network described in this invention;

[0031] Figure 6 This is a semantic graph interaction fusion network structure diagram as described in this invention;

[0032] Figure 7 This is a diagram of the multi-granularity prototype matching network structure described in this invention;

[0033] Figure 8 This diagram illustrates a comparison between the evaluation indicators of the method proposed in this invention and existing technical indicators. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example:

[0036] like Figure 1 As shown, an embodiment of the present invention proposes a few-sample classification method based on task-driven multi-granularity image-text prototype matching, which specifically includes the following steps:

[0037] Step 1, Construct the support set and query set: Extract samples from the publicly available small sample image dataset and divide them into a support set and a query set according to the proportions. The support set and query set are divided in the form of N-way K-shot.

[0038] It should be noted that two datasets were selected for the few-sample image dataset: MinimageNet and CUB-200-2011. MinimageNet, as a benchmark dataset for few-sample images, covers a wide range of categories and has diverse scene distributions, which can effectively verify the fusion capabilities of multimodal and multi-granular feature set extraction and semantic graph interaction mechanisms. CUB-200-2011 contains rich attribute labels, making it particularly suitable for multimodal tasks. Moreover, its data distribution shows large intra-class differences and high inter-class similarity, which can verify the feature set matching mechanism.

[0039] Step 2: Construct a multi-granularity feature set extraction network, which includes a multi-level image feature extraction network and a multi-view text feature extraction network. The support set and query set are input into the multi-granularity feature set extraction network, and the image modality is input into the multi-level image feature extraction network. This network adopts a deep feature encoder based on an improved ResNet12. By embedding a task-driven mapper selection module after each residual block, image features at three granularity levels (low, medium, and high) are extracted step by step to extract the image feature set. The text modality is input into the multi-view text feature extraction network, which calls the Large Language Model (LLM) three times to progressively generate word-level, sentence-level, and paragraph-level multi-view text semantic descriptions under the influence of multiple perspectives such as logic, fact, and sentiment. Then, the generated text is structurally optimized and semantically enhanced by a text editor to form a three-level text feature set.

[0040] It should be noted that, as Figure 2As shown, the predefined support set and query set are input into the small-sample classification method for multi-granularity image-text prototype matching based on a task-driven structure proposed in this invention. Feature extraction and fusion are performed through image modality and text modality processing flows, respectively. The image modality part employs a multi-level image feature extraction network to perform deep characterization of image samples, extracting image feature sets at different semantic levels. For example... Figure 3 As shown, this image feature extraction network is built based on an improved ResNet12 and consists of four residual blocks. While maintaining the hierarchical structure, the internal structure of each residual block is optimized for few-shot learning tasks. Specifically, each residual block first introduces depthwise separable convolutions to reduce the number of parameters and the risk of overfitting. Then, Group Normalization (GN), which is more suitable for small-batch training, replaces traditional Batch Normalization (BN), and is combined with the Swish activation function and pooling operations to enhance non-linear expressive power. Since the four residual blocks have the same structure, only a typical structure of one residual block is shown in the figure. Notably, a task-driven mapper selection module is embedded after each residual block. This module can automatically select the appropriate feature mapping path according to task requirements without disrupting the original residual connection structure, thereby effectively extracting multi-granularity feature sets of the image at low, mid, and high levels, laying a high-quality image feature foundation for subsequent cross-modal fusion and matching tasks.

[0041] It needs to be further explained that, such as Figure 4As shown, the task-driven mapper selection module consists of three structurally independent candidate mappers and a structural selection mechanism. The three types of mappers are, in order, an empty mapper, a lightweight mapper, and a high-expressiveness mapper, with their structural complexity and feature modeling capabilities increasing progressively, possessing a hierarchy of representational capabilities from low to high. Among them, the empty mapper is an identity mapping structure that directly outputs the input features without transformation, mainly used to preserve the original feature information while significantly reducing computational overhead; the lightweight mapper consists of a 1×1 convolutional layer and a linear transformation layer, suitable for task scenarios with low requirements for feature changes and weak requirements for semantic transformation, balancing lightweightness and expressiveness; the high-expressiveness mapper has the most complex structure, successively including a 1×1 convolution, a linear transformation, an attention mechanism, a learnable parameter adjustment module, a second linear transformation, a normalization layer, and a global average pooling operation, possessing strong feature reconstruction and discrimination capabilities, suitable for modeling complex semantic structures and long-range dependencies in modeling tasks. The three types of mappers described above are structurally independent, each acting on the output features of each residual block. The structure selection mechanism, however, can adaptively activate one or more mappers based on the feature complexity of different few-sample tasks, participating in the feature transformation and enhancement process of the current level. This module effectively improves the structural flexibility and task adaptability of the image feature extraction network, thereby constructing a multi-level image feature set that combines diversity and discriminative power, providing strong support for subsequent image-text fusion and classification decisions.

[0042] It needs to be further explained that, such as Figure 5 As shown, the multi-view text feature extraction network consists of a language model (LLM), a text editor, and a semantic consistency scoring function. Its aim is to semantically expand the support set category labels and construct high-quality text features. Given that category labels themselves have limited semantic information and are difficult to directly use as input for cross-modal modeling, a language model is needed for hierarchical and multi-view semantic enrichment. Specifically, the network progressively constructs text semantic expressions from shallow to deep by calling the LLM three times sequentially: the result generated in the previous round serves as the semantic guide input for the next round; the semantic consistency scoring function calculates the semantic correlation between the generated content and the category labels, dynamically adjusting subsequent generated content to ensure consistency and progression in semantic description across multiple levels. The specific expression of the semantic consistency scoring function is as follows:

[0043] Score = cos(Embed(G) t ),Embed(C))

[0044] Among them G t Generate text for the current round, where C is the category label, Embed(*) indicates that the text is encoded into a semantic description vector using LLM, and cos(*) indicates cosine similarity;

[0045] The first call to the Large Language Model (LLM) primarily focuses on word-level semantic expansion of category labels, extracting keywords and related concepts to form basic semantic information. The second call to LLM uses the output of the first call as input and calculates relevance using a semantic consistency scoring function. This dynamically adjusts the generated complete sentences to better match the label semantics, further enhancing contextual logic and semantic coherence. The third call follows a similar approach, using the previous round's output as input and combining it with the semantic consistency scoring function to generate paragraph-level text. This integrates background knowledge, typical features, and emotional tone, improving the comprehensiveness and expressiveness of the description. Throughout each level of text generation, LLM considers multiple perspectives, including logic, fact, and emotion, ensuring the text content is diverse, rich, and distinctive. The final multi-level text is then uniformly input into a text editor for structural optimization, language polishing, and semantic enhancement, forming a high-quality multi-perspective text feature set as input for subsequent semantic fusion modules.

[0046] Step 3, construct a semantic graph interaction fusion network: input the image and text feature sets into the semantic graph interaction fusion network, and achieve the alignment and fusion of multi-level and multi-view features through the combined effect of node and edge construction, cross-modal attention, graph convolution and gating fusion strategies, to generate the final fused feature set;

[0047] It should be noted that, as Figure 6 As shown, the semantic graph interaction fusion network includes a node and edge construction module, a cross-modal attention computation module, and a graph convolution fusion module. First, in the image feature construction stage, a hierarchical strategy is adopted to generate image graph nodes of different granularities: for low-level image features, input to the grid partitioning module and perform region average pooling to construct grid nodes of fixed size; for mid-level image features, non-maximum suppression (NMS) is used to extract salient regions, forming mid-level image nodes; and for high-level image features, global average pooling is used to generate high-level image nodes at the full graph level. For the text feature part, the model maps word-level, sentence-level, and paragraph-level text representations to corresponding text graph nodes to comprehensively capture semantic information. Subsequently, based on the structural relationships within the image, the semantic connections within the text, and the cross-modal semantic associations between the image and text, edges in the graph structure are constructed to explicitly model semantic connections between the same modality and across modalities. On this basis, a cross-modal attention mechanism is introduced to assign connection weights A between image nodes and text nodes in the graph. ij Dynamic calculations are performed to enhance the semantic complementarity between images and text, as shown in the following expression:

[0048]

[0049] Where softmax(*) represents the normalized exponential function that maps the input to a probability distribution, V i T represents the features of image node i.j Let Q(*) and K(*) represent the features of text node j, respectively. Let d represent the dimension of the feature vector.

[0050] This mechanism effectively suppresses invalid connections and strengthens the alignment of key semantics. Finally, a graph convolutional network is used for multiple rounds of aggregation operations, and a gating mechanism is introduced to regulate the fusion contributions of information from different levels and perspectives, ultimately obtaining a cross-level-multi-view fused feature set, which serves as the input for subsequent matching and classification. The final fused feature set F fused Its specific expression is as follows:

[0051] F fused =g1·H local +g2·H mid +g3·H global

[0052]

[0053] Where g is the gating weight, H * Represents the fusion features of the corresponding nodes: local represents low-level + word-level fusion features, mid represents mid-level + sentence-level fusion features, global represents high-level + paragraph-level fusion features, σ represents the sigmoid function, and W... * This represents the corresponding learnable gating parameter matrix, where g1+g2+g3=1 ensures weight normalization. This feature set not only improves the model's ability to model cross-modal semantic relationships but also significantly enhances its performance in classification and recognition tasks.

[0054] Step 4: Construct a multi-granularity prototype matching network: The fused feature set is input into the multi-granularity prototype matching network, which generates multi-granularity prototype representations at the low-level (word level), mid-level (sentence level), and high-level (paragraph level) according to semantics. These representations capture local details and global semantics respectively. A parallel matching mechanism is used to calculate the independent matching scores between prototypes at each level and query samples, while also combining the global matching results.

[0055] It should be noted that, as Figure 7 As shown, the fusion feature set F obtained in the previous stage fusedBuilding upon this foundation, a novel multi-granularity prototype representation is constructed, encompassing low-level (word-level), mid-level (sentence-level), and high-level (paragraph-level) representations, to replace the traditional single fixed prototype approach. This design aims to comprehensively characterize the semantic features of the target category from fine-grained to coarse-grained, thereby enhancing the model's expressive power in low-sample scenarios, reducing performance degradation caused by sample scarcity or interference, and improving overall robustness and task generalization ability. In the matching stage, a parallel matching mechanism is employed, independently calculating the similarity between the low-level (word-level), mid-level (sentence-level), and high-level (paragraph-level) prototypes and the features to be classified, obtaining multi-level matching scores. This strategy ensures that each semantic level can independently capture corresponding detailed features (such as local texture, structural combinations, and overall scene), thus improving the ability to distinguish complex semantics. Simultaneously, a global matching score is introduced to preserve the contextual relevance between different semantic levels, avoiding semantic fragmentation caused by information separation between levels. Finally, a gated weighted fusion mechanism is designed to dynamically fuse the multi-level scores and the global matching score, outputting the final classification matching result. The gating mechanism can adaptively adjust the contribution weight of scores at each level based on sample characteristics, achieving more robust and refined decision-making. The higher the matching score, the stronger the similarity between the target sample and the prototype of that class, and the greater the classification probability.

[0056] Step 5, Matching Score Prediction: Based on the fused matching scores, perform a classification prediction task, and train the overall network using a multi-granularity composite weighted loss function, then use the trained model for classification.

[0057] It should be noted that the multi-granularity composite weighted loss function includes dynamic marginal cross-entropy loss, contrast loss, and graph structure loss function;

[0058] Dynamic marginal cross-entropy loss L s It is an improved cross-entropy loss, used to enhance the ability to distinguish similar categories. Its calculation formula is as follows:

[0059]

[0060] Among them, y c It is the true label of category c, S c γ is the model's predicted score for category c, γ is the penalty strength, and θ is the score predicted for category c. c∈H S is an indicator function that class c belongs to the easily confused set H. j It is the model's predicted matching score for category j;

[0061] The contrastive loss L(y,D) narrows the distance between similar sample pairs and widens the distance between sample pairs of different classes. It is used to learn more discriminative features in small sample classification. Its calculation formula is as follows:

[0062]

[0063] Where y represents the label of the sample pair, D is the Euclidean distance between the sample pairs, and m is the boundary distance between the sample pairs;

[0064] Graph structure loss L g It is used to enhance the correlation between the predicted matrix and the true matrix of similar samples, while measuring the consistency of image and text structure, so that the model learns a more accurate representation. Its calculation formula is:

[0065]

[0066] Where A real Let A be the true relation matrix. pred For the model to predict the relation matrix, ||*|| 2 Let λ be the sum of squared differences of the matrices, λ be the weights balancing the two losses, tr(*) be the trace of the matrix, and L be the sum of squared differences of the matrix. img L is the graph structure matrix of image features. text The graph structure matrix represents the text features;

[0067] The total loss L consists of the losses from the three parts mentioned above, and can be calculated using a weighted combination. The specific formula for calculating the total loss L is as follows:

[0068] L=αL s +βL(y,D)+ηL g

[0069] Here, α, β, and η are the weight hyperparameters of the loss function, representing the proportion of different loss functions in the total loss, used to balance the three types of loss.

[0070] It should be further explained that the network model was trained using the aforementioned composite loss function for 200 epochs, and early stopping was used to monitor the validation set performance. Training was terminated early when the validation loss did not improve within 10 consecutive epochs to avoid overfitting. The initial learning rate was set to 0.001 and dynamically adjusted to ensure the model did not skip the optimal point during training. To optimize the network model parameters, the Adam adaptive learning rate optimizer was chosen, achieving good results without requiring fine-tuning. A weight decay of 5e-4 was set to prevent overfitting while maintaining generalization ability. When the composite loss function remained relatively stable for 5 consecutive epochs, it meant the model had converged and training was nearing completion. Finally, the trained classifier was used to classify images.

[0071] The evaluation indicators of the method proposed in this invention are compared with those of existing technologies, such as... Figure 8 As shown in the figure, the accuracy of the method proposed in this invention is significantly higher than that of the prior art, demonstrating the effectiveness of this method.

[0072] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A few-sample classification method based on multi-granularity image-text prototype matching with a task-driven structure, characterized in that, Includes the following steps: Step 1, Construct the support set and query set: Extract samples from the publicly available small sample image dataset and divide them into the support set and query set according to the proportion; Step 2: Construct a multi-granularity feature set extraction network, which includes a multi-level image feature extraction network and a multi-view text feature extraction network. The support set and query set are input into the multi-granularity feature set extraction network, and the image modality is input into the multi-level image feature extraction network. This network adopts a deep feature encoder based on an improved ResNet12. By embedding a task-driven mapper selection module after each residual block, image features at three granularity levels (low, medium, and high) are extracted step by step to extract the image feature set. The text modality is input into the multi-view text feature extraction network, which calls the Large Language Model (LLM) three times to progressively generate word-level, sentence-level, and paragraph-level multi-view text semantic descriptions under the influence of multiple perspectives such as logic, fact, and sentiment. Then, the generated text is structurally optimized and semantically enhanced by a text editor to form a three-level text feature set. Step 3, construct a semantic graph interaction fusion network: input the image and text feature sets into the semantic graph interaction fusion network, and achieve the alignment and fusion of multi-level and multi-view features through the combined effect of node and edge construction, cross-modal attention, graph convolution and gating fusion strategies, to generate the final fused feature set; Step 4: Construct a multi-granularity prototype matching network: The fused feature set is input into the multi-granularity prototype matching network, which generates multi-granularity prototype representations at the low-level (word level), mid-level (sentence level), and high-level (paragraph level) according to semantics. These representations capture local details and global semantics respectively. A parallel matching mechanism is used to calculate the independent matching scores between prototypes at each level and query samples, while also combining the global matching results. Step 5, Matching Score Prediction: Based on the fused matching scores, perform a classification prediction task, and train the overall network using a multi-granularity composite weighted loss function, then use the trained model for classification.

2. The few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching according to claim 1, characterized in that, In step 1, the support set and query set are divided in the form of N-way K-shot.

3. The few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching according to claim 1, characterized in that, In step 2, the residual block is composed of depthwise separable convolution, group normalization, and switch activation function. The task-driven mapper selection module includes three candidate mappers: empty mapper, lightweight mapper, and high expressiveness mapper, as well as a structure selection mechanism. The task-driven mapper selection module can dynamically select one or more adaptable mappers according to the task characteristics, and the structures of each mapper are independent of each other and used for feature extraction at different levels. In step 2, when the text modality generates multi-perspective text semantic descriptions, the semantic description results generated in the previous round are used as input for semantic guidance in the next round. Subsequently, the semantic consistency scoring function is used to calculate the semantic relevance between the generated content and the category labels, and the subsequent generated content is dynamically adjusted to ensure that the semantic descriptions maintain consistency and progression at multiple levels. The specific expression of the semantic consistency scoring function is as follows: Score=cos(Embed(G t ),Embed(C)) Among them G t Generate text for the current round, where C is the category label, Embed(*) indicates that the text is encoded into a semantic description vector using LLM, and cos(*) indicates cosine similarity; Each level generates multiple perspectives, including logic, facts, and emotions. Then, a text editor is introduced to optimize the structure and enhance the semantics of the generated text from each perspective, constructing a high-quality multi-perspective text feature set as input for the subsequent image-text fusion module.

4. The few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching according to claim 1, characterized in that, The semantic graph interaction fusion network in step 3 includes a node and edge construction module, a cross-modal attention calculation module, and a graph convolution fusion module. First, the features of each level of the image and text are mapped to nodes in the graph structure. Then, three types of edges are constructed: image-internal edges, text-internal edges, and cross-modal edges. Next, the cross-modal attention mechanism is used to model the relationship between each pair of image-text nodes, and the connection weights A of each type of edge are dynamically calculated. ij This enhances the semantic complementarity between images and text, as shown in the following expression: Where softmax(*) represents the normalized exponential function that maps the input to a probability distribution, V i T represents the features of image node i. j Let Q(*) and K(*) represent the features of text node j, respectively. Let d represent the dimension of the feature vector. Subsequently, a graph convolutional network is used for multiple rounds of aggregation operations, and a gating mechanism is introduced to regulate the fusion contribution of information from different levels and perspectives, ultimately obtaining a cross-level-multi-view fused feature set, which serves as the input for subsequent matching and classification. The final fused feature set F fused Its specific expression is as follows: F fused =g1·H local +g2·H mid +g3·H global Where g is the gating weight, H * Represents the fusion features of the corresponding nodes: local represents low-level + word-level fusion features, mid represents mid-level + sentence-level fusion features, global represents high-level + paragraph-level fusion features, σ represents the sigmoid function, and W... * This represents the corresponding learnable gating parameter matrix, and g1+g2+g3=1 ensures weight normalization.

5. The few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching according to claim 1, characterized in that, In step 4, a parallel matching mechanism is used to calculate the independent matching scores between each level prototype and the query sample, while combining the global matching results. The scores are then fused through a gating weighting mechanism to output the final classification result.

6. The few-sample classification method based on task-driven structure and multi-granularity image-text prototype matching according to claim 1, characterized in that, The multi-granularity composite weighted loss function in step 5 includes the dynamic marginal cross-entropy loss function, the contrast loss function, and the graph structure loss function. Dynamic marginal cross-entropy loss function L s It is an improved cross-entropy loss, used to enhance the ability to distinguish similar categories. Its calculation formula is as follows: Among them, y c It is the true label of category c, S c γ is the model's predicted score for category c, γ is the penalty strength, and θ is the score predicted for category c. c∈H S is an indicator function that class c belongs to the easily confused set H. j It is the model's predicted matching score for category j; The contrast loss function L(y,D) shortens the distance between similar sample pairs and widens the distance between sample pairs of different classes, and is used to learn more discriminative features in small sample classification. Its calculation formula is as follows: Where y represents the label of the sample pair, D is the Euclidean distance between the sample pairs, and m is the boundary distance between the sample pairs; Graph structure loss function L g It is used to enhance the correlation between the predicted matrix and the true matrix of similar samples, while measuring the consistency of image and text structure, so that the model learns a more accurate representation. Its calculation formula is: Where A real Let A be the true relation matrix. pred For the model to predict the relation matrix, ||*|| 2 Let λ be the sum of squared differences of the matrices, λ be the weights balancing the two losses, tr(*) be the trace of the matrix, and L be the sum of squared differences of the matrix. img L is the graph structure matrix of image features. text The graph structure matrix represents the text features; The total loss L consists of the losses from the three parts mentioned above, and can be calculated using a weighted combination. The specific formula for calculating the total loss L is as follows: L=αL s +βL(y,D)+ηL g Here, α, β, and η are the weight hyperparameters of the loss function, representing the proportion of different loss functions in the total loss, used to balance the three types of loss.

Images