Small sample classification method based on multi-granularity image-text prototype matching of task-driven structure

Abstract

The present application relates to the technical field of small sample image classification, in particular to a small sample classification method based on multi-granularity image-text prototype matching of task-driven structure; the method comprises the following steps: constructing a support set and a query set; inputting the two sets into a multi-granularity feature set extraction network; inputting an image mode into a multi-level image feature extraction network composed of an improved ResNet12 and an embedded task-driven mapper selection module to extract low, medium and high level image features; calling a large language model LLM three times in sequence to generate word-level, sentence-level and paragraph-level semantic descriptions, and combining a text editor to form multi-angle text features; inputting the image and text feature sets into a semantic graph interaction fusion network to realize multi-level information fusion through node and edge construction, cross-modal attention, graph convolution and gating mechanism; calculating matching scores of each layer through a multi-granularity prototype matching network to perform classification prediction; and adopting a multi-granularity compound weighted loss function to optimize the model performance in the training stage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of small sample image classification, and particularly relates to a small sample classification method based on multi-granularity image-text prototype matching of a task-driven structure. BACKGROUND

[0002] Small sample image classification, as a key challenge in the field of computer vision, aims to achieve accurate identification of image categories under the condition of extremely scarce labeled data. With the rapid evolution of deep learning technology, this research direction has shown important value in practical application scenarios such as medical image diagnosis and industrial quality inspection. However, this task faces challenges such as insufficient classification accuracy, model overfitting, and weak generalization ability due to data scarcity. These technical bottlenecks drive the academic community to seek innovative solutions to improve and break through the performance of small sample classification.

[0003] Chinese Patent Publication No. CN116863239A, entitled "Image Classification Method and Device Based on Feature Diffusion Model", introduces feature migration and diffusion. Deep features are denoised based on deep convolution, and a feature diffusion method is used to learn a feature extractor under a base data set to generate more discriminative features. Compared with the original diffusion algorithm, the method requires less data, saves cost, and improves the classification performance of small sample images. However, the feature extraction of this method only focuses on deep semantics, ignores the complementary value of multi-level features, and excessively relies on a single image modality, without effectively fusing multi-modal information to enhance feature representation capability. In addition, the single matching mechanism based on cosine similarity is difficult to adapt to complex scene requirements, which restricts the generalization performance of the model. Therefore, how to construct an efficient multi-granularity feature extraction mechanism to enhance the representation capability, design a robust multi-modal information fusion strategy to improve the classification accuracy, and optimize the classification module to alleviate the overfitting problem caused by sample scarcity and small inter-class difference, thereby enhancing the generalization performance of the model, has become a key technical challenge that needs to be broken through in the field of small sample image classification. SUMMARY

[0004] (I) Technical problems solved

[0005] The main purpose of the embodiments of the present application is to propose a small sample classification method based on multi-granularity image-text prototype matching of a task-driven structure, which aims to design a task-driven structure to achieve strong feature extraction, and at the same time, construct a multi-granularity image-text prototype matching mechanism to effectively solve the problems of insufficient feature representation, difficulty in cross-modal information fusion, and poor model generalization ability caused by data scarcity in small sample image classification.

[0006] (II) Technical solutions

[0007] The technical scheme for solving the above problems of the present application is to provide a small sample classification method based on a multi-granularity image-text prototype matching task-driven structure, comprising the following steps:

[0008] Step 1, constructing a support set and a query set: extracting samples from a public small sample image dataset, and dividing them into a support set and a query set according to a proportion;

[0009] Step 2, multi-granularity feature set extraction network: inputting the support set and the query set into a multi-granularity feature set extraction network, inputting an image modality into a multi-level image feature extraction network, which adopts an improved ResNet12-based deep feature encoder, and embedding a task-driven mapper selection module after each residual block to extract image features at three levels of low, medium and high granularity, and extracting an image feature set; inputting a text modality into a multi-view text feature extraction network, calling a large language model (LLM) three times to generate multi-view text semantic descriptions at the word, sentence and paragraph levels under the effects of logic, fact and emotion, and then optimizing the structure and enhancing the semantics of the generated text through a text editor to form a three-level text feature set;

[0010] Step 3, semantic graph interaction fusion network: inputting the image and text feature sets into a semantic graph interaction fusion network, and realizing alignment and fusion of multi-level and multi-view features through the joint action of node and edge construction, cross-modal attention, graph convolution and gating fusion strategy to generate a final fusion feature set;

[0011] Step 4, multi-granularity prototype matching network: inputting the fused feature set into a multi-granularity prototype matching network to generate multi-granularity prototype representations at the low-level-word, medium-level-sentence and high-level-paragraph levels, respectively, to capture local details and global semantics, and using a parallel matching mechanism to calculate the matching scores between each prototype and the query sample, while combining the global matching results;

[0012] Step 5, matching score result prediction: performing a classification prediction task based on the fused matching scores, and training the overall network using a multi-granularity composite weighted loss function, and using the trained model for classification.

[0013] Further, the support set and the query set in step 1 are divided in the form of N-way K-shot.

[0014] Further, the multi-granularity feature set extraction network in step 2 includes a multi-level image feature extraction network and a multi-view text feature extraction network. First, the image modal input is input into the multi-level image feature extraction network constructed by the improved ResNet12 to extract low, medium and high level image features. The residual block adopts a deep separable convolution, group normalization and switch activation function, and a task-driven mapper selection module is embedded after each residual block. The module includes three candidate mappers (empty mapper, lightweight mapper and high expression mapper) and a structure selection mechanism. The module can dynamically select one or more adaptive mappers according to the task characteristics, and the structures of the mappers are independent of each other for different level feature extraction.

[0015] The text modal input multi-view text feature extraction network progressively generates word-level, sentence-level and paragraph-level multi-view text semantic descriptions by calling a large language model (LLM) three times based on the category label. The semantic consistency scoring function is used to calculate the semantic association degree between the generated content and the category label, and the subsequent generated content is dynamically adjusted so that the semantic description remains consistent and progressive at multiple levels. The generation fusion logic, facts, emotions and other multi-perspective views are fused, and then a text editor is introduced to optimize the structure and enhance the semantics of the generated multi-perspective text, thereby constructing a high-quality multi-perspective text feature set as the input of the subsequent image-text fusion module.

[0016] Further, the semantic graph interaction fusion network in step 3 includes a node and edge construction module, a cross-modal attention calculation module, and a graph convolution fusion module. First, the image and text features at each level are mapped to nodes in the graph structure, and then three types of edges are constructed: intra-image edges, intra-text edges and cross-modal edges. Then, the cross-modal attention mechanism is used to model the association between each pair of image-text nodes, and the connection weights A ij of each type of edge are dynamically calculated to strengthen the semantic complementarity between images and texts. Subsequently, the graph convolution network is used for multi-round aggregation, and the gating mechanism is introduced to regulate the information fusion contribution from different levels and perspectives, and finally the cross-level and multi-view fusion feature set is obtained as the input of the subsequent matching and classification. The final fusion feature set F fused .

[0017] Further, in step 4, the fused feature set is input into the multi-granularity prototype matching network to generate low-level-word-level, medium-level-sentence-level and high-level-paragraph-level multi-granularity prototypes, and calculate the matching scores for classification prediction.

[0018] By innovatively constructing a multi-granularity prototype representation, efficient matching of cross-level features is realized: first, the feature set after fusion of the two modalities is input into the multi-granularity prototype matching network, and multi-granularity prototype representations of low-level-word level, middle-level-sentence level and high-level-paragraph level are respectively generated according to semantics, respectively capturing local details and global semantics, then, a parallel matching mechanism is adopted to respectively calculate independent matching scores between prototypes of each level and query samples, and meanwhile, the global matching result is combined, and the multi-level independent matching scores and the global matching result are dynamically fused through a gating weighting mechanism to output the final classification result.

[0019] Further, in step 5, the matching score result prediction adopts a multi-granularity composite weighted loss function to train the network model, and an image classification is performed by using the trained classifier, and the multi-granularity composite weighted loss function includes a dynamic marginal cross-entropy loss, a contrast loss and a graph structure loss function.

[0020] (III) Beneficial effects

[0021] Compared with the prior art, the present application provides a multi-granularity image-text prototype matching small sample classification method based on a task-driven structure, and has the following beneficial effects:

[0022] 1. The present application proposes a multi-modal multi-granularity feature extraction method suitable for small sample classification scenarios, in the image modality aspect, a deep feature encoder based on an improved ResNet12 is used, a task-driven mapper selection module is introduced, automatic selection and extraction of low-level, middle-level and high-level image features are realized, and the adaptability of the network to different task complexities is enhanced; in the text modality aspect, a large language model (LLM) is used to generate text semantic descriptions layer by layer in the order of word level, sentence level and paragraph level, and a text editor is used to optimize the structure and enhance the semantics, forming a multi-perspective text feature set with complete semantics. The complementarity between image and text features significantly improves the diversity, discriminability and robustness of feature expression, enhances the anti-interference ability and small sample learning ability of the model, and thus significantly improves the classification accuracy and stability.

[0023] 2. The present application proposes a semantic graph interaction fusion mechanism, by constructing a heterogeneous semantic graph that fuses image levels and text perspectives, introducing image internal edges, text internal edges and cross-modal edges, and accurately modeling the semantic relationships between and within modalities. The cross-modal attention mechanism dynamically calculates the connection strength of the edges in the graph, effectively improving the semantic alignment effect; the introduced gating mechanism then weights and regulates the information of different levels and perspectives during graph convolution fusion, thereby significantly enhancing the consistency and expressiveness of cross-modal feature fusion and improving the generalization ability and semantic integrity in small sample scenarios.

[0024] 3、The application proposes a multi-granularity prototype matching network, which constructs multi-granularity image-text prototypes covering low to high level semantic information, respectively performs fine-grained level matching, and calculates the matching scores between each level prototype and the query sample through a parallel matching mechanism; meanwhile, a gating weighting mechanism is designed to dynamically fuse the multi-level scores and the global matching results, significantly reducing the dependence on single level features and improving the classification performance of the model on complex samples, high inter-class similarity or large intra-class difference samples. In terms of optimization objectives, the proposed multi-granularity composite weighting loss function combines dynamic marginal cross-entropy loss, contrast loss and graph structure loss, forming a multi-dimensional optimization synergistic effect, effectively improving the training stability and enhancing the model robustness, especially suitable for practical application scenarios with extremely small sample quantities. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments of the application. Obviously, the drawings described below are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0026] Figure 1 The flow chart of the multi-granularity image-text prototype matching small sample classification method based on the task-driven structure described in the application;

[0027] Figure 2 The network block diagram of the multi-granularity image-text prototype matching small sample classification method based on the task-driven structure described in the application;

[0028] Figure 3 The structure diagram of the multi-level image feature extraction network described in the application;

[0029] Figure 4 The structure diagram of the task-driven mapper selection module described in the application;

[0030] Figure 5 The structure diagram of the multi-view image feature extraction network described in the application;

[0031] Figure 6 The semantic graph interaction fusion network structure diagram described in the application;

[0032] Figure 7 The multi-granularity prototype matching network structure diagram described in the application;

[0033] Figure 8 The corresponding evaluation index of the method proposed in the application and the existing technology index comparison diagram. DETAILED DESCRIPTION

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example:

[0036] like Figure 1 As shown, an embodiment of the present invention proposes a few-sample classification method based on task-driven multi-granularity image-text prototype matching, which specifically includes the following steps:

[0037] Step 1, Construct the support set and query set: Extract samples from the publicly available small sample image dataset and divide them into a support set and a query set according to the proportions. The support set and query set are divided in the form of N-way K-shot.

[0038] It should be noted that two datasets were selected for the few-sample image dataset: MinimageNet and CUB-200-2011. MinimageNet, as a benchmark dataset for few-sample images, covers a wide range of categories and has diverse scene distributions, which can effectively verify the fusion capabilities of multimodal and multi-granular feature set extraction and semantic graph interaction mechanisms. CUB-200-2011 contains rich attribute labels, making it particularly suitable for multimodal tasks. Moreover, its data distribution shows large intra-class differences and high inter-class similarity, which can verify the feature set matching mechanism.

[0039] Step 2: Construct a multi-granularity feature set extraction network, which includes a multi-level image feature extraction network and a multi-view text feature extraction network. The support set and query set are input into the multi-granularity feature set extraction network, and the image modality is input into the multi-level image feature extraction network. This network adopts a deep feature encoder based on an improved ResNet12. By embedding a task-driven mapper selection module after each residual block, image features at three granularity levels (low, medium, and high) are extracted step by step to extract the image feature set. The text modality is input into the multi-view text feature extraction network, which calls the Large Language Model (LLM) three times to progressively generate word-level, sentence-level, and paragraph-level multi-view text semantic descriptions under the influence of multiple perspectives such as logic, fact, and sentiment. Then, the generated text is structurally optimized and semantically enhanced by a text editor to form a three-level text feature set.

[0040] It should be noted that, as Figure 2As shown, the predefined support set and query set are input into the small-sample classification method for multi-granularity image-text prototype matching based on a task-driven structure proposed in this invention. Feature extraction and fusion are performed through image modality and text modality processing flows, respectively. The image modality part employs a multi-level image feature extraction network to perform deep characterization of image samples, extracting image feature sets at different semantic levels. For example... Figure 3 As shown, this image feature extraction network is built based on an improved ResNet12 and consists of four residual blocks. While maintaining the hierarchical structure, the internal structure of each residual block is optimized for few-shot learning tasks. Specifically, each residual block first introduces depthwise separable convolutions to reduce the number of parameters and the risk of overfitting. Then, Group Normalization (GN), which is more suitable for small-batch training, replaces traditional Batch Normalization (BN), and is combined with the Swish activation function and pooling operations to enhance non-linear expressive power. Since the four residual blocks have the same structure, only a typical structure of one residual block is shown in the figure. Notably, a task-driven mapper selection module is embedded after each residual block. This module can automatically select the appropriate feature mapping path according to task requirements without disrupting the original residual connection structure, thereby effectively extracting multi-granularity feature sets of the image at low, mid, and high levels, laying a high-quality image feature foundation for subsequent cross-modal fusion and matching tasks.

[0041] It needs to be further explained that, such as Figure 4As shown, the task-driven mapper selection module consists of three structurally independent candidate mappers and a structural selection mechanism. The three types of mappers are, in order, an empty mapper, a lightweight mapper, and a high-expressiveness mapper, with their structural complexity and feature modeling capabilities increasing progressively, possessing a hierarchy of representational capabilities from low to high. Among them, the empty mapper is an identity mapping structure that directly outputs the input features without transformation, mainly used to preserve the original feature information while significantly reducing computational overhead; the lightweight mapper consists of a 1×1 convolutional layer and a linear transformation layer, suitable for task scenarios with low requirements for feature changes and weak requirements for semantic transformation, balancing lightweightness and expressiveness; the high-expressiveness mapper has the most complex structure, successively including a 1×1 convolution, a linear transformation, an attention mechanism, a learnable parameter adjustment module, a second linear transformation, a normalization layer, and a global average pooling operation, possessing strong feature reconstruction and discrimination capabilities, suitable for modeling complex semantic structures and long-range dependencies in modeling tasks. The three types of mappers described above are structurally independent, each acting on the output features of each residual block. The structure selection mechanism, however, can adaptively activate one or more mappers based on the feature complexity of different few-sample tasks, participating in the feature transformation and enhancement process of the current level. This module effectively improves the structural flexibility and task adaptability of the image feature extraction network, thereby constructing a multi-level image feature set that combines diversity and discriminative power, providing strong support for subsequent image-text fusion and classification decisions.

[0042] It needs to be further explained that, such as Figure 5 As shown, the multi-view text feature extraction network consists of a language model (LLM), a text editor, and a semantic consistency scoring function. Its aim is to semantically expand the support set category labels and construct high-quality text features. Given that category labels themselves have limited semantic information and are difficult to directly use as input for cross-modal modeling, a language model is needed for hierarchical and multi-view semantic enrichment. Specifically, the network progressively constructs text semantic expressions from shallow to deep by calling the LLM three times sequentially: the result generated in the previous round serves as the semantic guide input for the next round; the semantic consistency scoring function calculates the semantic correlation between the generated content and the category labels, dynamically adjusting subsequent generated content to ensure consistency and progression in semantic description across multiple levels. The specific expression of the semantic consistency scoring function is as follows:

[0043] Score = cos(Embed(G) t ),Embed(C))

[0044] Among them G t Generate text for the current round, where C is the category label, Embed(*) indicates that the text is encoded into a semantic description vector using LLM, and cos(*) indicates cosine similarity;

[0045] The first call to the Large Language Model (LLM) primarily focuses on word-level semantic expansion of category labels, extracting keywords and related concepts to form basic semantic information. The second call to LLM uses the output of the first call as input and calculates relevance using a semantic consistency scoring function. This dynamically adjusts the generated complete sentences to better match the label semantics, further enhancing contextual logic and semantic coherence. The third call follows a similar approach, using the previous round's output as input and combining it with the semantic consistency scoring function to generate paragraph-level text. This integrates background knowledge, typical features, and emotional tone, improving the comprehensiveness and expressiveness of the description. Throughout each level of text generation, LLM considers multiple perspectives, including logic, fact, and emotion, ensuring the text content is diverse, rich, and distinctive. The final multi-level text is then uniformly input into a text editor for structural optimization, language polishing, and semantic enhancement, forming a high-quality multi-perspective text feature set as input for subsequent semantic fusion modules.

[0046] Step 3, construct a semantic graph interaction fusion network: input the image and text feature sets into the semantic graph interaction fusion network, and achieve the alignment and fusion of multi-level and multi-view features through the combined effect of node and edge construction, cross-modal attention, graph convolution and gating fusion strategies, to generate the final fused feature set;

[0047] It should be noted that, as Figure 6 As shown, the semantic graph interaction fusion network includes a node and edge construction module, a cross-modal attention computation module, and a graph convolution fusion module. First, in the image feature construction stage, a hierarchical strategy is adopted to generate image graph nodes of different granularities: for low-level image features, input to the grid partitioning module and perform region average pooling to construct grid nodes of fixed size; for mid-level image features, non-maximum suppression (NMS) is used to extract salient regions, forming mid-level image nodes; and for high-level image features, global average pooling is used to generate high-level image nodes at the full graph level. For the text feature part, the model maps word-level, sentence-level, and paragraph-level text representations to corresponding text graph nodes to comprehensively capture semantic information. Subsequently, based on the structural relationships within the image, the semantic connections within the text, and the cross-modal semantic associations between the image and text, edges in the graph structure are constructed to explicitly model semantic connections between the same modality and across modalities. On this basis, a cross-modal attention mechanism is introduced to assign connection weights A between image nodes and text nodes in the graph. ij Dynamic calculations are performed to enhance the semantic complementarity between images and text, as shown in the following expression:

[0048]

[0049] Where softmax(*) represents the normalized exponential function that maps the input to a probability distribution, V i T represents the features of image node i.j Q (x), K (x) are learnable projection matrices, d represents the dimension of the feature vector.

[0050] This mechanism can effectively suppress invalid connections and strengthen the alignment relationship of key semantics. Finally, the graph convolution network is used for multi-round aggregation operation, and the gating mechanism is introduced to regulate the contribution of information fusion from different levels and perspectives. Finally, the cross-level and multi-perspective fusion feature set is obtained as the input of subsequent matching and classification, and the final fusion feature set F fused is expressed as follows:

[0051] F fused = g1·H local + g2·H mid + g3·H global

[0052]

[0053] where g is the gating weight, H * represents the fusion feature of the corresponding node, local represents the low-level + word-level fusion feature, mid represents the middle-level + sentence-level fusion feature, global represents the high-level + paragraph-level fusion feature, and σ represents the sigmoid function. W * represents the corresponding learnable gating parameter matrix, and g1+g2+g3=1 ensures weight normalization. This feature set not only improves the model's ability to model cross-modal semantic relationships, but also significantly enhances the performance in classification and recognition tasks.

[0054] Step 4, build a multi-granularity prototype matching network: the fused feature set is input into the multi-granularity prototype matching network, and the multi-granularity prototype representation of low-level- word level, middle-level- sentence level, and high-level- paragraph level is generated according to the semantics, respectively capturing local details and global semantics, and using parallel matching mechanism to calculate the independent matching scores between each level prototype and query sample, while combining the global matching result;

[0055] It should be noted that, as Figure 7 shown, the fusion feature set F fusedOn this basis, a multi-granularity prototype representation of low-level-word level, middle-level-sentence level and high-level-paragraph level is innovatively constructed to replace the traditional single fixed prototype. This design aims to comprehensively depict the semantic features of the target class from fine granularity to coarse granularity, thereby enhancing the expression ability of the model in the few-shot scenario, reducing the performance degradation caused by sample scarcity or interference, and improving the overall robustness and task generalization ability. In the matching stage, a parallel matching mechanism is adopted to independently calculate the similarity between the low-level-word level, middle-level-sentence level and high-level-paragraph level prototypes and the target features to be classified, obtaining multi-level matching scores. This strategy ensures that each semantic level can independently capture the corresponding detailed features (such as local texture, structural combination, and overall scene), thereby improving the resolution of complex semantics. At the same time, a global matching score is introduced to preserve the context relevance between different levels of semantics, avoiding the semantic discontinuity problem caused by the fragmentation of information between levels. Finally, a gating weighted fusion mechanism is designed to dynamically fuse the multi-level scores and the global matching score, outputting the final classification matching result. The gating mechanism can adaptively adjust the contribution weight of each level score according to the sample characteristics, achieving more robust and fine decision-making. The higher the matching score, the stronger the similarity between the target sample and the prototype of this class, and the greater the classification probability.

[0056] Step 5, matching score result prediction: perform classification prediction task according to the fused matching score, and use multi-granularity composite weighted loss function to train the overall network, and use the trained model for classification.

[0057] It should be noted that the multi-granularity composite weighted loss function includes dynamic marginal cross-entropy loss, contrastive loss, and graph structure loss function.

[0058] Dynamic marginal cross-entropy loss L s is an improved cross-entropy loss used to improve the ability to distinguish similar classes, and its calculation formula is:

[0059]

[0060] where y c is the true label of class c, S c is the predicted score of class c by the model, γ is the punishment degree, θ c∈H is the indicator function of class c belonging to the easily confused set H, S j is the predicted matching score of class j by the model.

[0061] Contrastive loss L(y, D) pulls similar sample pairs closer and pushes different class sample pairs farther apart, which is used to learn more discriminative features in the few-shot classification, and its calculation formula is:

[0062]

[0063] wherein y represents the label of the sample pair, D is the Euclidean distance of the sample pair, and m is the boundary distance between the sample pairs;

[0064] the graph structure loss L g In order to enhance the relevance of the same sample prediction matrix and the real matrix, and measure the consistency of the image text structure, so that the model learns a more accurate representation, the calculation formula is:

[0065]

[0066] wherein A real is the real relationship matrix, A pred is the model predicted relationship matrix, ||*|| 2 is the square sum of the matrix difference, lambda is the weight of balancing the loss of two items, tr(*) is the trace of the matrix, L img is the graph structure matrix of the image features, and L text is the graph structure matrix of the text features;

[0067] The total loss L is composed of the loss of the above three parts, and can be calculated by weighted combination, and the specific calculation formula of the total loss L is as follows:

[0068] L = alpha L s + beta L (y, D) + eta L g

[0069] Wherein, alpha, beta and eta are weight hyperparameters of the loss function, representing the proportion coefficient of different loss functions in the total loss, used to balance the three losses.

[0070] It should be further pointed out that the network model is trained using the above composite loss function, 200 epochs are set, and the early stopping method is used to monitor the performance of the validation set, and the training is terminated in advance when the validation loss does not improve for 10 consecutive epochs, so as to avoid overfitting. The initial learning rate in the training process is set to 0.001, and the learning rate is dynamically adjusted to ensure that the model does not skip the optimal point during the training process. In order to optimize the network model parameters, the Adam adaptive learning rate optimizer is selected to achieve good results without fine-tuning, and the weight decay 5e-4 is set to prevent overfitting and maintain the generalization ability. When the composite loss function remains relatively stable for 5 consecutive epochs, it means that the model has converged and the training is close to completion. Finally, the trained classifier is used to classify the images;

[0071] The corresponding evaluation index of the method proposed by the application is compared with the index of the prior art, as shown in the figure. Figure 8 As can be seen from the figure, the accuracy of the method proposed by the application is significantly higher than that of the prior art, which shows the effectiveness of the method.

[0072] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that modifications can be made to the technical solutions described in the foregoing embodiments, or some of the technical features thereof can be replaced by equivalent features. Any modifications, equivalent replacements, improvements, and the like made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A small sample classification method based on multi-granularity image-text prototype matching of task-driven structure, characterized in that, The method comprises the following steps: Step 1, constructing a support set and a query set: extracting samples from a public small sample image dataset, and dividing them into a support set and a query set according to a proportion; Step 2, constructing a multi-granularity feature set extraction network, the multi-granularity feature set extraction network comprising a multi-level image feature extraction network and a multi-perspective text feature extraction network, the support set and the query set being input into the multi-granularity feature set extraction network, an image mode being input into the multi-level image feature extraction network, the network adopting a deep feature encoder based on an improved ResNet12, low, medium and high three granularities of image features being extracted through a task-driven mapper selection module embedded after each residual block, and an image feature set being extracted; a text mode being input into the multi-perspective text feature extraction network, a large language model LLM being called three times to generate multi-perspective text semantic descriptions of word, sentence and paragraph levels under the effects of logics, facts and emotions, and a text editor being used to optimize the structure and enhance the semantics of the generated text, so as to form three levels of text features; wherein the residual block comprises a depthwise separable convolution, a group normalization and a Switch activation function, the task-driven mapper selection module comprises three candidate mappers, namely an empty mapper, a lightweight mapper and a high-expression mapper, and a structure selection mechanism, the task-driven mapper selection module can dynamically select one or more adaptive mappers according to the task characteristics, and the structures of the mappers are independent of each other and used for feature extraction of different levels; wherein when the text mode generates the multi-perspective text semantic descriptions, the semantic description results of the previous round are taken as the input of the semantic guidance of the next round, a semantic consistency scoring function is used to calculate the semantic correlation between the generated content and the category label, and the subsequent generated content is dynamically adjusted, so that the semantic descriptions are consistent and progressive at multiple levels, and the specific expression of the semantic consistency scoring function is as follows: ; wherein generating text for the current round, a class label, denotes encoding the text into a semantic description vector using an LLM, denotes cosine similarity; Each level generates fusion logic, fact and emotion multi-perspective, then a text editor is introduced to optimize the structure and enhance the semantics of the generated multi-perspective text, so as to construct a high-quality multi-perspective text feature set as the input of a subsequent image-text fusion module; Step 3, constructing a semantic graph interaction fusion network: inputting the image and text feature sets into the semantic graph interaction fusion network, and realizing alignment and fusion of multi-level and multi-perspective features through joint action of node and edge construction, cross-modal attention, graph convolution and gating fusion strategy, to generate a final fusion feature set; Step 4, constructing a multi-granularity prototype matching network: inputting the fused feature set into the multi-granularity prototype matching network, generating multi-granularity prototype representations of low-level-word, medium-level-sentence and high-level-paragraph according to semantics, respectively capturing local details and global semantics, and using a parallel matching mechanism to calculate independent matching scores between the prototype and the query sample at each level, while combining the global matching result; Step 5, matching score result prediction: performing a classification prediction task according to the fused matching scores, and training the whole network by using a multi-granularity composite weighted loss function, and using the trained model for classification.

2. The method of claim 1, wherein the task-driven structure based multi-granularity text-image prototype matching small sample classification method is characterized in that, The support set and the query set in the step 1 are divided in the form of N-way K-shot.

3. The method of claim 1, wherein the task-driven structure based multi-granularity text-image prototype matching small sample classification method is characterized in that, The semantic graph interaction fusion network in the step 3 comprises a node and edge construction module, a cross-modal attention calculation module and a graph convolution fusion module, first, image and text hierarchical features are correspondingly mapped into nodes in a graph structure, then three types of edges are constructed, namely, intra-image edges, intra-text edges and cross-modal edges, then the cross-modal attention mechanism is used to model the association between each pair of image-text nodes, and the connection weights of various edges are dynamically calculated , the semantic complementarity between the image and the text is strengthened, and the specific expression is as follows: ; wherein represents a normalizing exponential function mapping the input to a probability distribution, represents a feature of image node i, represents a feature of text node j, , is a learnable projection matrix, represents the dimension of the feature vector; Subsequently, a multi-round aggregation operation is performed by using a graph convolution network, and a gating mechanism is introduced to regulate the information fusion contribution from different levels and perspectives. The specific expression is as follows: ; where g is a gating weight, fusion features representing corresponding nodes, fusion features representing low-level + word-level, fusion features representing middle-level + sentence-level, fusion features representing high-level + paragraph-level, σ represents a sigmoid function, represent corresponding learnable gating parameter matrices, g1+g2+g3=1 ensures weight normalization.

4. The method of claim 1, wherein the task-driven structure based multi-granularity text-image prototype matching small sample classification method is characterized in that, In the step 4, a parallel matching mechanism is used to calculate the independent matching scores between the prototypes of each level and the query samples, and the global matching result is combined, the scores are fused through a gating weighting mechanism, and the final classification result is output.

5. The method of claim 1, wherein the task-driven structure based multi-granularity text-image prototype matching small sample classification method is characterized in that, The multi-granularity composite weighted loss function in the step 5 includes a dynamic marginal cross-entropy loss function, a contrast loss function and a graph structure loss function. Dynamic margin cross-entropy loss function An improved cross-entropy loss that improves the ability to distinguish between similar classes is given by ; wherein, is the true label of class c, is the predicted score of class c by the model, and γ is the penalty strength, is an indicator function that class c belongs to the confusion-prone set H, is the predicted matching score of class j by the model; Contrastive loss function The contrastive loss function is used to pull similar sample pairs closer and push different class sample pairs farther apart, which is used to learn more discriminative features in few-shot classification. The calculation formula is as follows: ; Wherein y represents the label of the sample pair, D is the Euclidean distance of the sample pair, and m is the boundary distance between the sample pair. Graph structure loss function To enhance the relevance of the same sample prediction matrix and the real matrix, while measuring the consistency of the image text structure, so that the model learns a more accurate representation, and its calculation formula is: ; wherein is a real relation matrix, is a model predicted relation matrix, is a squared sum of matrix difference, λ is a weight balancing two loss terms, is a trace of a matrix, is a graph structure matrix of image features, is a graph structure matrix of text features; The total loss L is composed of the losses of the above three parts, and is calculated by weighted combination, and the total loss The specific calculation formula is as follows: ; Wherein, alpha, beta and eta are weight hyperparameters of the loss function, representing the proportion coefficients of different loss functions in the total loss, and used to balance the three losses.

Images