Multimodal large model small sample classification method based on task enhancement and efficient reasoning

Abstract

This invention discloses a multimodal large-model few-shot classification method based on task enhancement and efficient inference, comprising: synthesizing training samples with diverse backgrounds using a semantically guided background generation strategy to break the false visual correlation between foreground and background; constructing a meta-task containing negative samples of varying difficulty based on multimodal similarity clustering, and constructing a position-balanced candidate answer list during the instruction fine-tuning stage to force the model to focus on image content rather than positional shortcuts, thereby eliminating positional bias; and introducing a differentiated lexical pruning strategy during the inference stage, setting independent visual lexical retention ratios for the support set and query set in the few-shot task, and dynamically removing redundant features at the visual encoder and decoder layers. This invention effectively solves the problems of positional dependence, insufficient learning, and high inference latency in few-shot classification of multimodal large models through task construction enhancement during the training stage and computational pruning during the inference stage, significantly improving the classification accuracy and deployment efficiency of the model.

Description

Technical Field

[0001] This invention relates to a multimodal large-model small-sample classification method based on task enhancement and efficient reasoning, belonging to the field of small-sample classification technology. Background Technology

[0002] With the rapid development of deep learning technology, artificial intelligence has achieved remarkable success in the field of computer vision. However, traditional deep learning models typically rely on large-scale labeled data for training, which poses a significant challenge in real-world applications where data is scarce or labeling costs are high. To address this issue, few-shot techniques have emerged, aiming to mimic human cognitive abilities by rapidly learning new concepts and achieving accurate classification using only a very small number of samples.

[0003] In recent years, large-scale vision-language models have emerged as a new paradigm for solving few-shot classification tasks due to their massive pre-trained knowledge and powerful cross-modal understanding capabilities. These models can handle multimodal inputs including images and text through context learning or instruction fine-tuning. While large multimodal models perform well on general tasks, existing technologies still face the following critical challenges when directly applied to specialized few-shot classification scenarios: First, existing large multimodal models generally suffer from severe positional bias. When handling N-way K-shot classification tasks, models typically need to select the correct category from a given list of candidate answers. However, research shows that large models often exhibit irrational preferences regarding the position of options (e.g., tending to choose the last option in the list) rather than judging based on the visual features of the image content. This inherent positional bias severely impairs the model's classification accuracy in few-shot tasks.

[0004] Secondly, large models face the challenge of "insufficient learning." On the one hand, large models rely excessively on prior knowledge memorized during the pre-training stage during inference, making it difficult to effectively extract new knowledge from the limited support samples provided by the current task, thus limiting generalization ability. On the other hand, images in real-world scenes often contain complex background information, and models easily capture spurious visual correlations between foreground objects and specific backgrounds (e.g., strongly associating "boat" with "water surface"), thereby ignoring the discriminative features of the foreground objects themselves. Furthermore, training tasks constructed using traditional random sampling methods are often too simple, failing to force the model to learn robust, fine-grained features.

[0005] Finally, existing multimodal few-shot classification paradigms suffer from low inference efficiency and high computational cost. To enable the model to understand new categories, the standard practice is to concatenate all support set images (e.g., the 25 images in a 5-way 5-shot task) with the query set images into an extremely long image-text sequence input to the model. Since the computational complexity of Transformer-based models increases quadratically with sequence length, this long input sequence leads to extremely high memory usage and inference latency, severely limiting the deployment and application of this technology on edge devices or real-time systems.

[0006] In summary, how to eliminate the positional bias of large models, improve the model's ability to acquire knowledge from few samples, and significantly reduce the computational cost of inference while ensuring accuracy are the technical challenges that urgently need to be solved in the field of multimodal large model small sample classification. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a multimodal large model small sample classification method based on task enhancement and efficient inference, which enhances the model at each stage of fine-tuning and inference, thereby significantly reducing inference latency while ensuring high-precision classification performance and improving the model's generalization ability and deployment efficiency in small sample scenarios.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A multimodal large-model small-sample classification method based on task enhancement and efficient inference includes the following steps: Step 1: Obtain a small sample image classification dataset and divide it into a set of base classes for training and a set of new classes for testing; Step 2: Use the multimodal large model to generate common attribute dimensions that describe the background context, and randomly combine the common attribute dimensions and assign weights to each category in the base class set to generate weighted background description text prompts; Step 3: Use the pre-trained foreground / background segmentation model to process each original image sample in the base class set and extract the binarized foreground mask of each original image sample. Step 4: Input the original image sample, the binarized foreground mask extracted in Step 3, and the weighted background description text prompt generated in Step 2 into the pre-trained text-based image restoration model to perform image restoration and construct the background-enhanced image corresponding to the original image sample. Step 5: Add the background enhancement image corresponding to the original image sample to the base class set to construct the enhanced base class dataset. Calculate the multimodal similarity between each category based on the enhanced base class dataset, and perform clustering based on the multimodal similarity to obtain neighbor samples of different difficulty for each category. Step 6: Construct an N-way K-shot meta-task fine-tuning instruction set based on the neighbor samples obtained in Step 5. During the construction process, perform position balancing on the candidate answer list of the meta-task, and perform instruction fine-tuning on the multimodal large model based on the fine-tuning instruction set. Step 7: Construct a small-sample classification task to be tested based on the new class set, and input it into the fine-tuned multimodal large model. During the inference process, a differentiated word pruning strategy is used to speed up the inference process, and the final classification result is output.

[0009] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: 1. This invention introduces a semantically guided background generation strategy, utilizing a multimodal large model to generate weighted background cues covering various general attributes (such as weather, time, and environment), and combines this with image inpainting techniques to generate enhanced samples with diverse backgrounds. This method effectively breaks the spurious statistical correlation between foreground objects and specific backgrounds, forcing the model to pay more attention to the discriminative features of the foreground objects themselves during the learning process, thus maintaining extremely high classification accuracy even when facing complex or unseen test scenarios.

[0010] 2. This invention does not employ traditional random sampling to construct the fine-tuning task. Instead, it proposes a challenging negative sample mining strategy based on multimodal similarity. By fusing visual and textual features to calculate similarity, and mining neighbor samples at different difficulty levels (hard, medium, and easy) to construct a meta-task, the challenge of the training task is artificially increased. This training method effectively prevents the model from taking shortcuts using simple features, prompting the model to learn more discriminative, fine-grained features, and significantly improving classification accuracy among similar categories.

[0011] 3. This invention addresses the inherent flaw in large models that tend to select specific positions (such as the last option) when answering multiple-choice questions. It introduces a positional balancing strategy during the instruction fine-tuning stage. By forcibly constraining the frequency of the correct answer appearing in each position of the candidate list to be strictly consistent, this invention cuts off the model's dependence on positional information at the training source, ensuring that the model performs logical reasoning solely based on image content and text instructions, thereby significantly improving the reliability of the evaluation results.

[0012] 4. This invention innovatively designs a differentiated lexical pruning strategy during the inference stage. Considering the characteristic that the support set only serves as a reference in small-sample tasks while the query set needs to be precisely identified, this invention sets differentiated retention ratios for the two, significantly compressing redundant visual information; simultaneously, combined with text-guided secondary pruning, visual blocks irrelevant to the question are further eliminated. This scheme significantly reduces the length of the input sequence and computational complexity with almost no loss of classification accuracy, greatly improving inference speed and making it more suitable for practical applications with high real-time requirements. Attached Figure Description

[0013] Figure 1 This is a flowchart of the multimodal large model small sample classification method based on task enhancement and efficient reasoning of the present invention; Figure 2 This is a schematic diagram of the model method of the present invention. Detailed Implementation

[0014] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0015] like Figure 1 and Figure 2 As shown, this invention proposes a multimodal large-model small-sample classification method based on task enhancement and efficient reasoning, including the following steps: Step 1: Obtain a small sample image classification dataset and divide it into a base class set for training and a new class set for testing; the specific process is as follows: Step 1.1 involves collecting image classification datasets covering different scenes and domains, including general object recognition, fine-grained image classification, scene recognition, and remote sensing image classification. Specific datasets include three general domain datasets: CIFAR-FS, Tiered-ImageNet, and Mini-ImageNet; and five fine-grained datasets: CUB, Flower102, FGVC, StanfordDogs, and StanfordCars, covering birds, flowers, airplanes, dogs, and vehicles respectively. Nine publicly available image datasets were also used: GTSRB, Country211, EurosatClip, Resisc45_CLIP, Caltech-101, OxfordPets, Food101, MNIST, and Fer2013. Step 1.2: Divide each collected dataset into categories. Specifically, randomly divide all categories in the dataset in a 7:3 ratio. Allocate 70% of each dataset to the base class set for building the training and fine-tuning task, and allocate the remaining 30% to the new class set for evaluating small sample classification performance. The base class set and the new class set are mutually exclusive.

[0016] Step 2: Utilize the multimodal large model to generate common attribute dimensions describing the background context, and randomly combine attribute dimensions and assign weights to categories in the base class set to generate weighted background description text; the specific process is as follows: Step 2.1: Use the multimodal large model (GPT 4) to generate five general attribute dimensions that describe the background context. The attribute dimensions include: man-made environment, natural environment, functionality, time, and weather. Step 2.2: For each category in the base class set, GPT-4 randomly selects a subset from the five general attribute dimensions and assigns different weight values ​​to each attribute in the subset to clarify the hierarchical importance of each attribute in image synthesis. Finally, a complete weighted background description text is generated. The format is as follows: Current category: Automobile Randomly selected elements: artificial environment (1.4), time (1), weather (1.2) Weighted background description text: [(Mountain Road: 1.4), (Autumn: 1), (Sunny: 1.2)]; Step 2.3: Batch generate weighted background description text prompts for subsequent image background compositing.

[0017] Step 3: Process the samples in the base class set using the foreground / background segmentation model to extract the foreground mask of the original image; the specific process is as follows: Step 3.1: Load the pre-trained foreground / background segmentation model (BiRefNet). BiRefNet has been trained on a large-scale dataset and has the ability to extract salient objects in the image. Step 3.2: Traverse the image files in the base class dataset, read the images, and convert them to RGB format. Perform standard preprocessing operations on the input images, first resizing the images to... The image is then converted into tensors and normalized to fit the model's input requirements. Step 3.3: Input the preprocessed image tensor into the BiRefNet model for forward inference to obtain the output prediction value of the last layer of the model. Apply the Sigmoid activation function to the prediction value to map the pixel values ​​to the (0, 1) interval to obtain the probability map. Then, restore the probability map to the size of the original image; Step 3.4: Create a completely white subject layer and a completely black background layer. Using the probability map output by the model as the alpha channel, composite the white subject with the black background to obtain a high-precision foreground mask where the foreground area is white and the background area is black. .

[0018] Step 4: Input the original image, the foreground mask obtained in Step 3, and the weighted background description text generated in Step 2 into the text-based image model for image inpainting, and construct the enhanced base class dataset; the specific process is as follows: Step 4.1: Load the pre-trained stable diffusion inpainting model, which has the ability to modify specific regions of an image based on text prompts; Step 4.2: Use the original image from the base class set as the base image, and apply the binarized foreground mask generated in Step 3. As a constraint on the repair area, the weighted background description text generated in step 2 will be used as a prompt. As the guiding conditions for generation, all three are input into the text image restoration model; Step 4.3: Utilize the text-based image restoration model to perform the generation operation. While maintaining the pixel count of the object region covered by the foreground mask, redraw the background region of the image based on the weighted background description text prompts to obtain the background-enhanced image. The formula is as follows: , in, Represents the original image. This represents the binarized foreground mask generated in step 3. This represents the weighted background description text generated in step 2.

[0019] Step 5: Integrate the base class set described in Step 1 with the enhanced base class dataset described in Step 4. Calculate the multimodal similarity between each category based on the integrated dataset, and perform clustering based on the multimodal similarity to mine neighbor samples of different difficulty levels for each category. The specific process is as follows: Step 5.1: Merge the background enhancement image generated in Step 4 with the original image to construct a rich and diverse enhanced base class dataset. Step 5.2: Extract image and text features using a pre-trained multimodal encoder (OpenCLIP); First, based on the extracted image features, calculate any category. Visual prototype feature vector: Input all images under this category into the visual encoder to obtain feature embeddings, calculate the average of all feature embeddings and perform normalization. The specific calculation formula is as follows: , in, Indicate category The visual center features, For the collection of images in this category, For a single image eigenvectors, This is a normalization operation; Secondly, calculate any category Text feature vectors Enter the name of the category into a preset text template (e.g., "a photo of a [CLASS]"), input it into a text encoder to obtain feature embeddings, and normalize them, using the following formula:

[0020] in, Indicate category semantic features For text encoders; Step 5.3, calculate any two categories and Multimodal fusion similarity score between This score measures the combined visual and semantic distance between two categories, and is calculated using the following formula: , in, Represents the vector dot product; the first term The second term represents the similarity within a visual modality. For visual and textual cross-modal similarity; the third item Similarity within text modalities; These are the weighting coefficients for the three similarity factors mentioned above; Step 5.4, based on the calculated multimodal fusion similarity score Using the K-Means clustering algorithm to separate the target category All other categories are divided into three mutually exclusive sets: the hard negative sample set with the highest similarity to the target category, the medium negative sample set with intermediate similarity, and the simple negative sample set with the lowest similarity, thus completing the mining of neighbor samples.

[0021] Step 6: Construct an N-way K-shot instruction fine-tuning meta-task based on the neighbor samples mined in Step 5, and perform positional balancing on the candidate answer list of the meta-task during the construction process; the specific process is as follows: Step 6.1, for any target category in the enhanced base class dataset According to a preset sampling ratio (e.g., difficult:medium:easy = 2:1:1), negative samples from the difficult, medium, and easy difficulty groups extracted in step 5 are sampled. The negative sample classes, together with the target class, form the candidate class set for the current meta-task. ; Step 6.2, based on the candidate category set determined in Step 6.1 For each category in the set, random sampling is performed. The sampled images are used as support samples; the sampled support samples and their corresponding category names are arranged in an alternating "image-text" manner to construct a contextual example sequence; then, an image from the target category is stitched together. The query image to be categorized, along with a text question asking which category the image belongs to, and an appendix containing... All A list of candidate answers for each category name, formatted as follows: Image 1: <Image 1> What is this? {Difficulty Category 1} Image 2: <Image 2> What is this? {Simple Category 2} … Image N: <Image N> What is this? {Medium Category N} Search for image: <Search for image> What is this? Please select one answer from the list of candidate answers: [{Class Name 1}, {Class Name 2}, ..., {Class Name N}]; Step 6.3, in constructing the total When fine-tuning the instruction set for each meta-task, the candidate answer list for each task is subjected to a restricted random permutation; the specific constraint is: throughout the entire fine-tuning instruction set, ensure that the target category... The correct answer appears in the first to last section of the candidate answer list. The frequencies of each position are strictly equal, all being This forces the model to reason based on image content rather than the location of the options; Step 6.4: Based on the fine-tuning instruction set constructed above, LoRA (Low-Rank Adaptation) technology is used to fine-tune the pre-trained model. During the fine-tuning process, the original parameters of the pre-trained model are kept unchanged, and only the parameters of the additionally introduced low-rank matrix are trained and updated. This reduces the computational cost while enabling the model to efficiently align with the instruction intent and improve its performance in few-sample classification.

[0022] Step 7: Input the small-sample classification task to be tested into the fine-tuned multimodal large model. During the inference process, a differential lexical pruning strategy is used to speed up the inference process, and the final classification result is output. The specific process is as follows: Step 7.1, randomly select from the new class set There are several categories, and for each category, a random sample is taken. These images were used as the support set, and from this... One image is randomly selected from each category as the query set to construct a randomized test task that conforms to the N-way K-shot paradigm; Step 7.2: Format the random test task according to the fine-tuning instruction template in Step 6 to generate input data containing image sequences and text queries, and input it into the fine-tuned multimodal large model; Step 7.3: In a shallow layer (e.g., layer 8) of the multimodal large model visual encoder, calculate the attention score between the category term [CLS] and all image terms to measure the importance of the visual region; let... For the feature vector of category words, For length is Image word sequence, attention score The calculation formula is as follows: , in, and For the query and key projection matrix in the visual encoder, For the hidden layer dimension; according to Sort the image terms in descending order, and retain the first few images in the support set. The proportion of terms, for the query set image, retain the first part proportional lexical units (and satisfying) Unselected words are merged into their nearest selected words using a weighted average, thus updating the image word sequence. Step 7.4: In the interaction layer before the multimodal large model decoder layer, the updated image terms are further filtered using text query commands; assuming... This refers to the final word features in a text query command sequence. The text-image cross-attention score represents the image word sequence after the first stage of filtering. The calculation formula is as follows: , in, and This is the projection matrix for the cross-modal interaction layer. According to... Visual words that are less than a preset threshold of relevance to the text question are removed, and only the high-response visual words are input into the subsequent decoder layer to generate the final classification label text through autoregression.

[0023] To verify the effectiveness of the multimodal large-model small-sample classification method based on task enhancement and efficient reasoning proposed in this invention, a comparative experiment is conducted in this embodiment, as detailed below: Both model training and inference were performed on four NVIDIA GeForce RTX 3090 GPUs (24GB VRAM per card), with memory optimization utilizing DeepSpeed ​​ZeRO Stage 2 technology. Qwen2.5-VL-7B-Instruct was used as the multimodal base model. To achieve efficient parameter fine-tuning, low-rank adaptation (LoRA) was employed, freezing the visual encoder weights and training only the language model and the attention projection matrix in the multimodal projection layer.

[0024] To comprehensively evaluate the model's performance in different scenarios, this embodiment selects eight publicly available small-sample image classification benchmark datasets, covering areas such as general object classification and fine-grained image classification: General classification datasets: MiniImageNet (MINI), TieredImageNet (TIERED), CIFAR-FS (CIFAR); Fine-grained classification datasets: CUB-200-2011 (CUB), Stanford Dogs (Dogs), FGVC-Aircraft (FGVC), Oxford 102 Flowers (Flowers), Stanford Cars (Cars).

[0025] All datasets follow the standard "base class-new class" partitioning and are evaluated in a 5-way 1-shot setting.

[0026] The following evaluation indicators will be used for evaluation: Accuracy (Acc): Measured by the model's classification accuracy on new classes in the test set, expressed as a percentage (%). Computational complexity (FLOPs): Measures the number of floating-point operations during model inference to reflect the reduction in computational overhead (as a percentage relative to the unaccelerated version). Inference Time: Measures the total time required for the model to complete all test tasks, measured in minutes (min), and is used to evaluate inference speed.

[0027] The performance comparison between the proposed method and traditional methods on four general datasets and four fine-grained datasets is shown in Table 1 and Table 2, respectively.

[0028] Table 1. Performance comparison of the methods on four common datasets.

[0029] Table 2 compares the method performance on four fine-grained datasets.

[0030] This invention sets new accuracy records on representative datasets such as MINI and TIERED (e.g., 99.20% on MINI). It also performs exceptionally well on challenging tasks with minimal interclass differences, such as CUB birds and Stanford Dogs dogs (e.g., near 100% accuracy on the Cars and Flowers datasets). This demonstrates the method's strong generalization ability and robustness when dealing with diverse, cross-domain general visual concepts.

[0031] Table 3. Performance and efficiency comparison of the model after applying inference acceleration.

[0032] As shown in Table 3, the efficiency comparison demonstrates that after introducing a differentiated lexical pruning strategy, the number of floating-point operations (FLOPs) of the method in this invention is only 19.2% of that of the original model, meaning that the computational load is reduced by approximately 5 times. Simultaneously, the total inference time for completing the full test task is significantly reduced from 326.22 minutes to 257.42 minutes. Notably, despite the substantial reduction in computational resource consumption, the method in this invention maintains an average accuracy of 97.96% across eight datasets, showing only a slight decrease compared to the unaccelerated version, successfully achieving the optimal balance between model inference speed and classification accuracy.

[0033] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned multimodal large model small sample classification method based on task enhancement and efficient reasoning.

[0034] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned multimodal large model small sample classification method based on task enhancement and efficient reasoning.

[0035] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0036] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0037] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0038] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0039] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A multi-modal large model small sample classification method based on task enhancement and efficient inference, characterized in that, Includes the following steps: Step 1: Obtain a small sample image classification dataset and divide it into a set of base classes for training and a set of new classes for testing; Step 2: Use the multimodal large model to generate common attribute dimensions that describe the background context, and randomly combine the common attribute dimensions and assign weights to each category in the base class set to generate weighted background description text prompts; Step 3: Use the pre-trained foreground / background segmentation model to process each original image sample in the base class set and extract the binarized foreground mask of each original image sample. Step 4: Input the original image sample, the binarized foreground mask extracted in Step 3, and the weighted background description text prompt generated in Step 2 into the pre-trained text-based image restoration model to perform image restoration and construct the background-enhanced image corresponding to the original image sample. Step 5: Add the background enhancement image corresponding to the original image sample to the base class set to construct the enhanced base class dataset. Calculate the multimodal similarity between each category based on the enhanced base class dataset, and perform clustering based on the multimodal similarity to obtain neighbor samples of different difficulty for each category. Step 6: Construct an N-way K-shot meta-task fine-tuning instruction set based on the neighbor samples obtained in Step 5. During the construction process, perform position balancing on the candidate answer list of the meta-task, and perform instruction fine-tuning on the multimodal large model based on the fine-tuning instruction set. Step 7: Construct a small-sample classification task to be tested based on the new class set, and input it into the fine-tuned multimodal large model. During the inference process, a differentiated word pruning strategy is used to speed up the inference process, and the final classification result is output.

2. The multimodal large-model small-sample classification method based on task enhancement and efficient reasoning according to claim 1, characterized in that, In step 1, image classification datasets containing different scenes and domains are collected, including general object recognition, fine-grained image classification, scene recognition, and remote sensing image classification. According to the preset division ratio, all categories in the few-sample image classification dataset are randomly divided. 70% of each category is assigned to the base class set for building the training and fine-tuning task, and the remaining 30% is assigned to the new class set for evaluating the few-sample classification performance.

3. The task enhancement and efficient inference based multi-modal large model small sample classification method according to claim 1, characterized in that, In step 2, a multimodal large model is used to generate five general attribute dimensions describing the background context. These general attribute dimensions include: artificial environment, natural environment, image scene function, time, and weather. For each category in the base set of classes, a number of general attribute dimensions are randomly selected from the 5 general attribute dimensions as a subset, and different weight values are assigned to each general attribute dimension in the subset, thereby generating a weighted background description text prompt .

4. The task enhancement and efficient inference based multi-modal large model small sample classification method according to claim 1, characterized in that, In step 3, for each original image sample in the base class set, after converting the original image sample into RGB format, the image size is adjusted to pixels to adapt to the input requirements of the pre-trained foreground-background segmentation model, the image after size adjustment is converted into a tensor, and normalized processing is performed; The normalized image tensor is input into the pre-trained foreground and background segmentation model for forward inference. The output prediction value of the last layer of the pre-trained foreground and background segmentation model is obtained. The Sigmoid activation function is applied to the output prediction value to map the pixel value to the (0, 1) interval to obtain the probability map. The probability map is restored to the size of the original image sample. A full white main body layer and a full black background layer are created, a probability map restored to the original image sample size is used as an Alpha channel, the main body layer and the background layer are synthesized, and a foreground area is white and a background area is black, thereby obtaining a binary foreground mask .

5. The task enhancement and efficient inference based multi-modal large model small sample classification method according to claim 1, characterized in that, In step 4, the original image samples in the base class set are used as the base image, and the binarized foreground mask extracted in step 3 is used as the base image. As a constraint on the repair area, the weighted background description text generated in step 2 will be used as a prompt. As the guiding conditions for generation, the three are jointly input into the pre-trained text-based image restoration model; By using a text-based image restoration model to perform generation operations, while keeping the pixel count of the object region covered by the foreground mask unchanged, the background region of the original image sample is redrawn based on weighted background description text prompts, resulting in a background-enhanced image. The formula is as follows: ， wherein, denotes the original image sample, denotes a text-to-image inpainting model for generating new pixel content within the mask region according to the weighted background description of the text prompt.

6. The task enhancement and efficient inference based multi-modal large model small sample classification method according to claim 1, characterized in that, In step 5, a pre-trained multimodal encoder is used to extract image features and text features for each image in the enhanced base class dataset; For the augmented base class dataset, the visual center feature and semantic feature of each category are calculated using the following formula: ， ， in, Representing categories Visual center features and semantic features; For category The image set includes original image samples and background enhancement images corresponding to the original image samples; for The number of images in For visual encoders, For image eigenvectors, For text encoders, This indicates a preset text template. This is a normalization operation; Computing categories Similarity scores for multimodal fusion between all other categories, formula as follows: ， in, Indicate category With category Multimodal fusion similarity score between them Represents the vector dot product; the first term The second term represents the similarity within a visual modality. For visual and textual cross-modal similarity; the third item Similarity within text modalities; Representing categories Visual center features and semantic features These are the weight coefficients for the three similarity factors mentioned above; Based on the calculated multimodal fusion similarity score Using clustering algorithms to separate categories All other categories are divided into three mutually exclusive sets: with category The categories are obtained by dividing the data into three sets: the most similar difficult negative samples, the moderately similar medium negative samples, and the least similar simple negative samples. Neighbor samples of varying difficulty.

7. The task enhancement and efficient inference based multi-modal large model small sample classification method according to claim 6, characterized in that, In step 6, a total is constructed. A set of fine-tuning instructions for each meta-task, targeting any class in the enhanced base class dataset. According to the preset sampling ratio, co-sample the negative samples obtained in step 5 from the difficult, moderate, and easy sample sets. Each category, and category Together they form the candidate category set for the current meta-task ; Based on candidate category set For each category in the set, random sampling is performed. The sampled images are used as support samples; the sampled support samples and their corresponding category names are arranged in an image-text format to construct a contextual example sequence; a sequence of images from the category is then concatenated. The query image to be categorized and a text question asking for the category of that image are provided, with an appendix containing... All A list of candidate answers for each category name, wherein the correct answer to the image to be categorized appears in the first to last category of the candidate answer list. The frequencies of each position are strictly equal, all being Second-rate, for Multiples of positive integers; Based on the fine-tuning instruction set constructed above, low-rank adaptive technology is used to fine-tune the multimodal large model to obtain the fine-tuned multimodal large model.

8. The multimodal large-model small-sample classification method based on task enhancement and efficient reasoning according to claim 7, characterized in that, In step 7, random selection is performed from the new class set. There are 10 categories, and each category is randomly selected. These images were used as the support set, and from this... A random image is randomly selected from each category as the query set to construct a randomized test task that conforms to the N-way K-shot paradigm. The random test task is formatted according to the meta-task format described in step 6 to generate input data containing image sequences and text queries, which is then input into the fine-tuned multimodal large model. In the shallow layer of the visual encoder of the fine-tuned multimodal large model, the attention score between the category term and the image term sequence of each image in the support set is calculated, and the attention score between the category term and the image term sequence of the images in the query set is also calculated. The attention score formula is as follows: ， in, The attention score matrix, As category morphemes, For length is Image word sequence, and For the query and key projection matrix in the visual encoder, For the hidden layer dimension, This represents the normalized exponential function; according to Sort the image term sequences of each image in the support set and query set in descending order, and select the first few images from the support set. Proportional terms, for the image selection of the query set The proportion of lexical units, and satisfying Unselected words are merged into their nearest selected words using a weighted average method, thus updating the image word sequence for each image. In the interaction layer preceding the multimodal large model decoder layer, the updated image word sequence is further filtered using text query instructions from a random testing task, as shown in the following formula: ， in, For text-image cross-attention scores, This refers to the final word features in a text query command sequence. For the updated image word sequence, and The projection matrix of the cross-modal interaction layer; according to Image words with relevance to the text problem below a preset threshold are removed. The remaining image words are then input into the subsequent decoder layer to generate the final classification label text through autoregression.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal large model small sample classification method based on task enhancement and efficient reasoning as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal large model small sample classification method based on task enhancement and efficient reasoning as described in any one of claims 1 to 8.

Images