A method and system for cross-lingual few-shot learning based on dynamic adaptive routing

Abstract

The application discloses a kind of based on dynamic adaptive routing's picture-text mutual benefit small sample classification method and system, it is related to computer vision, natural language processing and multimodal machine learning technical field.The application proposes an innovative technical solution, by designing a kind of task and sample perception dynamic routing mechanism, effectively solve the problem that the guiding strategy in the prior art picture-text mutual benefit guiding method is static, rigid, cannot be adaptively adjusted according to specific task and sample.The mechanism is analyzed task characteristics and sample state in real time by context perception encoder, and utilizes dynamic routing network to generate adaptive path weight, guiding intensity and other control parameters, realize the leap from "static mutual benefit" to "dynamic adaptive mutual benefit".In addition, the application creatively injects the above dynamic parameters into the differentiable text-guided image and image-guided text dual-path calculation, realizes the real-time, accurate regulation and control of multimodal fusion strategy, significantly improves the classification accuracy, generalization ability and robustness of the model in open, complex scene under the condition of not training or only slightly fine-tuning a small number of parameters, the application framework maintains the high efficiency of training-free method, and has good compatibility and generalization to existing static method.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, natural language processing and multimodal machine learning, and specifically relates to a method for few-shot image classification, especially a method and system for image-text reciprocal few-shot classification based on dynamic adaptive routing. Background Technology

[0002] Contrastive language-image pre-trained models (such as CLIP) have become an important foundation in the field of few-shot learning due to their powerful cross-modal alignment capabilities and zero-shot generalization. To improve performance while avoiding time-consuming model fine-tuning, training-free few-shot learning methods have emerged, such as Tip-Adapter and GDA-CLIP. These methods typically utilize information from both image and text modalities independently for classification, ignoring the deep complementarity between modalities.

[0003] Recent research (such as TIMO) has recognized the importance of cross-modal interaction and proposed a text-image reciprocal guidance optimization framework. This framework addresses the issues of anomalous matching in image modalities and inconsistent quality of text prompts through two components: "text-guided image" and "image-guided text," significantly improving the performance of training free few-shot classification. However, existing reciprocal guidance methods have a core limitation: their guidance strategies are static. Specifically, key parameters such as the fusion weights between image and text modalities and the guidance strength are typically determined using a grid search on the validation set. This "one-size-fits-all" strategy cannot adapt to the inherent differences in different tasks (such as inter-class similarity and task complexity) and the characteristics of the samples themselves (such as image clarity and the accuracy of text descriptions). For example, in fine-grained classification tasks with small inter-class differences and difficult visual discrimination, stronger textual semantic guidance may be needed; while for tasks with clear images and distinct categories, static strong guidance may introduce textual noise. This static strategy limits the model's optimal adaptability and robustness in diverse and open scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for small-sample classification based on dynamic adaptive routing and text-image reciprocity. This invention aims to solve the problem that existing text-image reciprocity guidance methods have static and rigid guidance strategies that cannot adaptively adjust according to specific tasks and samples, thereby achieving more accurate and robust small-sample classification.

[0005] To achieve the above objectives, this invention proposes a task- and sample-aware dynamic routing mechanism in the first aspect. The core of this mechanism is the introduction of a lightweight context-aware encoder and a dynamic routing network. The context-aware encoder analyzes the characteristics of the current few-sample task (based on the support set) and the state of the query sample in real time, generating a condensed task-sample context vector. The dynamic routing network receives this vector and dynamically generates a set of control parameters, including: (1) Path selection weights: determining the contribution ratio of the two paths, image-guided text and text-guided image, in real time; (2) Adaptive guidance strength: dynamically adjusting the number of text features incorporated into the image classifier and the constraint strength of text prompt word fusion, replacing fixed hyperparameters; (3) Feature modulation signal: performing a lightweight conditional transformation on the input multimodal features to make them more adaptable to the current routing decision.

[0006] A second aspect of this invention provides an end-to-end differentiable reciprocal guidance architecture. This architecture seamlessly integrates the aforementioned dynamic routing mechanism with two core components: Text-Guided Image (TGI) and Image-Guided Text (IGT), forming a complete, differentiable dataflow graph. Dynamically generated parameters are injected into the computation of TGI and IGT, enabling the entire system to achieve adaptive inference from task sample input to final classification output via forward propagation. Furthermore, this architecture supports two modes: in "pure inference" mode, the dynamic routing network can use predefined or heuristically initialized parameters, maintaining the training-free characteristic; in "lightweight fine-tuning" mode, the CLIP backbone network can be fixed, and only a small number of meta-tasks are used to efficiently fine-tune the context-aware encoder and dynamic routing network to learn better dynamic decision-making strategies. This process requires far less computation and data than full model training.

[0007] The beneficial effects of this invention are as follows:

[0008] From static to dynamic, precise adaptation: This invention breaks through the limitations of existing methods that rely on fixed guidance strategies. It can adjust the multimodal fusion strategy in real time according to contextual information such as task difficulty and sample characteristics, achieving true "task-specific" and "sample-specific" adaptation, thereby improving the model's accuracy and scene adaptability.

[0009] Significantly enhanced generalization and robustness: The dynamic routing mechanism enables the model to adaptively adjust its dependence on each modality when facing challenges such as out-of-distribution data, noise interference, and fine-grained classification. For example, when the query image is blurry, the system can automatically increase its dependence on the text modality, thus exhibiting stronger robustness and generalization performance.

[0010] Maintaining high efficiency and low overhead: The newly added context-aware encoder and dynamic routing network structure are lightweight with very few parameters, adding only negligible computational overhead in forward inference. The main body of the system is still based on the frozen pre-trained model, perfectly inheriting the high efficiency advantage of the free training method. In the lightweight fine-tuning mode, only a very small number of parameters need to be optimized, resulting in extremely high training efficiency.

[0011] Flexible architecture and strong compatibility: The dynamic routing framework proposed in this invention is a generalization and enhancement of existing reciprocal routing methods (such as TIMO). When the dynamic network outputs constant parameters, this system degenerates into the original static method. Therefore, this invention has good backward compatibility and can be easily extended to more modal or more complex routing topologies. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall architecture of the image-text reciprocal small sample classification system based on dynamic adaptive routing provided in an embodiment of the present invention. Detailed Implementation

[0013] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] Example 1: System Overall Architecture and Workflow

[0015] like Figure 1 As shown, the system proposed in this invention includes the following core modules: a multimodal feature extractor, a task-sample perceptron, a dynamic routing network, a differentiable text-guided image module, a differentiable image-guided text module, and an adaptive fusion module.

[0016] Given a few-sample classification task, including a support set of images, a set of corresponding category text hints, and a query image, the execution steps of this invention are as follows:

[0017] Step 1: Feature Extraction. Using the frozen CLIP model, extract support set image features, support set text features, and query image features. Where N is the number of classes, K is the number of samples per class (shot), P is the number of text prompts per class, and D is the feature dimension.

[0018] Step 2: Context-Aware Encoding. First, calculate the class prototypes for the support set image features. Next, calculate task-level statistics, such as the within-class feature variance (representing within-class consistency) and the mean and variance of the between-class prototype distances (representing task discrimination difficulty). Simultaneously, calculate sample-level statistics, i.e., calculate the cosine similarity between the query features and each class prototype, obtaining a similarity distribution vector to represent the classification ambiguity of the query samples. Finally, concatenate the above task-level and sample-level statistics and encode them into a context vector using a lightweight multilayer perceptron task-sample perceptron.

[0019] Step 3: Dynamic Parameter Generation. The context vector is input into the dynamic routing network, which consists of another lightweight multilayer perceptron, used to generate the dynamic parameter set: Among them, the path weight is normalized by the softmax function and used as the path weight for image-text reciprocity guidance; it is scaled to a preset range by the sigmoid function and used as the adaptive guidance intensity parameter; and it is the bias vector used to modulate image and text features.

[0020] Step 4: Adaptive Reciprocity-Guided Computation. This step comprises two parallel computational sub-processes:

[0021] Step 4.1: Dynamic Text-Guided Image Calculation. First, image features are adjusted using dynamically generated modulation parameters. Then, the similarity weights between the text prompt and the original image are calculated. Next, the index corresponding to the most important text feature is selected based on the dynamic parameters. Finally, dynamic text-guided image features are constructed, and a classifier is built based on these features.

[0022] Step 4.2: Dynamic Image-Guided Text Calculation. First, text features are modulated. Then, guided by the image prototype and combined with dynamic constraint strength parameters, an optimization problem is solved to calculate the fusion weights of the text prompts. Afterward, the enhanced text prototype features are obtained, and a classifier is built based on them.

[0023] Step 5: Adaptive Fusion and Prediction. Using the path weights generated by the dynamic adaptive routing network, the outputs of the two sub-classifiers are weighted and fused to calculate the final logical value of the query sample. Finally, the category with the largest logical value is selected as the prediction result.

[0024] Example 2: Meta-learning training strategy for dynamic adaptive routing networks

[0025] To enable the dynamic adaptive routing network to learn to make optimal decisions, a meta-learning strategy can be used for lightweight fine-tuning. The specific steps are as follows:

[0026] Step 1: Prepare a meta-training set containing a large number of heterogeneous image classification tasks.

[0027] Step 2: Freeze the intrinsic computation graphs in the CLIP feature extractor, the differentiable text-guided image module, and the differentiable image-guided text module.

[0028] Step 3: Set the parameters of the task-sample perceptron and the dynamic routing network to a trainable state.

[0029] Step 4: Employ contextual training. In each training iteration, sample a batch of meta-tasks from the meta-training set.

[0030] Step 5: For each sampled meta-task, use its support set and query set to execute steps 1 to 5 as described in Example 1 to complete one forward propagation.

[0031] Step 6: Calculate the classification loss (using cross-entropy loss) for all tasks in this batch on their respective query sets, and sum these losses as the meta-loss for this iteration.

[0032] Step 7: Update only the parameters of the task-sample perceptron and the dynamic routing network using the gradient backpropagation algorithm. The parameters of the CLIP backbone and other modules remain unchanged.

[0033] This training process enables the dynamic routing network to learn and generate optimal dynamic parameters that minimize the query set classification error based on the input task-sample context, thus mastering the decision-making ability of "when and how" to guide text-image reciprocity.

[0034] Example 3: Comparison with existing methods and advantages

[0035] Using the static TIMO method as a baseline, comparative experiments were conducted on a fine-grained bird classification dataset. The task difficulty was set to "high" (high inter-class similarity). Experimental results show:

[0036] Step 1: TIMO (Static Parameters) test. Because its image-text fusion weights and guidance strength are fixed values, it may not be able to fully meet the needs of high-difficulty, fine-grained tasks, resulting in limited classification accuracy.

[0037] Step 2: Testing of this invention (dynamic routing, pure inference mode). Even without additional training, the dynamic routing network can automatically generate larger sum parameters based on the challenging signals output by the context-aware sensor. This indicates that when faced with difficult tasks, the system's decision-making relies more heavily on textual semantic information for finer discrimination, resulting in a significant improvement in classification accuracy compared to static methods.

[0038] Step 3: Testing of the Invention (Dynamic Routing, Lightweight Fine-tuning). After fine-tuning with the meta-learning strategy, the decision-making mechanism of the dynamic routing network is further optimized and becomes more accurate. Comprehensive evaluation on multiple datasets covering various difficulty levels shows that its average classification accuracy comprehensively surpasses the static method TIMO. Especially in generalization tests on out-of-distribution data, the adaptive advantage of the invention is more pronounced, demonstrating stronger robustness.

[0039] It should be noted that although the embodiments described above are illustrative, they are not intended to limit the invention. Therefore, the invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A graph-text reciprocal few-shot learning method and system based on dynamic adaptive routing, characterized in that, include: By freezing the pre-trained multimodal encoder, multimodal features of support set images, support set text prompts, and query images are extracted from few-sample tasks. Based on the extracted features, context-aware encoding is performed to generate a context vector that reflects the characteristics of the current task and the state of the query sample. The context vector is input into the dynamic routing network to dynamically generate a set of control parameters, which include at least the path weight of the graph-text reciprocity guidance and the adaptive guidance strength parameter. Using the dynamic parameters, the calculation processes of text-guided image paths and image-guided text paths are controlled respectively to obtain the first classification result and the second classification result; Based on the path weights in the dynamic parameters, the first classification result and the second classification result are weighted and fused to obtain the final classification result of the query image.

2. The graph-text reciprocal few-shot learning method and system based on dynamic adaptive routing according to claim 1, characterized in that, The context-aware encoding specifically includes: calculating image class prototypes and task-level statistics based on support set image features; calculating sample-level statistics based on query image features and the image class prototypes; and concatenating and encoding the task-level statistics and sample-level statistics to generate the context vector.

3. The graph-text reciprocal few-shot learning method and system based on dynamic adaptive routing according to claim 1, characterized in that, The dynamic parameter set Φ generated by the dynamic routing network dyn Includes: the weight α of the text-guided image path tgi The weight α of the image-guided text path igt The adaptive strength parameter β controls the number of text feature selections, the adaptive strength parameter γ controls the text prototype fusion constraint, and the parameter Δ is used to modulate image features. v And the parameter Δ used to modulate text features. t .

4. The graph-text reciprocal few-shot learning method and system based on dynamic adaptive routing according to claim 1, characterized in that, The calculation process of the text-guided image path includes: using dynamic parameter Δ v The image features of the support set are modulated; important features are selected from the text features of the support set based on the dynamic parameter β; the modulated image features are concatenated with the selected text features to construct a first classifier and obtain the first classification result. The calculation process of the image-guided text path includes: using the dynamic parameter Δ t The text features of the support set are modulated; guided by the image class prototype, the optimized fusion weight of the text features is calculated in combination with the dynamic parameter γ; the modulated text features are fused using the optimized fusion weight to obtain the enhanced text prototype, and a second classifier is constructed to obtain the second classification result.

5. A system for implementing the method of claim 1, characterized in that, include: The multimodal feature extraction module is used to extract multimodal features from support set images, support set text prompts, and query images in small sample tasks; Task-sample perceptron, used to generate context vectors based on extracted features; A dynamic routing network is used to generate dynamic control parameters based on the context vector; Differentiable text-guided image module, used to perform calculation of text-guided image paths; The differentiable image-guided text module is used to perform the calculation of the image-guided text path; The adaptive fusion module is used to weight and fuse the outputs of the two paths to obtain the final classification result.

Images