Two-stage fine-tuning and decoupled inference method and apparatus for visual language models

Abstract

The present application relates to the technical field of artificial intelligence, and proposes a two-stage fine-tuning and decoupling reasoning method and device of a visual language model, which comprises: in the first stage, panoramic view and subject view are obtained through center cropping, and a learnable prompt vector is initialized for each view, the encoder is frozen, and the prompt vector is optimized to decouple view-specific semantics; in the second stage, the prompt vector is frozen, the text embedding stack is stacked as a trainable weight matrix, the linear classifier is optimized to enhance the discriminability of the base class, and the optimal fusion weight is searched on the validation set; the decoupling reasoning of the third stage comprises fusing the logits of the two views using the optimal weight fusion for the base class, adopting Dempster-Shafer evidence theory fusion for new classes, and introducing an uncertainty measure to realize robust prediction. The present application effectively balances the recognition performance of the base class and the generalization ability of the new class, and exhibits superior accuracy and robustness in a multi-modal classification task.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a two-stage fine-tuning and decoupled reasoning method and apparatus for a visual language model. Background Technology

[0002] Existing visual language pre-training models map images and text to a unified semantic feature space through multimodal contrastive learning, achieving powerful zero-shot transfer capabilities. For example, the CLIP (Contrastive Language-Image Pre-training) model adopts a typical dual-tower architecture, containing an image encoder and a text encoder to extract visual and text features respectively, and calculates classification probabilities using cosine similarity.

[0003] In downstream task fine-tuning, to preserve the general knowledge of the pre-trained model and prevent catastrophic forgetting, existing techniques propose the Prompt Learning method. This method keeps the parameters of the image encoder and text encoder completely frozen and introduces only a set of learnable continuous vectors as prompts. Depending on the insertion position, prompts can be divided into text prompts and visual prompts. By optimizing the prompt parameters, the model can be adapted to a specific classification task.

[0004] However, existing fine-tuning methods based on cue learning still have the following significant technical drawbacks in practical applications:

[0005] (1) The "double-edged sword" effect of visual context cannot be effectively handled (context-dependent paradox): Visual language models have difficulty distinguishing between semantic cues and interference biases in visual context during reasoning. On the one hand, context (e.g., "runway" background) is a key cue for disambiguation and helps identify "airplane"; on the other hand, models are prone to over-reliance on context to establish false associations (e.g., misclassifying objects that only appear in a specific background). Existing technologies usually use background suppression or causal reasoning to treat context as interference factors to be eliminated. This "one-size-fits-all" approach ignores the positive role of context, resulting in the model being unable to take into account both context-dependent and context-independent samples.

[0006] (2) There is a fundamental conflict between base class learning and new class generalization: Existing methods handle task-specific learning and general ability preservation at the same time by optimizing a single set of cue parameters, resulting in a serious performance trade-off: When the model is optimized for the base class (the class that is visible in training) to improve the discriminative power, it often destroys the original feature space of the pre-trained model, resulting in a significant decrease in the zero-sample generalization ability for the new class (the class that has not been seen).

[0007] Therefore, there is an urgent need to propose a fine-tuning and decoupled reasoning method for visual language models that can effectively decouple the semantic information of different views and obtain optimal decision fusion on the base class while maintaining the robust generalization ability of the new class. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a two-stage fine-tuning and decoupled reasoning method and apparatus for visual language models. Through view-specific cue learning, base class optimal decision synthesis, and decoupled reasoning strategies, it achieves the optimal balance between the discriminative power of the basic category and the generalization ability of the new category.

[0009] This invention provides a two-stage fine-tuning and decoupled inference method for a visual language model, comprising: The first phase involves learning view-specific cues, including: Obtain different views of the input image, including a panoramic view and a subject view; Initialize a learnable cue vector for each view separately; Input the cue vectors and category text of each view into a frozen text encoder to obtain the text features of each view; input the images of each view into a frozen image encoder to obtain the image features of each view; With classification as the objective, a cross-entropy classification loss function is designed to optimize the learnable cue vectors corresponding to each view in parallel, thereby obtaining the trained view-specific cue templates. The second stage involves synthesizing the optimal decision for the base class, including: Freeze the post-training view-specific cue template; A trainable base class classifier is constructed using the text features of each view, and the base class classifier is optimized using the image features of each view to obtain a view-specific base class discrimination model. The optimal fusion weights for fusing view classification results are determined by searching on the base class validation set; The third stage involves decoupling reasoning, including: Obtain the classification results of the image to be classified in both panoramic and subject views; If the true category of the image to be classified belongs to the base category set, then the classification results of the two views are weighted and fused using the optimal fusion weight, and the predicted category is determined based on the fusion result. If the true category of the image to be classified belongs to the new category set, then the evidence theory is used to perform uncertainty fusion on the classification results of the two views, and the predicted category is determined based on the decision probability after fusion.

[0010] On the other hand, the present invention protects a two-stage fine-tuning and decoupled inference device for a visual language model, comprising: The first module is used for learning view-specific cues, including: Obtain different views of the input image, including a panoramic view and a subject view; Initialize a learnable cue vector for each view separately; Input the cue vectors and category text of each view into a frozen text encoder to obtain the text features of each view; input the images of each view into a frozen image encoder to obtain the image features of each view; With classification as the objective, a cross-entropy classification loss function is designed to optimize the learnable cue vectors corresponding to each view in parallel, thereby obtaining the trained view-specific cue templates. The second module is used for base class optimal decision synthesis, including: Freeze the post-training view-specific cue template; A trainable base class classifier is constructed using the text features of each view, and the base class classifier is optimized using the image features of each view to obtain a view-specific base class discrimination model. The optimal fusion weights for fusing view classification results are determined by searching on the base class validation set; The third module is used for decoupled reasoning, including: Obtain the classification results of the image to be classified in both panoramic and subject views; If the true category of the image to be classified belongs to the base category set, then the classification results of the two views are weighted and fused using the optimal fusion weight, and the predicted category is determined based on the fusion result. If the true category of the image to be classified belongs to the new category set, then the evidence theory is used to perform uncertainty fusion on the classification results of the two views, and the predicted category is determined based on the decision probability after fusion.

[0011] Compared with the prior art, the present invention achieves the following beneficial effects: 1. By decoupling the panoramic view and the subject view, and learning view-specific cue templates respectively, the model can capture both global context and local subject details simultaneously, avoiding the bias of a single view and thus effectively handling the double-edged sword effect of context.

[0012] 2. In the second stage, the invention fine-tunes the base classifier based on the frozen prompts and searches for the optimal fusion weights on the validation set to ensure the base class recognition performance. In the third stage, the invention adopts weighted fusion for the base class and evidence theory fusion for the new class, which realizes differentiated reasoning for different class types. This not only improves the accuracy of the base class, but also maintains the zero-sample generalization ability of the new class, thus balancing the discriminative power of the base class and the generalization ability of the new class.

[0013] 3. This invention introduces the Dempster-Shafer evidence theory into the new type of reasoning, explicitly models and predicts uncertainty through entropy measurement, and automatically suppresses conflicting evidence during multi-view fusion, thereby enhancing the robustness of the model in open-world scenarios. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the steps of a two-stage fine-tuning and decoupling inference method for a visual language model in one embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the overall framework and workflow of a two-stage fine-tuning and decoupling inference device for a visual language model in one embodiment of the present invention. It is the fusion weight. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] In one embodiment, reference is made to Figure 1 As shown, a two-stage fine-tuning and decoupled inference method for a visual language model is provided, including: The first phase involves learning view-specific cues, including: Obtain different views of the input image, including a panoramic view and a subject view; Initialize a learnable cue vector for each view separately; Input the cue vectors and category text of each view into a frozen text encoder to obtain the text features of each view; input the images of each view into a frozen image encoder to obtain the image features of each view; With classification as the objective, a cross-entropy classification loss function is designed to optimize the learnable cue vectors corresponding to each view in parallel, thereby obtaining the trained view-specific cue templates. The second stage involves synthesizing the optimal decision for the base class, including: Freeze the post-training view-specific cue template; A trainable base class classifier is constructed using the text features of each view, and the base class classifier is optimized using the image features of each view to obtain a view-specific base class discrimination model. The optimal fusion weights for fusing view classification results are determined by searching on the base class validation set; The third stage involves decoupling reasoning, including: Obtain the classification results of the image to be classified in both panoramic and subject views; If the true category of the image to be classified belongs to the base category set, then the classification results of the two views are weighted and fused using the optimal fusion weight, and the predicted category is determined based on the fusion result. If the true category of the image to be classified belongs to the new category set, then the evidence theory is used to perform uncertainty fusion on the classification results of the two views, and the predicted category is determined based on the decision probability after fusion.

[0018] Specifically, in the first phase, view-specific cue learning is conducted.

[0019] To capture view-specific semantics, a parallel framework was constructed in the first stage to align physically decoupled visual inputs (obtained from the main view) with independently optimized learnable vectors (learned through independent cues).

[0020] (1) Obtain different views of the input image, including panoramic view and subject view.

[0021] For the input image of the visual language model First, make a copy of the original input image as the panoramic view. Then, the central region of the input image is cropped out as the main view. The cropping ratio (i.e., the ratio of the cropped side length to the original image side length) is set to 0.2~0.8, while maintaining the original image aspect ratio. The above center cropping operation does not rely on any prior segmentation model or label, and can obtain physically decoupled view pairs, thereby avoiding the introduction of additional noise.

[0022] If the cropping ratio is too low, for example less than 0.2, resulting in a cropped area smaller than 4% of the original image area, the main subject area will be too small. After resizing to the standard input size by the network, pixel interpolation will be severe, potentially leading to the loss of discriminative structural information. However, extremely low ratios can be used to explore extreme subject-focusing scenarios, such as when the target object is small and the background noise is high, which may help suppress background interference.

[0023] If the cropping ratio is too high, for example, exceeding 0.8, resulting in the cropped area exceeding 64% of the original image area, the overlap between the main view and the panoramic view increases significantly, reducing the diversity between views and potentially leading to a decrease in subsequent fusion gain. However, a higher ratio can preserve more context and is suitable for tasks where the target and background are strongly correlated (such as scene recognition).

[0024] Therefore, setting the cropping ratio within a relatively wide range of 0.2 to 0.8 allows for flexible adjustment based on specific tasks and data characteristics: a lower ratio can enhance subject decoupling, while a higher ratio can preserve contextual information, achieving a continuous transition from a "pure subject" to a "near-panoramic view." In practical applications, the optimal ratio can be searched through validation set performance, or multiple ratios can be used simultaneously within a multi-scale fusion framework.

[0025] In a preferred embodiment, the cropping ratio is set to 0.4, which aims to balance subject integrity (usually centered objects are fully preserved) and adequate context in most natural images. This allows for robust and superior performance in general classification tasks without requiring hyperparameter search for specific datasets, and can be used as a default baseline setting.

[0026] (2) Initialize learnable cue vectors for each view to obtain cue vectors for each view.

[0027] In one embodiment, the cue length of the cue vector The dimension is set to 16, with each cue vector having the same dimensions as the embedding dimension of the text encoder, for example, 512 dimensions. The cue vectors are randomly initialized and updated during training.

[0028] (3) Perform feature extraction: input the cue vector and category text of each view into the frozen text encoder to obtain the text features of each view; input the image of each view into the frozen image encoder to obtain the image features of each view.

[0029] During parallel optimization, the image encoder and text encoder are kept in a frozen state, and only the learnable cue vectors are updated.

[0030] Text branch: The cue vector is concatenated with the category text and then input into the frozen text encoder. For example, for the category "dog", the panoramic view... and main view The corresponding input sequences are respectively and The text embeddings were obtained respectively. and .in, and These are the text tooltips for the text editor in the panoramic view and the main view, respectively.

[0031] Image branch: Panoramic view and main view Input the frozen image encoder separately to obtain the visual embedding. and .here and Simultaneously, it is inserted as a visual cue vector into the input sequence of the image encoder (e.g., concatenated before image patch embedding) to achieve visual modality cue learning.

[0032] (4) Taking the classification task as the goal, design the cross-entropy classification loss function and optimize the learnable prompt vectors corresponding to each view in parallel to obtain the trained view-specific prompt templates.

[0033] Design a cross-entropy classification loss function: ; in, For view Learnable cue vectors, For view Image input, For view Visual language models predict the true category The probability of; Indicates a panoramic view. This represents the main view.

[0034] The optimization objective in the parallel optimization process is: ; Only update the cue vector during training and The image encoder and text encoder are frozen throughout the process, and the resulting cue vector is used as the trained view-specific cue template.

[0035] In one embodiment, the AdamW optimizer is used to optimize the learnable cue vectors corresponding to each view in parallel, with a learning rate of 0.001, training for 10 epochs, and a batch size of 32.

[0036] In the second stage, the optimal decision synthesis for the base class is performed, specifically including: (1) Freeze the view-specific prompt template after training. Specifically, after the first stage of training is completed, fix the template. and The parameters will no longer be updated.

[0037] (2) Construct a trainable base class classifier using the text features of each view, and optimize the base class classifier using the image features of each view to obtain a view-specific base class discrimination model.

[0038] In one embodiment, the trainable base classifier is a linear classifier.

[0039] Constructing a linear classifier: for the base class set (e.g., all categories in the training set), text embeddings for each category are generated using the text encoders of the two views respectively; and a trainable weight matrix is ​​formed by stacking the view-specific text embeddings. The initialization method of the weight matrix is ​​expressed as follows: ; in, Represents a view The initial values ​​of the weight matrix; Stack represents the stacking operation; Represents a view The categories of text encoder output Text embedding, Represents a view The text prompt vector; the dimension of each weight matrix is... , The total number of base class categories. For feature dimensions.

[0040] Fine-tuning the classification: When classifying using the trained linear classifier, the visual embedding of each view is multiplied by the corresponding weight matrix, and a temperature coefficient is added. Zoom in to get the view The classification of logits: ; in, Indicates the transpose of a vector or matrix; It is a view Transpose of the weight matrix optimized using image features; Represents a view Visual cue vectors.

[0041] Utilizing image features (input linear classifier), and optimizing the weight matrix by minimizing the cross-entropy classification loss function. (include and At this point, all prompts and encoders are frozen, and only the weight matrix is ​​updated.

[0042] In one embodiment, the second phase of training lasts for 5 epochs with a learning rate of 0.01 and a temperature coefficient. Set it to 0.01.

[0043] (3) Search the base class validation set to determine the optimal fusion weights for fusing the view classification results, including: Define fusion weights Used for convex interpolation The fused classification results are obtained; among them, The logits represent the categories of the panoramic view. The logits represent the categories of the main view. Represents the fused classification logits; In the base class validation set The fusion weights are discretized using a preset step size. search space ; For search space Fusion weights for each candidate Calculate the classification accuracy after fusion; Choose the one with the highest classification accuracy. As the optimal fusion weight .

[0044] In one embodiment, the preset step size is 0.1, and the search space of the fusion weights... .

[0045] The aforementioned first and second stages embody the "two-stage fine-tuning" in the method of this invention. In the first stage, the object of fine-tuning in view-specific cue learning is the "learnable cue vector". In the second stage, the object of fine-tuning in base class optimal decision synthesis is the base class classifier (constructing a linear classifier) ​​and the weight matrix.

[0046] In the third stage, decoupling reasoning is performed, specifically including: (1) Obtain the classification results of the image to be classified in panoramic view and subject view.

[0047] For test images First, visual embeddings are obtained by image encoders of two views respectively. and .

[0048] (2) Based on the set of categories to which the category to be classified belongs (base class set) or a new category set Different processing methods are applied.

[0049] If the true category of the image to be classified belongs to the base class set Then, the classification results from the panoramic view and the subject view are weighted and fused using the optimal fusion weights, and the predicted category is determined based on the fusion result. Specifically: Calculate the classification logits for different categories using the weight matrix after the second-stage fine-tuning: , , ; in, Represents a view Category The classification of logits; It is a measure of cosine similarity; Corresponding to the discriminant classifier weights in the second stage of optimization, the optimized weight matrix is ​​taken. Corresponding category The row vector.

[0050] Utilizing optimal fusion weights Perform convex interpolation to determine the prediction category: ; in, Indicates the input image The prediction category and These are the panoramic view and the main view for the base class set, respectively. Medium category The category of logits.

[0051] If the true category of the image to be classified belongs to the new category set Then, evidence theory is used to fuse the classification results under panoramic view and subject view, and the predicted category is determined based on the decision probability after fusion.

[0052] First, since the new class did not appear during training, the text embeddings are calculated using the view-specific cue templates learned in the first stage, and the classification logits are obtained using cosine similarity: , , .

[0053] Then, the Dempster-Shafer evidence theory is used to fuse and determine the prediction category, including: The categorical logits of each view are converted into a probability distribution using softmax. ; Calculation uncertainty: First, calculate the normalized entropy of the probability distribution for each view: ; in, Represents a view The probability distribution, It is a category Corresponding probability value; normalization factor normalized entropy It takes a value between 0 and 1; Obtaining quality of uncertainty ,in This is the scaling factor.

[0054] In one embodiment, the scaling factor .

[0055] Based on the normalized entropy, construct the basic probability assignment (BPA) for each view: , ; in, It is a category The basic probability distribution; Ω represents global ignorance (i.e., uncertainty). It is an uncertain allocation.

[0056] Using Dempster's combination rule, evidence from two views is aggregated by fusing BPA values: , ; in, As a conflict factor; Indicates the fusion of categories Joint probability assignment; This indicates the global uncertainty after fusion; For view The hypothetical subset, A hypothetical subset representing a panoramic view Hypothetical subset of the main view The intersection is ; Represents a view Hypothesis subset The basic probability allocation.

[0057] To simplify the calculation, we can directly use the normalized orthogonal sum formula: , ; Then normalization makes .

[0058] Perform a Pignistic transformation to convert the fused BPA into the final decision probability: ; in, This represents the total number of new categories.

[0059] Take the final decision probability The category corresponding to the maximum value is used as the input image. Prediction categories: ; in, Represents a new set of categories Any of the categories.

[0060] In one embodiment, the visual language model of the present invention is a CLIP model or a variant of the CLIP model. The publicly available ImageNet dataset is used as the base class dataset, and a subset of categories from the base class dataset is selected as a new set of categories for decoupled reasoning and simulation experiments.

[0061] The image encoder from the CLIP model employs a pre-trained ResNet or ViT (VisionTransformer) architecture, while the text encoder employs a pre-trained Transformer architecture.

[0062] In another embodiment of the present invention, a two-stage fine-tuning and decoupled inference apparatus for a visual language model is provided, the apparatus comprising: The first module is used for learning view-specific cues, including: Obtain different views of the input image, including a panoramic view and a subject view; Initialize a learnable cue vector for each view separately; Input the cue vectors and category text of each view into a frozen text encoder to obtain the text features of each view; input the images of each view into a frozen image encoder to obtain the image features of each view; With classification as the objective, a cross-entropy classification loss function is designed to optimize the learnable cue vectors corresponding to each view in parallel, thereby obtaining the trained view-specific cue templates. The second module is used for base class optimal decision synthesis, including: Freeze the post-training view-specific cue template; A trainable base class classifier is constructed using the text features of each view, and the base class classifier is optimized using the image features of each view to obtain a view-specific base class discrimination model. The optimal fusion weights for fusing view classification results are determined by searching on the base class validation set; The third module is used for decoupled reasoning, including: Obtain the classification results of the image to be classified in both panoramic and subject views; If the true category of the image to be classified belongs to the base category set, then the classification results of the two views are weighted and fused using the optimal fusion weight, and the predicted category is determined based on the fusion result. If the true category of the image to be classified belongs to the new category set, then the evidence theory is used to perform uncertainty fusion on the classification results of the two views, and the predicted category is determined based on the decision probability after fusion.

[0063] The framework and workflow of the two-stage fine-tuning and decoupling inference device for the visual language model in this embodiment are as follows: Figure 2 As shown, the three modules (module 1, module 2, and module 3) correspond to the three stages of the aforementioned two-stage fine-tuning and decoupling inference method for the visual language model. The device framework and workflow are summarized as follows: The first module is used for view-specific cue learning (corresponding to the first stage in the method steps). It obtains a panoramic view and a subject view by centering the input image; it initializes a learnable cue vector for each view, concatenates it with the category text and inputs it into the frozen text encoder, and at the same time inputs the view image into the frozen image encoder; it optimizes the cue vector of each view in parallel with the classification task as the goal, and obtains the view-specific cue template. The second module is used for base class optimal decision synthesis (corresponding to the second stage in the method steps). It freezes the prompt templates trained in the first stage; initializes a linear classifier using text embeddings from each view (represented by a linear classifier legend in the diagram), and fine-tunes the classifier using image features; and searches for the optimal fusion weights on the base class validation set. This is used for the weighted fusion of the classification results of the panoramic view and the main view in subsequent tests; The third module, used for decoupling reasoning (corresponding to the third stage in the method steps), first extracts features from the panoramic view and the main view of the image to be classified. If the category belongs to the base class set, the classification logits of the two views are weighted and fused using the fusion weight α obtained in the second stage to determine the predicted category; if it belongs to the new class set, the classification results of the two views are fused using the Dempster-Shafer evidence theory to achieve uncertainty, and finally the prediction result is output based on the fused decision probability.

[0064] Figure 2 The text uses different symbols to label the learnable parameters (first-stage cue vector, second-stage linear classifier) ​​and frozen parameters (the rest), clearly showing the parameter status at each stage.

[0065] In summary, the beneficial effects achieved by this invention through the design of a two-stage fine-tuning and decoupled reasoning method and apparatus for a visual language model are as follows: 1. By decoupling the panoramic view and the subject view, and learning view-specific cue templates respectively, the model can capture both global context and local subject details simultaneously, avoiding the bias of a single view and thus effectively handling the double-edged sword effect of context.

[0066] This invention effectively resolves the context-dependent paradox and improves the model's robustness to visual biases. By employing center cropping as a label-free approximation method, the input image is physically decoupled into a "panoramic view" and a "subject view." Based on this, the first stage of Dual Independent Prompt Learning extracts specific semantic knowledge for both the panoramic and subject views. This explicit decoupling modeling allows the model to flexibly handle contextual information: retaining positive semantic cues from the context in disambiguation (e.g., the runway's aid in aircraft identification) through the panoramic view, while suppressing spurious correlations and biases introduced by the independent subject view. This enables the model to perform well on difficult samples where only context or only the subject can be correctly classified, abandoning traditional implicit feature alignment or complex causal inference debiasing methods.

[0067] 2. In the second stage, the base classifier is fine-tuned based on the frozen prompts, and the optimal fusion weight is searched through the validation set to ensure the base class recognition performance. In the third stage, weighted fusion is used for the base class and evidence theory fusion is used for the new class, which realizes differentiated reasoning for different class types. This not only improves the accuracy of the base class, but also maintains the zero-sample generalization ability of the new class, thus balancing the discriminative power of the base class and the generalization ability of the new class.

[0068] This invention breaks the performance trade-off between base class discriminative power and new class generalization ability. Addressing the difficulty of existing methods in simultaneously handling known classes and unknown classes, this invention employs a two-stage training combined with a double-decoupled inference strategy. During training, the model's discriminative power on base classes is maximized by freezing prompts and initializing text embeddings as linear classifiers for fine-tuning. During inference, this invention uses a double-decoupled inference strategy: for base classes, the optimal fusion weights determined by validation set search are used to accurately aggregate multi-view predictions, ensuring high-precision classification decisions; for new classes, Dempster-Shafer (DS) evidence fusion theory is used for uncertainty fusion. This design ensures that while the model utilizes the discriminative power gain obtained in the second stage to improve base class accuracy, it strictly retains the visual-language alignment ability learned in the first stage, thereby achieving efficient zero-shot transfer to unseen new classes and simultaneously improving performance on both base and new classes.

[0069] 3. The Dempster-Shafer evidence theory is introduced into the new type of reasoning. Uncertainty is predicted by explicitly modeling through entropy measurement, and conflicting evidence is automatically suppressed during multi-view fusion, which enhances the robustness of the model in open-world scenarios.

[0070] For unknown categories in open worlds, this invention introduces the Dempster-Shafer evidence theory in the new category reasoning stage. This method explicitly models and predicts uncertainty through entropy measurement and automatically suppresses conflicting or inconsistent evidence based on its confidence level during multi-view fusion. This mechanism significantly enhances the model's robustness to out-of-distribution samples or unknown categories, avoiding misjudgments caused by conflicting predictions between views.

[0071] Furthermore, the method of this invention is low-cost, highly efficient, and plug-and-play, eliminating the need for expensive semantic segmentation models or additional ground truth classification labels for foreground separation, thus reducing computational costs and data annotation barriers. Simultaneously, as a model-independent module, this invention can be easily integrated into existing mainstream prompting learning methods (such as CoOp, MapLe, PromptSRC, etc.). Experimental results show that the method of this invention consistently improves the harmonic mean accuracy of various baseline models on 11 benchmark datasets, including ImageNet, demonstrating its superior generalization performance and practical value.

[0072] In one embodiment, the present invention provides a computer device, which may be a server, comprising a processor, memory, a network interface, and a database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores two-stage fine-tuning and decoupling inference data for a visual language model. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the two-stage fine-tuning and decoupling inference method for the visual language model.

[0073] Those skilled in the art will understand that the description of the device technical features in the above embodiments does not constitute a limitation on all devices to which the present invention is applied. Specific devices may include more or fewer components, or combinations of certain components, or different component arrangements.

[0074] In another embodiment, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the steps of the aforementioned two-stage fine-tuning and decoupled reasoning method for the visual language model.

[0075] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0076] Matters not covered in this invention are common knowledge.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A two-stage fine-tuning and decoupled reasoning method for a visual language model, characterized in that, include: The first phase involves learning view-specific cues, including: Obtain different views of the input image, including a panoramic view and a subject view; Initialize a learnable cue vector for each view to obtain the cue vector corresponding to each view; Input the cue vectors and category text of each view into a frozen text encoder to obtain the text features of each view; input the images of each view into a frozen image encoder to obtain the image features of each view; With classification as the objective, a cross-entropy classification loss function is designed to optimize the learnable cue vectors corresponding to each view in parallel, thereby obtaining the trained view-specific cue templates. The second stage involves synthesizing the optimal decision for the base class, including: Freeze the post-training view-specific cue template; A trainable base class classifier is constructed using the text features of each view, and the base class classifier is optimized using the image features of each view to obtain a view-specific base class discrimination model. The optimal fusion weights for fusing view classification results are determined by searching on the base class validation set; The third stage involves decoupling reasoning, including: Obtain the classification results of the image to be classified in both panoramic and subject views; If the true category of the image to be classified belongs to the base category set, then the classification results under the panoramic view and the subject view are weighted and fused using the optimal fusion weight, and the predicted category is determined based on the fusion result. If the true category of the image to be classified belongs to the new category set, then the evidence theory is used to perform uncertainty fusion on the classification results under the panoramic view and the subject view, and the predicted category is determined based on the decision probability after fusion.

2. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 1, characterized in that, The acquisition of the main view includes: cropping the central region of the input image as the main view, setting the cropping ratio to 0.4, and maintaining the original image aspect ratio; and retaining the original input image as a panoramic view, thereby obtaining physically decoupled view pairs without relying on additional prior segmentation models or labels.

3. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 1, characterized in that, In the first stage, the image encoder and text encoder are kept frozen during the parallel optimization process, and only the learnable cue vectors are updated. The optimization objective of the parallel optimization process is: ； in, For view Learnable cue vectors; The designed cross-entropy classification loss function is expressed as: ； In the above formula, For view Image input, For view Visual language models predict the true category The probability of; Indicates a panoramic view. This represents the main view.

4. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 3, characterized in that, In the second stage, the trainable base class classifier is a linear classifier; The weight matrix of the linear classifier is composed of view-specific text embeddings stacked together, and is initialized as follows: ； in, Represents a view The initial values ​​of the weight matrix; Stack represents the stacking operation; Represents a view The categories of text encoder output Text embedding, Represents a view The text prompt vector, For the base class category set; When classifying using the trained linear classifier, the image is obtained by multiplying the image features by the transpose of the weight matrix and then scaling using the temperature parameter. The classification of logits: ； in, For view Visual embedding, Represents a view Visual cue vectors; It is a view Transpose of the weight matrix optimized using image features; This refers to the temperature parameter.

5. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 4, characterized in that, In the second stage, the search for and determination of the optimal fusion weights for fusing the view classification results on the base class validation set includes: Define fusion weights Used for convex interpolation The fused classification results are obtained; among them, The logits represent the categories of the panoramic view. The logits represent the categories of the main view. Represents the fused classification logits; Discretize the fusion weights on the base class validation set with a preset step size. search space ; For search space Fusion weights for each candidate Calculate the classification accuracy after fusion; Choose the one with the highest classification accuracy. As the optimal fusion weight .

6. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 5, characterized in that, The preset step size is 0.1, and the search space for the fusion weights... .

7. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 5 or 6, characterized in that, In the third stage, for images to be classified whose true category belongs to the base class set, the optimal fusion weights are used. Perform convex interpolation to determine the prediction category: ； in, Indicates the input image The prediction category and These are the panoramic view and the main view for the base class set, respectively. Medium category The category of logits.

8. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 5 or 6, characterized in that, In the third stage, for images to be classified whose true category belongs to the new category set, the Dempster-Shafer evidence theory is used to fuse and determine the predicted category, including: Convert the categorical logits of each view into a probability distribution. And calculate the normalized entropy. ; Construct the basic probability assignment for each view based on the normalized entropy: ； ； in, It is a category The basic probability distribution; For uncertain quality, Scaling factor Represents a view The probability distribution, It is a category The corresponding probability value; This indicates a complete lack of understanding of the overall situation. It is an uncertain allocation; Aggregate evidence from two views using Dempster's combination rule: ； ； in, Indicates the fusion of categories Joint probability assignment; This indicates the global uncertainty after fusion; For view The hypothetical subset, A hypothetical subset representing a panoramic view Hypothetical subset of the main view The intersection is ; Represents a view Hypothesis subset The basic probability distribution; The final decision probability is obtained through the Pignistic transformation: ； in, This represents the total number of new categories. Take the final decision probability The category corresponding to the maximum value is used as the input image. Prediction categories: ； in, Represents a new set of categories Any of the categories.

9. The two-stage fine-tuning and decoupled reasoning method for the visual language model according to claim 1, characterized in that, The visual language model is a CLIP model or a variant of the CLIP model; the image encoder uses a pre-trained ResNet or ViT architecture, and the text encoder uses a pre-trained Transformer architecture.

10. A two-stage fine-tuning and decoupled reasoning device for a visual language model, characterized in that, include: The first module is used for learning view-specific cues, including: Obtain different views of the input image, including a panoramic view and a subject view; Initialize a learnable cue vector for each view separately; Input the cue vectors and category text of each view into a frozen text encoder to obtain the text features of each view; input the images of each view into a frozen image encoder to obtain the image features of each view; With classification as the objective, a cross-entropy classification loss function is designed to optimize the learnable cue vectors corresponding to each view in parallel, thereby obtaining the trained view-specific cue templates. The second module is used for base class optimal decision synthesis, including: Freeze the post-training view-specific cue template; A trainable base class classifier is constructed using the text features of each view, and the base class classifier is optimized using the image features of each view to obtain a view-specific base class discrimination model. The optimal fusion weights for fusing view classification results are determined by searching on the base class validation set; The third module is used for decoupled reasoning, including: Obtain the classification results of the image to be classified in both panoramic and subject views; If the true category of the image to be classified belongs to the base category set, then the classification results of the two views are weighted and fused using the optimal fusion weight, and the predicted category is determined based on the fusion result. If the true category of the image to be classified belongs to the new category set, then the evidence theory is used to perform uncertainty fusion on the classification results of the two views, and the predicted category is determined based on the decision probability after fusion.

Images