An image classification method and system based on visual adaptation and dynamic prompting

Abstract

This invention belongs to the field of image classification technology and discloses an image classification method and system based on visual adaptation and dynamic cues. It addresses the problems of inconsistent cross-modal representations and insufficient generalization ability caused by unimodal adaptation in existing visual language models during few-shot learning. The method constructs a bilateral collaborative adaptation mechanism between the visual and textual modalities, combined with a cross-modal alignment optimization strategy, to achieve consistency constraints and joint optimization in the cross-modal semantic space. This enables the model to better adapt to new categories and distribution changes under few-shot conditions, thereby improving the model's generalization performance on few-shot tasks such as base-to-new-class generalization, cross-dataset transfer, and cross-domain scenarios. The system includes a data input module, a visual feature extraction module, a visual feature adaptation module, a dynamic cues generation module, a text feature enhancement module, a cross-modal alignment module, and a prediction module.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, specifically relating to an image classification method and system based on visual adaptation and dynamic cues. Background Technology

[0002] In recent years, with the emergence of large-scale multimodal data and breakthroughs in computing power, image classification methods based on visual language models have gradually become a research hotspot. Large-scale visual language models, pre-trained on hundreds of millions of image and text datasets, have demonstrated powerful cross-modal representation capabilities and good transferability in tasks such as image classification and object recognition. However, in few-shot learning scenarios, due to the limited training samples, directly fine-tuning all parameters of the pre-trained visual language model can easily lead to overfitting and incur high computational costs.

[0003] Existing technologies typically employ efficient parameter tuning strategies to achieve rapid adaptation to downstream tasks while maintaining the core structure of the pre-trained model. Cue learning, by introducing learnable continuous cue vectors at the text level, expands or reconstructs the input text sequence, guiding the model to generate text feature representations more suited to the target task. This achieves excellent performance on a small number of samples without fine-tuning a large pre-trained model. Visual feature adaptation methods introduce a lightweight adaptation module at the visual encoder output to locally adjust visual features, enhancing the adaptability of visual representations to the target task. By compensating for the visual features, this approach alleviates the mismatch between pre-trained features and downstream tasks to some extent.

[0004] However, existing methods are limited to a single-modal adaptation strategy, that is, adjusting only the text or visual modality while keeping the other modality completely frozen. This single-modal adaptation strategy has significant shortcomings in practical applications. On the one hand, cue learning methods cause text features to continuously change during training, while visual features remain fixed, leading to a gradual shift in the representations of the two modalities in the same semantic space, thus affecting cross-modal matching performance. On the other hand, visual feature adaptation methods only make local adjustments within the visual space, lacking synergistic constraints with text semantics, making it difficult for the adjusted visual features to form a stable and consistent semantic correspondence with text features. Furthermore, the core advantage of large-scale visual language models lies in their cross-modal semantic alignment capabilities, while existing single-modal adaptation methods lack overall modeling of the interaction between the visual and text modalities. This makes it difficult to effectively bridge the semantic gap between visual and text features in application scenarios with distributional shifts, thus limiting the model's classification performance and generalization ability under few-sample conditions. Moreover, the representation spaces of different modalities have inherent differences, and simply optimizing a single modality independently makes it difficult to achieve consistency constraints in the semantic space, thus limiting the overall performance improvement of the model.

[0005] Therefore, while maintaining parameter efficiency, it is crucial to achieve collaborative adaptation and consistency constraints between visual and textual representations, thereby improving cross-modal semantic alignment capabilities and few-shot classification performance. Summary of the Invention

[0006] To address the issues of inconsistent cross-modal representations and insufficient generalization ability caused by unimodal adaptation in existing visual language models during few-shot learning, the present invention aims to provide an image classification method and system based on visual adaptation and dynamic cues. This method constructs a bilateral collaborative adaptation mechanism between the visual and textual modalities, and combines a cross-modal alignment optimization strategy to achieve consistency constraints and joint optimization in the cross-modal semantic space. To achieve the above technical objective, the technical solution adopted by the present invention is as follows:

[0007] Firstly, this paper provides an image classification method based on visual adaptation and dynamic cues, including the following:

[0008] Obtain labeled samples for the target task, wherein the labeled samples include image data and corresponding category text descriptions;

[0009] The image data is input into the visual encoder in the pre-trained visual language model to extract features from the input image data and obtain basic visual feature representations.

[0010] A visual adapter module is constructed to perform content-aware feature calibration processing on the basic visual feature representation to obtain an enhanced visual feature representation.

[0011] A dynamic prompt generation module is constructed to generate instance-level prompt vectors related to the input image based on basic visual features, and then the instance-level prompt vectors are fused with category text embeddings to obtain a text input sequence;

[0012] The text input sequence is fed into the text encoder in a pre-trained visual language model to obtain an enhanced text feature representation.

[0013] A cross-modal alignment optimization mechanism is constructed, which jointly optimizes the enhanced visual feature representation and the enhanced text feature representation based on the similarity between enhanced visual features and enhanced text features, thereby reducing the cross-modal semantic gap;

[0014] Based on the similarity between the enhanced visual feature representation and the enhanced text feature representation, the category prediction probability is calculated, the image is classified, and the final classification result is output.

[0015] In this invention, a small number of labeled samples of the target task are first obtained. The image data in the labeled samples are input into the visual encoder in the pre-trained visual language model to extract the corresponding basic visual feature representations. The basic visual features are then dynamically calibrated with content awareness by constructing a conditional visual adapter, so that the visual features can be adaptively adjusted according to the current task.

[0016] Then, based on the aforementioned basic visual features, a meta-cue generation mechanism is constructed to generate instance-level cue vectors related to the input image, and these vectors are combined with category text embeddings to construct a text input sequence.

[0017] At the same time, the text input sequence is input into the text encoder in the pre-trained visual language model to obtain the corresponding enhanced text feature representation, so that the text representation can perceive the information of the specific input sample;

[0018] Finally, by constructing a cross-modal alignment mechanism, visual features and text features are jointly optimized in a unified semantic space, and classification prediction is achieved based on the similarity between the two, thereby improving the model's generalization ability in few-shot learning and cross-domain scenarios.

[0019] Secondly, the present invention provides an image classification system based on visual adaptation and dynamic cues, for implementing the aforementioned image classification method based on visual adaptation and dynamic cues, including...

[0020] Data input module;

[0021] Visual feature extraction module;

[0022] Visual feature adaptation module;

[0023] Dynamic prompt generation module;

[0024] Text feature enhancement module;

[0025] Cross-modal alignment module; and

[0026] Prediction module.

[0027] In this invention, the data input module is configured to receive labeled samples of the target task and preprocess them to obtain standardized input data, including image data and corresponding category text descriptions.

[0028] The visual feature extraction module is configured to extract basic visual feature representations of image data through a visual encoder in a pre-trained visual language model.

[0029] The visual feature adaptation module is configured to input the basic visual feature representation into the conditional visual adapter, perform content-aware feature calibration, perform nonlinear mapping of visual features through a bottleneck structure, and generate an enhanced visual feature representation by combining residual connections.

[0030] The dynamic prompt generation module is configured to generate instance-level contextual prompt vectors related to the input image through a prompt generation network based on the basic visual features, and combine them with categorical text embeddings to construct a text input sequence;

[0031] The text feature enhancement module is configured to input the text input sequence into the text encoder in a pre-trained visual language model, and extract enhanced text feature representations through the text encoder.

[0032] The cross-modal alignment module is configured to construct a collaborative optimization objective, and based on the similarity calculation between enhanced visual features and enhanced text features, jointly optimize the conditional visual adapter and the meta-cue generator so that the two are aligned in a unified space, achieving bilateral collaborative adaptation of visual and text representations.

[0033] The prediction module is configured to perform a classification task on new image samples based on the similarity calculation between the enhanced visual feature representation and the enhanced text feature representation, and output the classification result.

[0034] The beneficial effects of this invention are:

[0035] 1. By introducing a visual adapter module into the visual modality, the features output by the pre-trained visual encoder are dynamically calibrated with content awareness, enabling the visual features to be adaptively adjusted according to the current task. While maintaining the original semantic information, the visual features are effectively transferred to downstream tasks, thereby improving the discriminative ability and task relevance of the visual features. This solves the problem that visual features are difficult to adapt to low-sample tasks in the existing technology.

[0036] 2. By constructing a dynamic prompt generation module, instance-level prompt vectors are dynamically generated based on the visual features of the input image, enabling text representation to perceive the semantic information of specific input samples. This enhances the expressive and discriminative power of text features, avoids the problem that existing static prompt methods cannot effectively characterize sample differences, and achieves more refined cross-modal alignment.

[0037] 3. By simultaneously modeling the visual and textual modalities and combining them with a cross-modal alignment optimization mechanism, the visual and textual features are jointly constrained, making the representations of the two modalities more consistent in a unified semantic space. This effectively alleviates the problem of inconsistent modal representations in existing single-modal matching strategies and improves the model's cross-modal semantic alignment capability.

[0038] 4. During training, only the parameters of the visual adaptation module and the dynamic prompt generation module are updated. While keeping the main structural parameters of the pre-trained model frozen, the number of training parameters and computational overhead of the model are significantly reduced, while still achieving high classification performance. This achieves an effective balance between parameter efficiency and model performance.

[0039] 5. By constructing a bilateral collaborative adaptation mechanism between vision and text, the model is trained from the perspective of cross-modal overall optimization, enabling the model to better adapt to new categories and distribution changes under few-sample conditions, thereby improving the model's generalization performance on few-sample tasks such as base class to new class generalization, cross-dataset transfer, and cross-domain scenarios. Attached Figure Description

[0040] The present invention can be further illustrated by the non-limiting embodiments given in the accompanying drawings;

[0041] Figure 1 This is a flowchart of Embodiment 1 of the present invention;

[0042] Figure 2 This is a structural diagram of the prediction period in Embodiment 1 of the present invention;

[0043] Figure 3 This is a system structure diagram of Embodiment 2 of the present invention. Detailed Implementation

[0044] The technical solutions of the present invention will be described in detail below with reference to specific embodiments and accompanying drawings. The embodiments described herein are specific implementations of the present invention, used to illustrate the concept of the present invention; these descriptions are explanatory and exemplary, and should not be construed as limiting the implementation methods or the scope of protection of the present invention. In addition to the embodiments described herein, those skilled in the art can employ other obvious technical solutions based on the content disclosed in the claims and specification of this application. These technical solutions include those that make any obvious substitutions and modifications to the embodiments described herein.

[0045] Example 1

[0046] like Figure 1 , 2 As shown, this embodiment provides an image classification method based on visual adaptation and dynamic cues, including the following steps:

[0047] (1) Data acquisition and task modeling

[0048] Obtain a small dataset of labeled samples for the target task. The dataset can be represented as follows:

[0049]

[0050] in, This represents the i-th input image. Here, N represents the corresponding category label, K represents the total number of samples, and K represents the number of categories. The above dataset constitutes the basic training data for few-shot learning scenarios. Its characteristic is that the number of samples corresponding to each category is limited, which places higher demands on the model's generalization ability and cross-modal alignment ability.

[0051] For each category k, construct its corresponding category text description, represented in the form of a prompt template:

[0052]

[0053] Here, [CLASS] represents the category name of the k-th class. Further, the category name is mapped to a vector representation using word embeddings:

[0054]

[0055] in This represents the text embedding vector corresponding to the k-th class. This represents the dimension of the text feature space. Through the above processing, it transforms discrete category labels into vector representations in a continuous semantic space, providing a foundation for subsequent cross-modal alignment.

[0056] (2) Visual feature extraction

[0057] Input image Input the visual encoder in the pre-trained visual language model to obtain the basic visual feature representation:

[0058]

[0059] in, The visual encoder mapping function representing the freeze parameters. Representing an image The visual feature vector, where d is the visual feature dimension.

[0060] The visual encoder extracts a high-level visual representation containing global semantic information by segmenting images, linear embedding, and multi-layer self-attention modeling. The parameters of the visual encoder are frozen during training, thus fully preserving the general semantic knowledge learned in the large-scale pre-training phase, while avoiding overfitting during few-sample training.

[0061] (3) Visual adapter feature calibration

[0062] To enhance the adaptability of visual features in downstream few-shot tasks, a visual adapter module is constructed to perform content-aware feature calibration on the basic visual features, resulting in enhanced visual feature representations.

[0063]

[0064] in, It is a learnable gating coefficient, initially set to 0, to ensure that the influence of the adapter is introduced gradually in the early stages of training, avoiding damage to the existing semantic structure of the pre-trained model. The feature offsets learned by the adapter module.

[0065] The feature offset is calculated through the bottleneck structure. First, it is obtained through a dimensionality reduction matrix.

[0066] Projecting high-dimensional visual features v onto a low-dimensional bottleneck space:

[0067]

[0068] in, Next, layer normalization (LN) is applied to h to stabilize the training:

[0069]

[0070] in , Here, h represents the mean and standard deviation, respectively. and These are learnable scaling and offset parameters.

[0071] Next, the GELU activation function is applied to the normalized features, and then the dimension is increased by an upscaling matrix.

[0072] Projecting it back to the original dimension yields the feature offset:

[0073]

[0074] The entire adapter employs a minimalist bottleneck structure design, resulting in a compact structure and a significantly lower number of parameters compared to the vision encoder itself. In actual deployment, the additional computational overhead is negligible, fully adhering to the principle of efficient parameter fine-tuning, making it suitable for deployment on resource-constrained edge devices.

[0075] (4) Dynamic prompt generation

[0076] A dynamic cue generation module is constructed, which uses a lightweight meta-cue generator to generate a set of context vectors related to the image content, conditioned on the basic visual features v of the input image.

[0077]

[0078] Where m is the number of context tags. This is the dimension of text embedding. This design makes text prompts no longer static or category-general, but dynamically adjustable based on the visual content of each input image, achieving instance-level adaptation of text semantics. The parameter is The prompt generates the network.

[0079] Based on the above cue vector, it is concatenated with the category embedding vector to construct the text input sequence:

[0080]

[0081] in, Indicates the input image The text sequence constructed with category k consists of dynamically generated contextual cue vectors and category word vectors, thereby enhancing the model's ability to characterize fine-grained semantic differences.

[0082] (5) Text feature encoding

[0083] The constructed text input sequence is input into the text encoder of the pre-trained visual language model to be encoded into the corresponding text feature representation:

[0084]

[0085] in, This represents the text feature representation of the i-th image and the k-th class. This represents the text encoder mapping function. The text encoder performs semantic processing on the input sequence through a multi-layer self-attention mechanism.

[0086] Modeling is used to output text representations in a unified semantic space.

[0087] (6) Cross-modal alignment optimization

[0088] The model's final prediction is based on the enhanced visual features v′ and dynamically generated text features. The semantic similarity between them. We use cosine similarity as a metric and calculate the probability of belonging to each category using the softmax function:

[0089]

[0090] Here, τ represents the temperature parameter, used to adjust the smoothness of the probability distribution, thereby affecting the model's sensitivity to class distinctions. This collaborative prediction mechanism ensures that visual and textual representations interact and match within the same semantic space.

[0091] A training batch contains N samples. For the i-th sample, its enhanced visual features after processing by the visual adapter module are represented as follows: The corresponding category is represented by the text features obtained through the dynamic prompt generation module and the text encoder. Contrast alignment loss aims to... Zoom in At the same time, it extends the text features of all other samples. The distance, that is, forcing the visual features of each sample to be as close as possible to the text features of its own category, while keeping them as far away as possible from the text features of all other categories. Defined as:

[0092]

[0093] in, Indicates the contrast alignment loss. This represents the similarity between the visual features of the i-th sample and its true category text features. The indicator function is defined as follows:

[0094] This loss function constructs a contrast relationship within the same batch, making the visual features of each sample as close as possible to the text features of its corresponding category, while keeping them away from the text features of other samples. This forms clear category boundaries in a unified semantic space, efficiently driving the alignment of cross-modal features and effectively improving the consistency and discriminative ability of cross-modal features.

[0095] Based on this, we introduce classification loss:

[0096]

[0097] The final overall optimization objective function of the model is composed of the traditional classification loss and the contrastive alignment loss mentioned above:

[0098]

[0099]

[0100] in, It uses a standard classification loss based on cross-entropy, ensuring the model's basic discriminative ability. This represents the balancing parameter, used to adjust the weight relationship between classification loss and contrastive alignment loss. Our proposed contrastive alignment loss is the engine for bridging the modality gap and achieving collaborative adaptation. This joint optimization objective guides both the visual adapter and the meta-cue generator, ensuring classification performance while explicitly constraining the alignment of visual and textual features in the semantic space. This drives the visual adapter module and the dynamic cue generation module to perform collaborative optimization, achieving unification and enhancement of cross-modal representations.

[0101] (7) Classification prediction

[0102] After the model training is complete, for the input image The similarity between the data and each category is calculated using the steps described above, and the corresponding category probability distribution is obtained:

[0103]

[0104] Finally, by selecting the category with the highest probability as the output, the predicted category is output, thus achieving the image classification task.

[0105] Example 2

[0106] like Figure 3 As shown in the figure, this embodiment presents an image classification system based on visual adaptation and dynamic cues: including

[0107] Data input module;

[0108] Visual feature extraction module;

[0109] Visual feature adaptation module;

[0110] Dynamic prompt generation module;

[0111] Text feature enhancement module;

[0112] Cross-modal alignment module; and

[0113] Prediction module.

[0114] In the above embodiments, the present invention introduces a visual adapter module into the visual modality to perform content-aware dynamic calibration on the features output by the pre-trained visual encoder. This enables visual features to adaptively adjust according to the current task, achieving effective transfer of visual features to downstream tasks while preserving the original semantic information. This improves the discriminative ability and task relevance of visual features, thus solving the problem of visual features being difficult to adapt to tasks with few samples in the prior art. By constructing a dynamic cue generation module, instance-level cue vectors are dynamically generated based on the visual features of the input image, enabling text representation to perceive the semantic information of specific input samples. This enhances the expressive and discriminative power of text features, avoiding the problem that existing static cue methods cannot characterize sample differences, and achieving more refined cross-modal alignment. By simultaneously modeling the visual and text modalities and combining cross-modal alignment... The optimization mechanism jointly constrains visual and textual features, making the representations of the two modalities more consistent in a unified semantic space. This effectively alleviates the inconsistency in modal representations in existing single-modal matching strategies and improves the model's cross-modal semantic alignment capability. During training, only the parameters of the visual adaptation module and the dynamic prompt generation module are updated. While keeping the main structural parameters of the pre-trained model frozen, the number of training parameters and computational overhead are significantly reduced, while still achieving high classification performance, thus achieving an effective balance between parameter efficiency and model performance. By constructing a bilateral collaborative adaptation mechanism between vision and text, the model is trained from the perspective of overall cross-modal optimization. This enables the model to better adapt to new categories and distribution changes under few-sample conditions, thereby improving the model's generalization performance on few-sample tasks such as base-to-new-class generalization, cross-dataset transfer, and cross-domain scenarios.

[0115] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for image classification based on visual adaptation and dynamic cues, characterized in that: Includes the following: Obtain labeled samples for the target task, wherein the labeled samples include image data and corresponding category text descriptions; The image data is input into the visual encoder in the pre-trained visual language model to extract features from the input image data and obtain basic visual feature representations. A visual adapter module is constructed to perform content-aware feature calibration processing on the basic visual feature representation to obtain an enhanced visual feature representation. A dynamic prompt generation module is constructed to generate instance-level prompt vectors related to the input image based on basic visual features, and then the instance-level prompt vectors are fused with category text embeddings to obtain a text input sequence; The text input sequence is fed into the text encoder in a pre-trained visual language model to obtain an enhanced text feature representation. A cross-modal alignment optimization mechanism is constructed, which jointly optimizes the enhanced visual feature representation and the enhanced text feature representation based on the similarity between enhanced visual features and enhanced text features, thereby reducing the cross-modal semantic gap; Based on the similarity between the enhanced visual feature representation and the enhanced text feature representation, the category prediction probability is calculated, the image is classified, and the final classification result is output.

2. An image classification system based on visual adaptation and dynamic cues, used to implement the image classification method based on visual adaptation and dynamic cues as described in claim 1, characterized in that: include Data input module; Visual feature extraction module; Visual feature adaptation module; Dynamic prompt generation module; Text feature enhancement module; Cross-modal alignment module; and Prediction module.

3. The image classification system based on visual adaptation and dynamic cues according to claim 2, characterized in that: The data input module is configured to receive labeled samples of the target task and preprocess them to obtain standardized input data, including image data and corresponding category text descriptions. The visual feature extraction module is configured to extract basic visual feature representations of image data through a visual encoder in a pre-trained visual language model. The visual feature adaptation module is configured to input the basic visual feature representation into the conditional visual adapter, perform content-aware feature calibration, perform nonlinear mapping of visual features through a bottleneck structure, and generate an enhanced visual feature representation by combining residual connections. The dynamic prompt generation module is configured to generate instance-level contextual prompt vectors related to the input image through a prompt generation network based on the basic visual features, and combine them with categorical text embeddings to construct a text input sequence; The text feature enhancement module is configured to input the text input sequence into the text encoder in a pre-trained visual language model, and extract enhanced text feature representations through the text encoder. The cross-modal alignment module is configured to construct a collaborative optimization objective, and based on the similarity calculation between enhanced visual features and enhanced text features, jointly optimize the conditional visual adapter and the meta-cue generator so that the two are aligned in a unified space, achieving bilateral collaborative adaptation of visual and text representations. The prediction module is configured to perform a classification task on new image samples based on the similarity calculation between the enhanced visual feature representation and the enhanced text feature representation, and output the classification result.

Images