Dynamic knowledge alignment image classification method based on prompt tuning

Abstract

The invention provides a dynamic knowledge alignment image classification method based on prompt tuning, and belongs to the field of artificial intelligence and multi-modal learning. In order to solve the problem that an existing prompt tuning method is poor in performance in fine-grained recognition, new-class generalization and domain migration tasks, the invention provides a DkaPT framework which comprises a dynamic text embedding module, a fine-grained feature alignment module and a noise enhancement module. Dynamic text embedding realizes text feature dynamic coding through a double-layer self-attention network; the fine-grained feature alignment adopts a cross attention mechanism to realize accurate feature alignment; adaptive noise is introduced in noise enhancement to improve model robustness. The method does not need to manually design a fixed prompt template or adapt to the complex field, and the result proves that the method has obvious advantages in various visual identification tasks, and especially has outstanding performance when processing fine-grained identification and complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, multimodal learning, and computer vision. Specifically, it uses the Dynamic Knowledge Alignment Prompt Tuning (DkaPT) framework, which is improved by the Visual Language Model (CLIP) in deep learning, to achieve multimodal knowledge fusion and representation. The goal is to improve the quality of textual knowledge representation and cross-modal knowledge alignment capabilities. Background Art

[0002] With the development of artificial intelligence (AI), multimodal prompt learning has become a core paradigm in vision-language model applications. Traditional prompt tuning methods rely primarily on manually designed templates and fixed visual-text correspondences. These methods exhibit significant limitations when handling complex cross-modal tasks. In particular, effective prompt design and tuning are crucial factors influencing model performance in vision-language model applications such as CLIP.

[0003] The visual language model (CLIP) projects images and text into a shared feature space through contrastive learning, achieving zero-shot transfer capabilities. However, existing CLIP prompt tuning methods still face many challenges when handling fine-grained visual recognition tasks: first, manually designed text prompts are often lacking in specificity, making it difficult to fully activate the model's task-specific knowledge; second, static prompts are difficult to adapt to different visual scenes and changes in data distribution; finally, the model is highly sensitive to small changes in prompts, affecting the stability and accuracy of cross-modal matching.

[0004] Currently, research on CLIP cue tuning focuses primarily on text cue optimization, visual cue learning, and cross-modal alignment. Text cue methods are simple to implement but have limited expressive power; visual cue learning allows for more direct feature adjustment but often incurs additional computational overhead; and cross-modal alignment methods have achieved significant progress in specific domains but lack generalizability in open-domain applications. These difficulties have become a significant factor limiting the further development of multimodal cue learning techniques.

[0005] In recent years, the research on prompt tuning has gradually deepened, and has made important breakthroughs in tasks such as fine-grained image recognition, visual question answering, and cross-modal retrieval. Most of the existing CLIP prompt tuning methods use statically trained prompt templates or learnable continuous prompts, which are difficult to adapt to the dynamic changes of different visual scenes. At the same time, they lack consideration of fine-grained semantics between visual-linguistic modalities, which limits the performance of the model in complex scenes. The present invention realizes efficient prompt tuning and cross-modal knowledge fusion based on CLIP by designing a dynamic knowledge alignment prompt tuning (DkaPT) framework, combining dynamic text embedding and fine-grained feature alignment technology. Summary of the Invention

[0006] In order to overcome the insufficient generalization ability and insufficient utilization of prior knowledge of traditional methods, a dynamic knowledge alignment prompt tuning (DkaPT) method is proposed.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: Step 1: Build a Pytorch-based development environment on the GPU The hardware computing platform of this invention uses an Intel Xeon Silver 4314 processor, an RTX 4090 GPU, and 16GB of memory. The experimental software platform is based on the Ubuntu system and uses CUDA 12.1 and Pytorch 2.3.1 to build the software environment.

[0008] Step 2: Experimental setup and dataset selection To comprehensively evaluate the performance of the Dynamic Knowledge Alignment Hint Tuning (DkaPT) framework, we constructed a complete experimental environment covering a variety of datasets, baseline methods, and evaluation metrics.

[0009] We selected five widely used image classification datasets: Caltech101, a general object recognition dataset; Flowers102 and FGVCAircraft, fine-grained recognition datasets; UCF101, an action recognition dataset; and EuroSAT, a remote sensing image dataset. These datasets cover visual recognition tasks in different domains and at different granularities, enabling a comprehensive evaluation of the model's generalizability and adaptability.

[0010] In the domain generalization experiments, we selected ImageNet as the source domain and used two of its variants as target domains for evaluation: ImageNet-A and ImageNet-R. These variant datasets maintain the same category system as the source domain, but have significant differences in visual representation, which can effectively verify the cross-domain generalization ability of the model.

[0011] In terms of baseline method selection, we mainly compared the current state-of-the-art prompt tuning methods: (1) text prompt tuning methods: CoOp, CoCoOp, KgCoOp, and TCP; (2) multimodal prompt tuning methods: MaPLe and PromptSRC. These methods represent different technical routes and latest progress in the field of prompt learning.

[0012] Our experiments are based on the CLIP model implementation with a ViT-B / 16 backbone. We employ a few-shot learning strategy, randomly sampling 16 samples from each class for training. The prompt length M is set to 4, and all experiments are averaged over three random seeds (1, 2, and 3) to ensure robustness. SGD is used as the optimizer, with an initial learning rate of 0.002, a cosine annealing strategy for learning rate decay, and a batch size of 32. All experiments are conducted on an RTX 4090 GPU to ensure consistent computing resources and reproducible results.

[0013] Step 3: Construction of dynamic embedded module In this paper, the Dynamic Text Embedding (DTE) module takes prior text features as input and generates dynamically optimized text representations through an adaptive learning mechanism. This module adopts a two-layer self-attention network structure to achieve efficient encoding and conversion of text features.

[0014] The DTE module mainly consists of two Self-Attention (SA) blocks and a middle layer QuickGELU activation function. Specifically, the lower layer SA is responsible for converting prior knowledge Mapped to , the upper SA further Mapped to meet the requirements of the text trainer This feature is then combined with the learnable text tokens and trained jointly through subsequent layers of the text trainer. , M and D represent the length and dimension of the prompt respectively. After being processed by the text encoder, the final feature representation is .

[0015] Through the DTE module, prior text features are converted into more expressive dynamic text embeddings, providing a solid foundation for subsequent fine-grained feature alignment. This module is a key component of the DkaPT framework. Its dynamic nature enables the model to adapt to different tasks and changes in data distribution, significantly improving its performance in various visual recognition tasks.

[0016] Step 4: Construction of fine-grained feature alignment module The Fine-grained Feature Alignment (FFA) module is used to process two key inputs and The module adopts the cross attention mechanism, taking as the query and as the key and value. And calculates the precise attention weights through the MultiheadAttention mechanism to realize the dynamic information interaction between the two features. Subsequently, the FFA module cleverly introduces residual connections and applies layer normalization. This design ensures that fine-grained text feature alignment can be achieved at the micro level before calculating the global loss function. Through this mechanism, the module not only effectively retains the key base class information contained in , but also seamlessly integrates the supplementary knowledge obtained from it through the cross attention mechanism, and finally through the formula Form a fused text representation with richer semantics and more comprehensive expression.

[0017] Compared to traditional feature fusion methods, the FFA module's advantage lies in its ability to achieve fine-grained feature alignment at the micro level, rather than simply concatenating or averaging features. This fine-grained alignment mechanism enables the model to accurately capture complementary information in different feature spaces, enabling effective knowledge transfer and integration. Especially when processing samples from new categories, the FFA module can extract semantic information relevant to the current task from prior knowledge, significantly improving the model's generalization capabilities.

[0018] The FFA module bridges the semantic gap between learnable and prior text features, resulting in a richer and more accurate text representation. This module is a key component of the DkaPT framework for knowledge alignment and transfer, contributing significantly to the model's superior performance in various visual recognition tasks.

[0019] Step 5: Construction of noise enhancement module In this paper, the Noise Enhancement (NE) module is designed to improve the robustness and generalization of the model by introducing adaptive noise into the feature space, enabling the model to cope with the variability and uncertainty encountered in real-world applications. This module, located after the image encoder, enhances the diversity of feature representations through a precisely controlled noise injection mechanism.

[0020] The core of the NE module is an adaptive noise generation and scaling mechanism. The module first obtains the adaptive adjustment factor by calculating the mean of the image feature x embedding , ensuring that the noise intensity matches the feature amplitude. For a given Gaussian noise z, the NE module accurately scales it using an adaptive adjustment factor, and then adds the scaled noise to the original image features. The image features after processing by the NE module are expressed as: .

[0021] The introduction of the NE module enables the model to explore a broader feature space, avoiding overfitting to the specific training sample distribution. During training, noise augmentation forces the model to learn more robust feature representations, improving its ability to cope with domain shifts and new categories of samples. In particular, the NE module significantly enhances the model's generalization performance by simulating potential data variations in small-sample learning and domain generalization scenarios.

[0022] Through the processing of the NE module, feature representations that may have been overfitted become more robust and universal, effectively preventing the model from falling into local optimal solutions and prompting it to search for the global optimal solution in a wider solution space. This module works in conjunction with the DTE and FFA modules to form the complete technical system of the DkaPT framework.

[0023] Step 6: Model training and testing The present invention uses a batch processing method for training. The DkaPT framework inputs 32 pairs of pre-processed small batch image samples each time, and outputs the corresponding visual-language matching prediction results. During the training process, the image is first extracted with features through the CLIP visual encoder, and the text prompt is processed by the dynamic text embedding (DTE) module and the fine-grained feature alignment (FFA) module and then converted into text features through the CLIP text encoder. After being processed by the noise enhancement (NE) module, the image features are calculated with the text features for similarity, which is used to calculate the loss function value. The SGD optimization algorithm is used to perform gradient descent operations according to the changes in the loss function to update the network parameters. The initial learning rate is set to 0.002, and the cosine annealing strategy is used for learning rate adjustment, and the weight attenuation coefficient is 5e-4. The parameters of each module are updated and iterated synchronously until the set training rounds are reached or the early stopping condition is met.

[0024] During testing, unseen image samples are input into the trained DkaPT framework to obtain the final classification prediction results. To ensure the reliability of the experimental results, all test processes are repeated on three different random seeds (1 / 2 / 3), and the average value is taken as the performance indicator. The testing phase is new class generalization, which uses category samples that have not been seen in training to evaluate the zero-sample generalization ability of the model. In addition, we also designed a domain generalization experiment to use a dataset with distribution shift to test the model's ability to cope with real-world changes. Experimental results show that the DkaPT framework has achieved significant performance improvements in various visual recognition tasks, especially when dealing with new category samples and domain shift scenarios.

[0025] Step 7: Use evaluation indicators to evaluate the training results The DkaPT framework of the present invention and other comparative methods were evaluated using a variety of visual recognition evaluation indicators. The classification performance of the model on the base class and the new class was evaluated using accuracy; the overall ability of the model was evaluated using the harmonic mean; and the ability of the model to cope with distribution shift was evaluated using domain generalization accuracy. The names of various methods are shown in Table 1. The specific comparison results from the base to the new generalization are listed in Table 2, the specific comparison results in the few-shot learning classification are listed in Table 3, and the specific comparison results in the domain generalization task are listed in Table 4. As can be seen from the data in the table, the comprehensive results of the present invention are the best, and the feature representation generated by DkaPT has better generalization ability, which verifies the effectiveness of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Figure 1 It is a flow chart of the specific implementation steps of the present invention; Figure 2 is a network architecture diagram of the Dynamic Knowledge Alignment Prompt Tuning (DkaPT) framework in an embodiment of the present invention; Figure 3 is a network structure diagram of the noise enhancement (NE) module in an embodiment of the present invention; Figure 4 is a network structure diagram of the fine-grained feature alignment (FFA) module in an embodiment of the present invention; Figure 5 Schematic diagram of feature visualization analysis in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0029] Figure 1 This is a flowchart of the specific steps of the present invention. The invention is mainly divided into 7 steps: (1) building a development environment based on Pytorch; (2) experimental setup and dataset selection; (3) construction of a dynamic text embedding module; (4) construction of a fine-grained feature alignment module; (5) construction of a noise enhancement module; (6) model training and testing; and (7) evaluation of training results using evaluation indicators.

[0030] Figure 2This is a network architecture diagram of the dynamic knowledge alignment prompt tuning (DkaPT) framework in the implementation scheme of the present invention. The DkaPT framework generally includes three core modules: dynamic text embedding (DTE) module, fine-grained feature alignment (FFA) module and noise enhancement (NE) module. These three modules work together to achieve accurate alignment of visual-language representations and knowledge transfer. The input of the framework is an image and a text prompt. The image is extracted from the feature representation through the visual encoder of CLIP, and the text prompt "A photo of a [class]" is processed by the DTE module and the FFA module after passing through the text encoder of CLIP to generate a text feature representation. At the same time, the visual features are processed by the NE module to enhance the diversity and robustness of the features. The similarity between the processed visual features and the text features is calculated for subsequent classification tasks. The entire framework is trained through a multi-objective optimization strategy, combined with a variety of loss functions, to comprehensively guide model learning, significantly improving the performance of the model in many aspects such as base class performance, new class generalization and domain transfer.

[0031] Figure 3 This is a network structure diagram of the noise enhancement (NE) module in the implementation scheme of the present invention. The NE module mainly includes an adaptive noise generation and scaling mechanism, which aims to improve the robustness and generalization ability of the model. The NE module takes the feature embedding output by the image encoder as input, and first obtains an adaptive adjustment factor by calculating the feature mean to ensure that the noise intensity matches the feature amplitude. Subsequently, the module generates random noise that conforms to the Gaussian distribution and accurately scales the noise according to the adjustment factor. Finally, the scaled noise is added to the original image features to form an enhanced feature representation. The design of the NE module enables the model to explore a wider feature space and avoid overfitting a specific training sample distribution.

[0032] Figure 4 This is a network structure diagram of the fine-grained feature alignment (FFA) module in the implementation scheme of the present invention. The FFA module mainly adopts the cross-attention mechanism to achieve precise alignment and fusion between learnable text features and prior text features. The FFA module first uses the learnable text features as the query (Query), and the prior text features as the key (Key) and value (Value), and calculates the precise attention weights through the MultiheadAttention structure to achieve dynamic information interaction and mapping between the two feature representation spaces. The module contains a LayerNorm layer for feature normalization to ensure the stability of training and the consistency of feature distribution.

[0033] Figure 5It is a schematic diagram of feature visualization analysis in an implementation scheme of the present invention. The figure shows the feature distribution of different methods on the EuroSAT and UCF101 datasets, and the high-dimensional features are mapped to a two-dimensional plane for visualization through the t-SNE dimensionality reduction technology. The visualization analysis results clearly show that the DkaPT framework proposed in the present invention can generate more discriminative feature representations than the baseline method CoOp. On the EuroSAT dataset, the features generated by DkaPT show more compact intra-class clustering and clearer inter-class boundaries, and the separation between different categories (represented by different colors) is significantly higher than that of the CoOp method. Similarly, on the UCF101 dataset, the feature distribution of DkaPT also shows better clustering effects, especially for easily confused action categories, DkaPT can better distinguish them.

Claims

1. In order to overcome the insufficient generalization ability of traditional methods and the insufficient utilization of prior knowledge, a dynamic knowledge alignment prompt tuning (DkaPT) method is proposed.

2. To achieve the above objectives, the present invention utilizes the following development environment and equipment: The hardware computing platform utilizes an Intel Xeon Silver 4314 processor, an RTX 4090 GPU, and 16GB of memory. The experimental software platform is based on the Ubuntu system and uses CUDA 12.1 and PyTorch 2.3.1 to build the software environment.

3. Our experiments are based on the CLIP model with ViT-B / 16 bones, using a few-shot learning strategy. 16 samples are randomly sampled from each category for training. The prompt length M is set to 4, and all experiments are averaged over three random seeds (1 / 2 / 3) to ensure the reliability of the results. The optimizer uses SGD with an initial learning rate of 0.002, a cosine annealing strategy for learning rate decay, and a batch size of 32. All experiments are conducted in an environment equipped with an RTX 4090 GPU to ensure the consistency of computing resources and the repeatability of experimental results.

4. Construction of dynamic embedding module: In this invention, the dynamic text embedding (DTE) module takes the prior text features as input and generates dynamically optimized text representation through an adaptive learning mechanism; the module adopts a two-layer self-attention network structure to achieve effective encoding and conversion of text features; the DTE module mainly consists of two Self-Attention (SA) blocks and a middle layer QuickGELU activation function. Specifically, the lower layer SA is responsible for converting the prior knowledge Mapped to The upper SA further converts t down Mapped to meet the requirements of the text trainer This feature is then combined with the learnable text token and trained jointly through the subsequent layers of the text trainer; where D′ = M×D, M and D represent the length and dimension of the prompt respectively; after processing through the text encoder, the final feature representation is Through the processing of the DTE module, prior text features are converted into more expressive dynamic text embeddings, providing a solid foundation for subsequent fine-grained feature alignment; this module is a key component of the DkaPT framework. Its dynamic characteristics enable the model to adapt to changes in different tasks and data distribution, significantly improving the model's performance in various visual recognition tasks.

5. Construction of fine-grained feature alignment module: The Fine-grained Feature Alignment (FFA) module is used to process two key inputs t clip and t dte features; this module adopts the cross-attention mechanism, taking as the query and as the key and value; and calculates the precise attention weights through the MultiheadAttention mechanism to realize the dynamic information interaction between the two features; then, the FFA module cleverly introduces the residual connection and applies layer normalization processing. This design ensures that fine-grained text feature alignment can be achieved at the micro level before calculating the global loss function; through this mechanism, the module not only effectively retains the key base class information contained in , but also seamlessly integrates the supplementary knowledge obtained from it through the cross-attention mechanism, and finally through the formula t dka =Layer Norm(t dte +MA(t dte ,t clip ))Form a fused text representation with richer semantics and more comprehensive expression.

6. Construction of the Noise Enhancement Module: In this paper, the Noise Enhancement (NE) module is designed to improve the robustness and generalization ability of the model. By introducing adaptive noise in the feature space, the model can cope with the various variations and uncertainties in real applications. This module is located after the image encoder and enhances the diversity of feature representations through a precisely controlled noise injection mechanism. The core of the NE module is an adaptive noise generation and scaling mechanism; the module first obtains the adaptive adjustment factor by calculating the mean of the image feature x embedding Ensure that the noise intensity matches the characteristic amplitude; For a given Gaussian noise z, the NE module uses an adaptive adjustment factor to accurately scale it, and then adds the scaled noise to the original image features; The image features after processing by the NE module are expressed as: This module works together with the DTE and FFA modules to form the complete technical system of the DkaPT framework.

7. Model training and testing: The present invention uses a batch processing method for training. The DkaPT framework inputs 32 pairs of preprocessed small batch image samples each time, and outputs the corresponding visual-language matching prediction results. During the training process, the image is first extracted with features through the CLIP visual encoder, and the text prompt is processed by the dynamic text embedding (DTE) module and the fine-grained feature alignment (FFA) module and then converted into text features through the CLIP text encoder. The image features are processed by the noise enhancement (NE) module and the similarity with the text features is calculated for the calculation of the loss function value. The SGD optimization algorithm is used to perform gradient descent operations according to the change of the loss function to update the network parameters. The initial learning rate is set to 0.002, and the cosine annealing strategy is used for learning rate adjustment. The weight decay coefficient is 5e-4. ; The parameters of each module are updated and iterated synchronously until the set training rounds are reached or the early stopping condition is met; during testing, unseen image samples are input into the trained DkaPT framework respectively to obtain the final classification prediction results; to ensure the reliability of the experimental results, all test processes are repeated on three different random seeds (1 / 2 / 3), and the average value is finally taken as the performance indicator; the testing phase is new class generalization, and new class generalization uses category samples that have not been seen in training to evaluate the zero-sample generalization ability of the model; in addition, we also designed a domain generalization experiment, using a data set with distribution offset to test the model's ability to cope with real-world changes; experimental results show that the DkaPT framework has achieved significant performance improvements in various visual recognition tasks, especially when dealing with new category samples and domain offset scenarios.

8. Evaluation indicators are used to evaluate the training results: the classification performance of the model on the base class and the new class is evaluated by accuracy; the overall ability of the model is evaluated by harmonic mean; the ability of the model to cope with distribution shift is evaluated by domain generalization accuracy; the names of various methods are shown in Table 1, the specific comparison results from base to new generalization are listed in Table 2, the specific comparison results in few-shot learning classification are listed in Table 3, and the specific comparison results in domain generalization tasks are listed in Table 4; it can be seen from the data in the table that the comprehensive result of the present invention is the best, and the feature representation generated by DkaPT has better generalization ability, which verifies the effectiveness of the present invention.

9. Table 1 Comparison method name Abbreviated full name and description CoOp Context Optimization CoCoOp Conditional Context Optimization KgCoOp Knowledge-guided Context Optimization MaPLe Multi-modal Prompt Learning PromptSRCPrompting with Self-regulating Constraints, prompting with self-regulating constraints TCP Textual-based Class-aware Prompt tuning, text-based class-aware prompt tuning 10. Table 2 Basic to New Generalization Evaluation 11. Table 3 Few-shot classification evaluation Table 4 Domain generalization evaluation

Images