Unsupervised domain adaptation hint fine-tuning method and related device based on attention mechanism

Abstract

This invention discloses an unsupervised domain adaptation prompt fine-tuning method and related devices based on an attention mechanism. By introducing a small number of learnable parameters through prompt learning and freezing the entire CLIP model, the model can be quickly and efficiently adapted to domain adaptive image classification tasks. At the same time, to further improve accuracy, an attention mechanism is introduced to establish feature libraries of two domains to obtain attention feature pairs. The attention feature pairs and the features of the input image are input into the attention module to obtain features that fuse the source and target domain information. These features are used for subsequent classification. The designed attention branch framework can better capture the domain-invariant features of the target domain, thereby improving the accuracy of domain adaptive image classification.

Description

Technical Field

[0001] The present invention relates to an unsupervised domain adaptation hint fine-tuning method based on an attention mechanism and a related device. Background Art

[0002] In recent years, unsupervised domain adaptation (UDA) has attracted increasing attention due to its ability to avoid the time-consuming and expensive process of acquiring sufficient amounts of manually labeled data. Unsupervised domain adaptation aims to transfer knowledge learned from a labeled source domain to an unlabeled and heterogeneous target domain. However, poor performance is often observed due to the domain shift problem.

[0003] To address domain shift, most methods focus on aligning the distributions between the source and target domains and strive to obtain domain-invariant representations. For example, adversarial training introduces a domain discriminator to distinguish samples from different domains, enabling the trained model to generate domain-invariant features that can deceive the domain discriminator. Methods such as metric learning learn domain-invariant features by minimizing the difference in distance between metric distributions. Recently, some methods based on category-level alignment have leveraged the idea of generating pseudo-labels for semi-supervision to achieve distribution alignment. However, these methods suffer from the problem of inaccurate or non-robust generated noisy pseudo-labels.

[0004] The emergence of large-scale pre-trained visual language models, such as the Contrastive Language-Image Pre-Training (CLIP) model, has introduced semantic information, enabling these multimodal models to possess extremely powerful zero-shot reasoning capabilities. Furthermore, various prompt tuning methods have been proposed for CLIP. These methods have demonstrated excellent performance in several areas, such as few-shot learning, domain adaptation, and domain generalization, while significantly reducing computational complexity and making their practical applications feasible.

[0005] Therefore, in order to better apply it to practical scenarios, it is still worth exploring a simple and efficient fine-tuning-based multimodal classification model solution with cross-domain capabilities, relying on the powerful zero-shot reasoning capabilities of multimodal models such as CLIP and the low computational complexity of the prompt fine-tuning method. The research of such a solution can enable the general multimodal models such as CLIP to be applied to specific environments and have strong generalization capabilities. At the same time, the simplicity and efficiency of the solution also help to reduce implementation and operating costs. To this end, the present invention proposes a prompt fine-tuning method based on an attention mechanism, which combines the attention mechanism to enable the model and prompt to learn domain-invariant features. Summary of the Invention

[0006] The purpose of the present invention is to provide an unsupervised domain adaptation hint fine-tuning method and related devices based on the attention mechanism to solve the technical problems that general multimodal models are difficult to apply to actual specific problems and that existing unsupervised domain adaptation algorithms are difficult to apply in practice due to their high complexity.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides an unsupervised domain adaptation hint fine-tuning method based on an attention mechanism, comprising the following steps:

[0009] Freeze contrastive learning of learnable parameters for the image feature extractor and text feature extractor in the CLIP language-image pre-training model;

[0010] Learn the parameters of the cue and attention modules in the CLIP language-image pre-training model for comparative learning, and construct source and target domain feature libraries;

[0011] The contrastive learning language-image pre-training model CLIP is divided into a baseline branch and an attention branch. It is trained based on the source domain feature library and the target domain feature library to complete the unsupervised domain adaptation prompt fine-tuning of the contrastive learning language-image pre-training model CLIP.

[0012] In a second aspect, the present invention provides an unsupervised domain adaptation hint fine-tuning system based on an attention mechanism, comprising:

[0013] The parameter freezing module is used to freeze the learnable parameters of the image feature extractor and text feature extractor in the contrastive learning language-image pre-training model CLIP;

[0014] The parameter learning module is used to learn the parameters of the cue and attention modules in the CLIP contrastive learning language-image pre-training model and to construct the source domain feature library and the target domain feature library;

[0015] The model training module is used to split the contrastive learning language-image pre-training model CLIP into a baseline branch and an attention branch, and complete the unsupervised domain adaptation prompt fine-tuning of the contrastive learning language-image pre-training model CLIP by training based on the source domain feature library and the target domain feature library.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps of the above method are implemented.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] This paper presents an unsupervised domain adaptive cue fine-tuning method based on an attention mechanism, applying the attention mechanism to the fine-tuning of large multimodal models for unsupervised domain adaptation. This method is easy to implement, has a small number of parameters, and can adapt the model to downstream classification tasks in a short amount of time based on the existing pre-trained model. This method improves the generalization capability of existing large multimodal models (CLIP) and outperforms other state-of-the-art unsupervised domain adaptive classification algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 Flowchart of the method of the present invention.

[0022] Figure 2 Schematic diagram of the system of the present invention.

[0023] Figure 3 It is a schematic diagram of the non-episodic model framework based on parameter-efficient modules of the present invention.

[0024] Figure 4 Schematic diagram of the meta-learning model framework based on parameter-efficient modules of the present invention.

[0025] Figure 5 It is a schematic diagram of the metric learning model framework based on the parameter efficient module of the present invention.

[0026] Figure 6 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0030] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0032] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0033] The present invention is described in further detail below with reference to the accompanying drawings:

[0034] See also Figure 1 , an embodiment of the present invention discloses an unsupervised domain adaptation hint fine-tuning method based on an attention mechanism, comprising the following steps:

[0035] Freeze contrastive learning of learnable parameters for the image feature extractor and text feature extractor in the CLIP language-image pre-training model;

[0036] Learn the parameters of the cue and attention modules in the CLIP language-image pre-training model for comparative learning, and construct source and target domain feature libraries;

[0037] The contrastive learning language-image pre-training model CLIP is divided into a baseline branch and an attention branch. It is trained based on the source domain feature library and the target domain feature library to complete the unsupervised domain adaptation prompt fine-tuning of the contrastive learning language-image pre-training model CLIP.

[0038] In a feasible embodiment of the present invention, the method of constructing the source domain feature library and the target domain feature library includes:

[0039] The zero-shot capability of the contrastive learning language-image pre-training model CLIP is used to generate confidence for each source domain image, and the similarity logits of image and text features are used to generate confidence and corresponding pseudo labels for each target domain image; for the i-th category confidence of image x:

[0040]

[0041] Among them, p is the probability, y is the label of image x, N is the number of categories, t i is the text embedding of the i-th category, f is the feature extractor for images, g is the feature extractor for text, τ is the temperature parameter, and sim represents the cosine similarity; the index of the maximum value is the pseudo label of the image; the visual features with the top K maximum probabilities in each class are selected from the source domain data and the target domain data, and the N-wayK-shot source domain feature library and the target domain feature library are constructed.

[0042] In another feasible embodiment of the present invention, the present invention uses the source domain feature library and the target domain feature library for training, and for the benchmark branch:

[0043] The prompt is inserted into the input at both ends of the image and text, and the predicted posterior probability distribution of image x is expressed as formula (1); the pseudo label is generated using the contrastive learning language-image pre-training model CLIP A fixed threshold τ is set to filter out unreliable pseudo labels. If the maximum probability predicted by the contrastive learning language-image pre-training model CLIP for a given image is lower than this threshold, the pseudo label will be discarded. The cross entropy is used as the loss function for model training:

[0044]

[0045]

[0046] in, is the loss function of the source domain data in the benchmark score, N sis the batch size of source domain data, argmax() is the index of the maximum value, logits base，s is the probability distribution of a batch of source domain data, is the loss function of the target domain data in the benchmark score, Nt is the batch size of the target domain data, logits base，t is the probability distribution of a batch of target domain data is the source domain image data, is the true source domain label, is the target domain image data, is the pseudo label after filtering the target domain, logitS base，s and logits base，s are the similarities between the source domain image and target domain image features and their text features, and ω is a learnable parameter of the prompt.

[0047] In another feasible embodiment of the present invention, the present invention uses the source domain feature library and the target domain feature library for training, and for the attention branch:

[0048] First, a weight-sharing projection layer f is applied pre , an input image feature F v , source domain feature library K s and the target domain feature library F t Convert to the corresponding query Q v , Q s , Q t , key K v , K s , K t Sum V v , V s , V t Indicates that:

[0049] Q s , K s , V s =f pre (K s ) (4)

[0050] Q t , K t , V t =f pre (F t ) (5)

[0051] Q v , K v , V v =f pre (F v ) (6)

[0052] Self-attention and cross-attention are applied to capture the dependencies between features, specifically expressed as:

[0053]

[0054]

[0055] Among them, F′ sa for, is, T is, ∈ is, F′ ta for, F′ sa is the fusion feature with the source domain feature library, is the transpose of the value after the source domain feature library is converted, T is the transpose, ∈ is the scaling hyperparameter, F′ ta is the fusion feature with the target domain feature library, is the transpose of the converted value of the target domain feature library;

[0056] Use another weight-sharing projection layer f post Project the attention value again, expressed as:

[0057] F sa =f post (F′ sa ) (9)

[0058] F ta =f post (F′ ta ) (10)

[0059] Among them, F sa is the fusion feature of the remapped and source domain feature library, F ta is the fusion feature of the remapped and source domain feature library;

[0060] After obtaining the attention visual features, the sum normalization module is applied to combine them with the original features and normalize the results, which is expressed as:

[0061]

[0062]

[0063] Among them, ||·||2 represents the 2-norm; F vs and F vt The similarity between the original text features is calculated to obtain logits1 and logits2. The logits after the attention module are expressed as:

[0064] logits attn，s =β1logits 1，s +β2logits 2，s(13)

[0065] logits attn，t =β1logits 1，t +β2logits 2，t (14)

[0066] Among them, logits attn，s Indicates that logits attn，t Represents logits attn，s Represents the probability distribution of source domain data obtained by the attention branch, logits attn，t represents the probability distribution of the target domain data obtained by the attention branch, s and t indicate that the input image comes from the source domain and the target domain respectively, logits1 and logits2 represent the similarity between the attention features and text features of the source domain and the target domain respectively, β1 and β2 are the trade-off coefficients; a fixed threshold τ is used to filter the unreliable pseudo labels of the target domain image, and we get Cross entropy is used as the loss function to train the model:

[0067]

[0068]

[0069] in, is the loss function obtained for the source domain data of the attention branch, The loss function obtained for the target domain data of the attention branch, is the source domain image data, is the true source domain label, is the target domain image data, is the pseudo-label after filtering in the target domain, ω is the learnable parameter of the prompt, and θ is the parameter of the attention module.

[0070] In another feasible embodiment of the present invention, the loss function of the entire training process using the source domain feature library and the target domain feature library is:

[0071]

[0072] in, is the loss function of the entire training process.

[0073] like Figure 2 As shown, an embodiment of the present invention provides an unsupervised domain adaptation hint fine-tuning system based on an attention mechanism, comprising:

[0074] The parameter freezing module is used to freeze the learnable parameters of the image feature extractor and text feature extractor in the contrastive learning language-image pre-training model CLIP;

[0075] The parameter learning module is used to learn the parameters of the cue and attention modules in the CLIP contrastive learning language-image pre-training model and to construct the source domain feature library and the target domain feature library;

[0076] The model training module is used to split the contrastive learning language-image pre-training model CLIP into a baseline branch and an attention branch, and complete the unsupervised domain adaptation prompt fine-tuning of the contrastive learning language-image pre-training model CLIP by training based on the source domain feature library and the target domain feature library.

[0077] Example

[0078] As shown Figure 3 、 Figure 4 、 Figure 5 and Figure 6 As shown, this embodiment provides an unsupervised domain adaptive hint fine-tuning algorithm based on the attention mechanism, which introduces the attention mechanism to learn a feature extractor with generalization performance, thereby improving the detection accuracy of the small sample image classification method.

[0079] The unsupervised domain adaptive hint fine-tuning algorithm based on the attention mechanism of this embodiment includes the following steps:

[0080] Step 1: Freeze the learnable parameters of the image and text feature extractors in the CLIP model, and only learn the parameters of the cue and attention modules. Learning only a small number of parameters makes the model simple and efficient for downstream tasks.

[0081] Step 2: Build feature libraries for the source domain and target domain respectively. Generate confidence for each source domain image through the powerful zero-shot reasoning capability of the CLIP model and use logits base (i.e. the similarity between image features and text features) Generate confidence and its corresponding pseudo label for each target domain image. The confidence of the i-th category of an image can be expressed as p(y=i|x)=softmax(logits base )[i], where the index of the maximum value is the pseudo label of the image. We select the visual features of the top K images with the largest confidence (i.e. probability) for each category and construct an N-way K-shot source domain feature library and target domain feature library. Specifically:

[0082] Establish a source domain feature library and a target domain feature library. Figure 4 As shown in Figure 1, in order to build a robust and accurate feature library, we first use CLIP's powerful zero-shot capability to generate confidence for each source domain image, and use logits (i.e., the similarity between image and text features) to generate confidence and corresponding pseudo labels for each target domain image. For the i-th category confidence of image x, it can be expressed as:

[0083]

[0084] Among them, t i is the text embedding for the i-th category, f is the image feature extractor, g is the text feature extractor, τ is the temperature parameter, and sim represents the cosine similarity. The index of the maximum value is the pseudo-label for the image. From the source and target domain data, we select the top K visual features with the highest probability (i.e., confidence) for each class and construct an N-way K-shot source and target feature library.

[0085] Step 3: During the training phase, the entire model is divided into a baseline branch and an attention branch, such as Figure 3 shown.

[0086] Step 3.1: For the baseline branch, the image side and text side input the inserted image prompt and text prompt respectively, obtain the corresponding image features and text features through the image feature extractor and text feature extractor, calculate the cosine similarity and use cross entropy to optimize the prompt in the model. Specifically:

[0087] By inserting the prompt into the input at both ends of the image and text, the predicted posterior probability distribution of image x can be expressed as (1). For the source domain, the image data and the corresponding labels can be accessed. However, for the target domain, the true labels cannot be accessed, so the CLIP model is used to generate pseudo labels. In order to filter out unreliable pseudo labels, a fixed threshold τ is set. If the maximum probability predicted by CLIP for a given image is lower than this threshold, the pseudo label will be discarded. To train the model, cross entropy is used as the loss function:

[0088]

[0089]

[0090] in, is the source domain image data, is the true source domain label, is the target domain image data, Pseudo labels after filtering for the target domain, logits base，s and logits base，t are the similarities between the source domain image and target domain image features and their text features, and ω is a learnable parameter of the prompt.

[0091] Step 3.2: For the attention branch, the image features obtained in step 3.1 are combined with the source domain feature library and the target domain feature library to form two attention pairs. Through self-attention and cross-attention mechanisms, features that fuse the source and target domain information are obtained. The fused features are combined with the text features to calculate the cosine similarity, and cross-entropy is used to optimize the hint and attention modules in the model. Specifically:

[0092] like Figure 5 As shown, we first apply a weight-sharing projection layer f pre , an input image feature F v , source domain feature library K s and the target domain feature library F t Convert to the corresponding query Q v , Q s , Q t , key K v , K s , K t Sum V v , V s , V t It can be expressed as:

[0093] Q s , K s , V s =f pre (K s ) (4)

[0094] Q t , K t , V t =f pre (F t ) (5)

[0095] Q v , K v , V v =f pre (F v ) (6)

[0096] Then, self-attention (e.g., source image and source domain feature library) and cross-attention (e.g., source image and target domain feature library) are applied to capture the dependencies between features. Specifically, it can be expressed as:

[0097]

[0098]

[0099] Then use another weight-sharing projection layer f post Projecting the attention value again, it can be expressed as:

[0100] FSa =f post (F′ sa ) (9) F ta =f post (F′ ta ) (10)

[0101] After obtaining the attentive visual features, an additive normalization module is applied to combine them with the original features and normalize the results, which can be expressed as:

[0102]

[0103]

[0104] Among them, ||·||2 represents the 2-norm. These final attention visual features F vs and F vt Containing both original features and attention information helps the model selectively focus on important feature information and improves its ability to accurately classify images from the source and target domains. vs and F vt The similarity between the original text features is calculated to obtain logits1 and logits2, so the logits after the attention module can be expressed as:

[0105] logits attn，s =β1logits 1，s +β2logits 2，s (13)

[0106] logits attn，t =β1logits 1，t +β2logits 2，t (14)

[0107] Among them, s and t indicate that the input image comes from the source domain and the target domain, respectively, logits1 and logits2 represent the similarity between the attention features and text features of the source domain and the target domain, respectively, and β1 and β2 are weight coefficients. A fixed threshold τ is used to filter out unreliable pseudo labels of the target domain image, and we get Then, cross entropy is used as the loss function:

[0108]

[0109]

[0110] in, is the source domain image data, is the true source domain label, is the target domain image data, is the pseudo-label after filtering the target domain, ω is the learnable parameter of the prompt, and θ is the parameter of the attention module. The loss function of the entire training process is:

[0111]

[0112] Step 4: Testing phase. The probability distribution logits obtained by the baseline branch and the attention branch base and logits attn The final probability distribution is added together and used for classification. Specifically:

[0113] Add the logits of the two branches, expressed as:

[0114] logits=logits base +αlogits attn (18)

[0115] Among them, logits base To test the similarity between image features and text features, logits attn To test the similarity between the features of the image after the attention module and the text features, α is the weighting coefficient.

[0116] This invention mainly solves the image classification task in the scenario where a large multimodal model is applied to downstream tasks. By introducing a small number of learnable parameters through prompt learning and freezing the entire CLIP model, the model can be quickly and efficiently adapted to the domain adaptive image classification task. At the same time, in order to further improve the accuracy, an attention mechanism is introduced to establish a feature library of two domains to obtain attention feature pairs. The attention feature pairs and the features of the input image are input into the attention module together to obtain features that fuse the source domain and target domain information. This feature is used for subsequent classification. The designed attention branch framework can better capture the domain-invariant features of the target domain, thereby improving the accuracy of domain adaptive image classification.

[0117] The specific test examples of this embodiment are as follows:

[0118] 1. Test environment

[0119] The test environment of the cross-domain small sample image classification method based on the parameter-efficient module proposed in this invention is: the system environment is Linux18.04, the GPU is NVIDIA GeForce RTX 3090, the memory is 370GB, and the hard disk storage is 2TB.

[0120] 2. Experimental Data

[0121] Experiments are conducted on three popular UDA benchmark datasets, namely Office-Home, Office-31, and VisDA-2017 datasets.

[0122] 1) The Office-Home dataset is a medium-sized domain adaptation benchmark dataset. It contains 15,500 images from four different domains: art (A), clip art (C), product (P), and real world (R). Each domain contains 65 categories of objects commonly found in office and home environments.

[0123] 2) The Visual Domain Adaptation Challenge 2017 (VisDA17) dataset is a more challenging large-scale synthetic-to-real domain adaptation benchmark dataset. It contains 152,397 synthetic images across 12 categories, which are generated by rendering 3D models from different angles and lighting conditions, and 55,388 real-world images collected from the Microsoft Common Objects in Context (MSCOCO) dataset.

[0124] 3) The Office-31 dataset is a popular small-scale domain adaptation benchmark dataset. It contains 4,110 images from 31 categories in three different domains: Amazon (A), webcam (W), and DSLR (D).

[0125] 3. Baseline Model

[0126] To evaluate the effectiveness of our proposed method, we selected seven benchmark algorithms for the hint fine-tuning method, namely CoOp, CoCoOp, VPT, VPT*, IVLP, MaPLe, and DAPL. We also compared APT with the state-of-the-art methods based on ResNet and ViT, including DANN, JAN, MDD, SDAT, MCD, and DAPL, and CDTrans, SDAT, TVT, and DAPL.

[0127] 4. Experimental Setup

[0128] During the training phase, random flipping, center cropping to 224*224, and regularization are performed; during the testing phase, scaling to 224*224 and regularization are performed.

[0129] To evaluate the effectiveness of our method, we used three different baseline models: ResNet50 (RN50), ResNet101 (RN101), and ViT-B / 16 (ViT). We fixed the parameters in the encoder and used the SGD optimizer to perform hint fine-tuning for 10 epochs on the Office-Home and VisDA-2017 datasets, and for 20 epochs on the Office31 dataset, with a batch size of 32. For the hint fine-tuning methods, the learning rate was set to 0.001 for CoCoOp, 0.0025 for VPT and VPT*, 0.003 for CoOp, DAPL, and APT, and 0.0035 for IVLP and MaPLe, with a cosine annealing rule used to gradually decrease the learning rate. In addition, the context token length was set to 2 for MaPLe, 10 for VPT and VPT*, and 16 for CoOp and CoCoOp. The pseudo-label threshold τ was set to 0.8 (Office-Home), 0.9 (Office-31), and 0.6 (VisDA-2017). The weights β1 and β2 of the attention logits were both 0.1, and the weight α of the final logits was set to 0.5. For the N-way K-shot feature library, N was set to the number of categories in each dataset, and K was set to 5. For the Office-Home and VisDA-2017 datasets, due to their high average accuracy on zero-shot CLIP, the logits from zero-shot CLIP were used to generate pseudo-labels. However, for the Office-31 dataset, the average accuracy of zero-shot CLIP was lower. Therefore, a warm-up strategy was adopted, using the logits from zero-shot CLIP for the first few epochs and the logits from fast fine-tuning CLIP for the last few epochs. If the accuracy was still low, the pseudo-labels were only used in the last few epochs.

[0130] 5. Experimental Results

[0131] Tables 1 and 2 below compare the experimental results of existing mainstream solutions and our proposed method, validated on the Office-Home, VisDA-2017, and Office-31 datasets. These results demonstrate that our proposed method (with APT) significantly improves the generalization performance of existing cued learning methods for unsupervised domain adaptation image classification.

[0132] Table 1 Comparison of experimental results of the present invention and other image classification algorithms with prompt learning on the Office-Home dataset

[0133]

[0134] Table 2 Comparison of experimental results of the present invention and other image classification algorithms with prompt learning on the Office-31 dataset

[0135]

[0136] At the same time, in comparison with other state-of-the-art unsupervised domain adaptive image classification algorithms, as shown in Tables 3 and 4, the method of the present invention also performs very well, and has a very high advantage in algorithm complexity and operation time compared with other state-of-the-art algorithms.

[0137] Table 3 Comparison of experimental results of the present invention and other unsupervised image classification algorithms on the Office-Home dataset

[0138]

[0139] Table 4 Comparison of experimental results of the present invention and other unsupervised image classification algorithms on the VisDA-2017 dataset

[0140]

[0141] One embodiment of the present invention provides a computer device. The computer device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of each of the aforementioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in each of the aforementioned apparatus embodiments are implemented.

[0142] The computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to accomplish the present invention.

[0143] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory.

[0144] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0145] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0146] If the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0147] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An unsupervised domain adaptation hint fine-tuning method based on attention mechanism, characterized by: The following steps are involved: Freeze contrastive learning of learnable parameters for the image feature extractor and text feature extractor in the CLIP language-image pre-training model; Learn the parameters of the cue and attention modules in the CLIP language-image pre-training model for comparative learning, and construct source and target domain feature libraries; The contrastive learning language-image pre-training model CLIP is divided into a baseline branch and an attention branch. It is trained based on the source domain feature library and the target domain feature library to complete the unsupervised domain adaptation prompt fine-tuning of the contrastive learning language-image pre-training model CLIP. The method is used to train based on the source domain feature library and the target domain feature library. For the benchmark branch: Insert the hint into the input at both ends of the image and text, image The predicted posterior probability distribution of is expressed by formula (1); the pseudo label is generated using the contrastive learning language-image pre-training model CLIP ; Set a fixed threshold , used to filter out unreliable pseudo labels; if the maximum probability predicted by the contrastive learning language-image pre-training model CLIP for a given image is lower than this threshold, the pseudo label will be discarded; the cross entropy is used as the loss function for model training: in, is the loss function of the source domain data in the benchmark branch, is the batch size of source domain data, is the index of the maximum value, is the probability distribution of a batch of source domain data, is the loss function of the target domain data in the benchmark branch, is the batch size of target domain data, is the probability distribution of a batch of target domain data, is the source domain image data, is the true source domain label, is the target domain image data, is the pseudo label after filtering the target domain, and are the similarities between the source domain image and target domain image features and their text features, is the learnable parameter of the prompt; The utilization is based on the source domain feature library and the target domain feature library for training, for the attention branch: First, a weight-sharing projection layer is applied , an input image feature , source domain feature library and target domain feature library Convert to the corresponding query ,key Sum Indicates that: Self-attention and cross-attention are applied to capture the dependencies between features, specifically expressed as: in, is the fusion feature with the source domain feature library, is the transpose of the value after conversion of the source domain feature library, is the transpose, To scale the hyperparameters, is the fusion feature with the target domain feature library, is the transpose of the converted value of the target domain feature library; Use another weight-sharing projection layer Project the attention value again, expressed as: in, is the fusion feature of the remapped and source domain feature library, It is the fusion feature of the remapped and source domain feature library.

2. The unsupervised domain adaptation hint fine-tuning method based on the attention mechanism according to claim 1 is characterized in that The constructing of the source domain feature library and the target domain feature library includes: The zero-shot capability of CLIP, a contrastive learning language-image pre-training model, is used to generate confidence for each source domain image, and the similarity logits of image and text features are used to generate confidence and corresponding pseudo labels for each target domain image. No. Class confidence: in, is the probability, For images The label of is the number of categories, For the Text embeddings for categories, is the feature extractor for the image, is the feature extractor for text, is the temperature parameter, Represents cosine similarity; the index of the maximum value is the pseudo label of the image; the visual features with the top K maximum probabilities in each class are selected from the source domain data and the target domain data, and the N-wayK-shot source domain feature library and the target domain feature library are constructed.

3. The unsupervised domain adaptation hint fine-tuning method based on the attention mechanism according to claim 1 is characterized in that After obtaining the attention visual features, the sum normalization module is applied to combine them with the original features and normalize the results, which is expressed as: in, represents the 2-norm; and Respectively represent the domain original text features to obtain similarity and , the logits after the attention module are expressed as: in, represents the probability distribution of the source domain data obtained by the attention branch, represents the probability distribution of the target domain data obtained by the attention branch, and Respectively indicate that the input image comes from the source domain and the target domain, and Represents the similarity between the attention features and text features of the source domain and the target domain respectively, and is the trade-off coefficient; using a fixed threshold To filter out unreliable pseudo labels of target domain images, we get ; Use cross entropy as the loss function to train the model: in, is the loss function obtained for the source domain data of the attention branch, The loss function obtained for the target domain data of the attention branch, is the source domain image data, is the true source domain label, is the target domain image data, is the pseudo-label after filtering the target domain, p is the learnable parameter of the prompt, are the parameters of the attention module.

4. The unsupervised domain adaptation hint fine-tuning method based on the attention mechanism according to claim 3 is characterized in that The loss function of the entire training process using the source domain feature library and the target domain feature library is: in, is the loss function of the entire training process, is the weight coefficient.

5. The unsupervised domain adaptation hint fine-tuning method based on the attention mechanism according to claim 4 is characterized in that Also includes: The model is tested by weighted summing of the logits of the two branches.

6. The unsupervised domain adaptation hint fine-tuning method based on the attention mechanism according to claim 5 is characterized in that The method of adding the logits of the two branches to test the model includes: in, To test the similarity between image features and text features, To test the similarity between the features of the image after the attention module and the text features, is the trade-off coefficient.

7. An unsupervised domain adaptation hint fine-tuning system based on an attention mechanism for implementing the method of claim 1, characterized in that: include: The parameter freezing module is used to freeze the learnable parameters of the image feature extractor and text feature extractor in the contrastive learning language-image pre-training model CLIP; The parameter learning module is used to learn the parameters of the cue and attention modules in the CLIP contrastive learning language-image pre-training model and to construct the source domain feature library and the target domain feature library; The model training module is used to split the contrastive learning language-image pre-training model CLIP into a baseline branch and an attention branch, and complete the unsupervised domain adaptation prompt fine-tuning of the contrastive learning language-image pre-training model CLIP by training based on the source domain feature library and the target domain feature library.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images