Method for extracting domain-invariant features and optimizing text transfer learning for CLIP

Abstract

The application relates to a transfer learning method for extracting domain-invariant features and optimizing text for CLIP, which comprises the following steps: training a specific domain classifier by using training images; mapping the training images by using an encoder-decoder network, and separating specific domain information from the training images by using the trained specific domain classifier; establishing a text optimizer, training the text optimizer by using the training images, text vectors corresponding to the training images and the specific domain information separated from the training images, and obtaining a trained text optimizer; establishing a domain-invariant feature extraction model, training the domain-invariant feature extraction model by using training images of different domains, and obtaining a trained domain-invariant feature extraction model; and integrating the trained text optimizer and the trained domain-invariant feature extraction model to obtain a CLIP pre-training model with improved generalization performance. The application can improve the generalization performance of the CLIP pre-training model.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a transfer learning method for extracting domain-invariant features and optimizing text using CLIP. Background Technology

[0002] Machine learning (ML) has achieved remarkable success in various fields such as computer vision, natural language processing, and healthcare. The goal of ML is to design a model that can learn general and predictive knowledge from training data and then apply that model to new (test) data. Traditional ML models are based on the assumption that the training and test data are identical and independently distributed. However, this assumption does not always hold true in reality. When the probability distributions of the training and test data differ, the performance of ML models often degrades due to the domain distribution discrepancy. Collecting data from all possible domains to train ML models is expensive, and often highly impractical. Therefore, improving the generalization ability of ML models is of great significance in both industry and academia.

[0003] There are many research topics related to generalization, such as domain adaptation, meta-learning, and transfer learning. In recent years, domain generalization (DG) has received widespread attention. The goal of domain generalization is to learn a model from one or more different but related domains (i.e., different training datasets), and this model will generalize well on an unknown test domain. For example, given a training set consisting of images from sketches and cartoons, domain generalization requires the model to be trained to recognize images of another style (given a cartoon-style dog, it can recognize a real-world dog), images with a significantly different distribution from the images in the training set. In the past few years, domain generalization has made significant progress in various fields such as computer vision and natural language processing. At the same time, large-scale pre-trained models are constantly evolving, and the exponential growth in the size and complexity of these models has significantly improved their ability to understand natural language, enabling them to adapt to a variety of downstream tasks.

[0004] The CLIP visual model is trained on billions of image-text pairs, enabling it to extract highly generalized visual features. Its core concept is to learn image representations through comparison using textual descriptions of images. Specifically, it measures the alignment between text and images to classify them. CLIP uses a two-stream branching approach, with two encoders processing text and image data separately. It then calculates the cosine similarity between the two modalities, maximizing the similarity of N matching image-text pairs and minimizing the similarity of non-matching pairs. Finally, it outputs the corresponding logistic vector for prediction. CLIP's performance is studied through benchmarking on over 30 existing computer vision datasets, covering tasks such as OCR, action recognition in videos, geolocation, and fine-grained object classification of many types. CLIP achieves comparable, or even better, performance on most tasks compared to supervised training frameworks. It is worth noting that CLIP can achieve this effect without any specific training on any dataset. However, the CLIP model has the following limitations: (1) The CLIP pre-trained model is huge. Training the parameters of the entire network will affect the original good feature expression of CLIP and will consume additional computing resources and time; (2) DG focuses on predicting the performance of unknown domains. CLIP's text prompts lack flexibility in domain optimization; (3) In DG, domain transfer requires the model to have the ability to extract domain-invariant features. CLIP lacks the ability to extract more robust domain-invariant features. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a transfer learning method for extracting domain-invariant features and optimizing text using CLIP, which can improve the generalization ability of CLIP pre-trained models.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide a transfer learning method for extracting domain-invariant features and optimizing text using CLIP, comprising the following steps:

[0007] A domain-specific classifier is trained using training images, enabling the domain-specific classifier to recognize images from a specific domain.

[0008] An encoder-decoder network is used to map the training images, and a trained domain-specific classifier is used to separate domain-specific information from the training images.

[0009] A text optimizer is established by training images, text vectors corresponding to the training images, and information from specific domains extracted from the training images, to obtain a trained text optimizer.

[0010] A domain-invariant feature extraction model is established, and the domain-invariant feature extraction model is trained using training images from different domains to obtain a trained domain-invariant feature extraction model.

[0011] By integrating a pre-trained text optimizer and a pre-trained domain-invariant feature extraction model, a CLIP pre-trained model with improved generalization performance is obtained.

[0012] The text optimizer includes:

[0013] A text encoder takes the text vector corresponding to the training image and information from a specific domain extracted from the training image as input, and outputs a text vector.

[0014] An image encoder takes the training image as input and outputs a matrix vector;

[0015] The prompt generator takes the text vector and matrix vector as input and outputs text modality information.

[0016] When training the text optimizer, the parameters of the text encoder and the image encoder are frozen.

[0017] The prompt generator is constructed as a three-layer MLP, with each layer consisting of a linear layer, a Tahn activation function, and another linear layer and a Tahn activation function.

[0018] The domain-invariant feature extraction model comprises a sequentially connected encoder-decoder network and a trained domain-specific classifier.

[0019] The process of training the domain-invariant feature extraction model using training images from different domains specifically involves:

[0020] Training images from different domains are input into a pre-trained domain-specific classifier for retraining;

[0021] After the classifier for the specific domain is retrained, its parameters are frozen, and training images from different domains are input into the encoder-decoder network to train the encoder-decoder network.

[0022] When retraining the trained domain-specific classifier, the loss function minimizes the classification loss to approximate the label value of the image in that domain; when training the encoder-decoder network, the loss function maximizes the classification loss to distance itself from the label values ​​of images in other domains.

[0023] The minimized classification loss is expressed as: The maximization of classification loss is expressed as: ,in, Let i represent the data in the i-th domain, where i∈(1,…,N). This represents the input value and label value of the j-th image in the i-th domain. This represents the classification operation of a domain-specific classifier. The loss function representing classification. Represents the uncertainty loss function. Represents network parameters, Represents the i-th domain. This represents a set of domains with a total number of N. This represents the input image features of the j-th image in the i-th domain. This represents the input image label value for the j-th image in the i-th domain.

[0024] Beneficial effects

[0025] By adopting the above-mentioned technical solutions, this invention has the following advantages and positive effects compared with the prior art: This invention establishes a text optimizer that integrates separated domain-specific information for training, enabling text prompts to contain better semantic information; This invention also proposes a domain-invariant feature extractor that can effectively separate domain-specific information, thereby improving domain-invariant features and achieving stronger DG performance. Attached Figure Description

[0026] Figure 1 This is a flowchart of the transfer learning method for extracting domain-invariant features and optimizing text using CLIP, according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the text optimizer in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of a domain-invariant feature extraction model in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the CLIP pre-trained model obtained by an embodiment of the present invention. Detailed Implementation

[0030] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0031] like Figure 1As shown, the embodiments of the present invention relate to a transfer learning method for extracting domain-invariant features and optimizing text using CLIP. This method can obtain a CLIP pre-trained model with improved generalization performance, namely DALSCLIP (see...). Figure 4 This model consists of a text optimizer and a domain-invariant feature extraction model, which can dynamically adjust the prompts for test examples and learn stronger domain-invariant features for cross-domain general feature representations.

[0032] Training DALSCLIP is performed in two steps. In the first step, domain-specific classifiers are trained using training images. Each domain has its own classifier that can recognize images from its own domain but not images from other domains. Then, an encoder-decoder network is used to map the training images, and the domain-specific classifiers are used to separate domain-specific information from these images. The goal is for the domain-specific classifiers to be unable to recognize images from their corresponding domains after mapping, indicating that the mapped images contain fewer domain-specific details and have better domain-invariant features. A text optimizer is then built, trained using the training images, the corresponding text vectors, and the domain-specific information separated from the training images. The parameters of the encoder-decoder network are frozen during training. In this embodiment, the text optimizer integrates the previously separated domain-specific information during training. At this stage, due to the improved domain-invariant properties, text prompts can contain better semantic information.

[0033] like Figure 2 As shown, in this embodiment, the text optimizer includes: a text encoder, which takes the text vector corresponding to the training image and domain-specific information extracted from the training image as input, and outputs a text vector; an image encoder, which takes the training image as input and outputs a matrix vector; and a cue generator, which takes the text vector and the matrix vector as input and outputs text modality information. A cue generator is trained on the training set, where given an input image from each distribution and a fixed-length text vector, the cue generator is updated while keeping other network parameters frozen. Throughout the training process, the cue generator learns domain-specific features from the images and incorporates them into the text cue. Because all network parameters except the cue generator in the entire text optimizer are frozen, the core features of the CLIP pre-trained model are preserved to the greatest extent, resulting in a stable performance improvement.

[0034] Specifically, a text vector is first output through a text encoder. A text encoder is composed of transformers, in which... It consists of two parts: derived from the original features Embedding vectors and domain-specific features The embedding vector. and The format is as follows:

[0035]

[0036] In the formula, This represents an embedding vector with the same dimension as the embedded word. Simultaneously, optimization is achieved by updating the cross-entropy loss. :

[0037]

[0038] In the formula, This represents an input x in a domain s and its corresponding label y. The loss function is represented by the loss function. For label values, Text vectors representing network updates. Determined by the following formula:

[0039]

[0040] in, This indicates that the CLIP image encoder encodes the input x. This indicates that the CLIP text encoder is used to process text vector p. k Encode it.

[0041] Each time the network propagates backwards, it updates. The process continues until the entire network converges, at which point the model has found the optimal cue vector. In this implementation, the cue generator is constructed as a three-layer MLP, with each layer consisting of a linear layer, a Tahn activation function, and another linear layer, followed by a Tahn activation function. This network not only has few parameters and fast computation speed but also exhibits powerful performance in fine-tuning CLIPs.

[0042] The second step proposes a novel CLIP method to remove domain-specific features and learn better domain-invariant features, i.e., to construct a domain-invariant feature extraction model, such as... Figure 3 As shown, the domain-invariant feature extraction model in this embodiment includes an encoder-decoder network connected in sequence and a trained domain-specific classifier. To achieve this, a domain-specific classifier is first trained that can only recognize images from its specific domain and cannot classify images from other domains. Then, a mapping network is used to project the image into a new feature space, and the domain-specific classifier is used to eliminate domain-specific information. Figure 3 middle, Representing the Input images for each domain, Indicates the first A domain-specific classifier, which uses a transformer and employs a classification loss to approximate the label value of the image within that domain. The minimized classification loss is expressed as:

[0043]

[0044] After completing the individual training session Classifiers for a specific domain Then, a mapping encoder-decoder network is used to map the input image features to a high-dimensional space. The loss function is to maximize the classification loss to avoid the label values ​​of other domain images. The maximization of the classification loss is expressed as:

[0045]

[0046] in, Let i represent the data in the i-th domain, where i∈(1,…,N). This represents the input value and label value of the j-th image in the i-th domain. This represents the classification operation of a domain-specific classifier. The loss function representing classification. Represents the uncertainty loss function. Represents network parameters, Represents the i-th domain. This represents a set of domains with a total number of N. This represents the input image features of the j-th image in the i-th domain. This represents the input image label value for the j-th image in the i-th domain.

[0047] The intuitive meaning of the above operation is that the codec network M cannot recognize the first... Classifiers for a specific domain Images that can be recognized, and the first Classifiers for a specific domain Enriched with domain-specific information, in other words, the encoding / decoding network M forces the input image features to be subtracted from the domain-specific features, thus making the final output image features more robust and domain-invariant.

[0048] In general, domain-specific classifiers Together with the codec network M, it forms a domain-invariant feature extraction model. By utilizing this model to obtain more powerful domain-invariant features, and integrating it with the text optimizer, we can obtain DALSCLIP. In this way, it can be connected to subsequent modules during the training phase, thereby achieving stronger DG performance.

[0049] To verify the effectiveness of the algorithm, its performance was extensively evaluated on three publicly available benchmark datasets within the PyTorch framework, following the public settings of DG.

[0050] Experimental datasets: Office-Home contains images from four different domains: art, paper cutting, products, and the real world, with 15,588 samples across 65 categories. VLCS is another widely used public image classification benchmark. It contains four sub-datasets: Caltech10, LabelMe, SUN09, and VOC2007, with 10,729 samples across 5 classes. PACS includes four domains: art drawing, cartoons, photographs, and sketches. It has 9,991 images across 7 classes, including dogs, elephants, giraffes, guitars, horses, houses, and people.

[0051] The experiment follows a general training framework, with the dataset divided into training, validation, and test sets. In Data Generation (DG), one subset of the dataset is chosen as the target domain, i.e., the test set. The remaining subsets are the source domain, i.e., the training set. Example models are obtained from the training set, tested in the unknown target domain, and their performance is verified.

[0052] Model Evaluation Method: Following the evaluation methods for image classification tasks on publicly available datasets, the accuracy of predicted labels on the test set is reported. Since the algorithm is based on a CLIP pre-trained model, its accuracy is not derived from the final classification layer of a typical network. Instead, it learns by comparing image features with text features to derive a logical vector as the output. This method utilizes multimodal information and offers more powerful performance.

[0053] Model Details: In the experiments, the pre-trained model CLIP, with ViT-B / 16 as its backbone, was used to extract text and image features. For other algorithms, ViT-B / 16 was used as the backbone network unless otherwise specified. Following this, the entire experiment was built using a Domainbed framework. Twenty random trials were conducted on each algorithm in the test domain to find the optimal hyperparameter distribution. Each dataset was divided into two subsets: 80% was used for model training and evaluation, and the remaining 20% ​​was used to search for optimal hyperparameters. For all datasets, one domain was reserved for testing, while the rest were used for training. The entire experiment was then repeated three times with different random seeds. Finally, the mean accuracy and corresponding standard deviation of the three overall experiments are reported. For each input image, the dimension was 3*224*224. During text optimization, the vector was first enlarged to 1024*1024 dimensions, and the output mapping was restored to the original feature size. 16-bit tokens were used for the text vectors. The learning rate is dynamically adjusted to adapt to the appropriate learning rate for the current training epoch. The loss function is the standard cross-entropy function. Backpropagation uses the stochastic gradient descent (SGD) algorithm with initial parameters. The model's batch size is set to 36. The initial parameters of CLIP are completely frozen throughout the training process.

[0054] Experimental Results: Our proposed method was compared with several existing classic algorithms in two directions. First, our algorithm was extensively compared with various high-performance algorithms on the DG task, demonstrating the powerful performance of the model. Simultaneously, text hints were generated through two unique types: the standard hint "{class name photo" and the optimized hint "[v]1…[v]m[class name]". The table shows the generalization results for each task on two major DG benchmarks.

[0055]

[0056]

[0057] It is easy to see that this invention establishes a text optimizer that integrates separated domain-specific information for training, enabling text prompts to contain better semantic information. This invention also proposes a domain-invariant feature extractor that can effectively separate domain-specific information, thereby improving domain-invariant features and achieving stronger DG performance.

Claims

1. A transfer learning method for extracting domain-invariant features and optimizing text using CLIP, characterized in that, Includes the following steps: A domain-specific classifier is trained using training images, enabling the domain-specific classifier to recognize images from a specific domain. An encoder-decoder network is used to map the training images, and a trained domain-specific classifier is used to separate domain-specific information from the training images. A text optimizer is established by training images, text vectors corresponding to the training images, and information from specific domains extracted from the training images, to obtain a trained text optimizer. A domain-invariant feature extraction model is established, and the domain-invariant feature extraction model is trained using training images from different domains to obtain a trained domain-invariant feature extraction model. By integrating a pre-trained text optimizer and a pre-trained domain-invariant feature extraction model, a CLIP pre-trained model with improved generalization performance is obtained.

2. The transfer learning method for extracting domain-invariant features and optimizing text using CLIP as described in claim 1, characterized in that, The text optimizer includes: A text encoder takes the text vector corresponding to the training image and information from a specific domain extracted from the training image as input, and outputs a text vector. An image encoder takes the training image as input and outputs a matrix vector; The prompt generator takes the text vector and matrix vector as input and outputs text modality information.

3. The transfer learning method for extracting domain-invariant features and optimizing text using CLIP as described in claim 2, characterized in that, When training the text optimizer, the parameters of the text encoder and the image encoder are frozen.

4. The transfer learning method for extracting domain-invariant features and optimizing text using CLIP as described in claim 2, characterized in that, The prompt generator is constructed as a three-layer MLP, with each layer consisting of a linear layer, a Tahn activation function, and another linear layer and a Tahn activation function.

5. The transfer learning method for extracting domain-invariant features and optimizing text using CLIP as described in claim 1, characterized in that, The domain-invariant feature extraction model comprises a sequentially connected encoder-decoder network and a trained domain-specific classifier.

6. The transfer learning method for extracting domain-invariant features and optimizing text using CLIP as described in claim 5, characterized in that, The process of training the domain-invariant feature extraction model using training images from different domains specifically involves: Training images from different domains are input into a pre-trained domain-specific classifier for retraining; After the classifier for the specific domain is retrained, its parameters are frozen, and training images from different domains are input into the encoder-decoder network to train the encoder-decoder network.

7. The transfer learning method for extracting domain-invariant features and optimizing text using CLIP as described in claim 6, characterized in that, When retraining a domain-specific classifier, the loss function minimizes the classification loss to approximate the label value of the image in that domain; when training the encoder-decoder network, the loss function maximizes the classification loss to avoid the label values ​​of images from other domains.

8. The transfer learning method for extracting domain-invariant features and optimizing text using CLIP as described in claim 7, characterized in that, The minimized classification loss is expressed as: The maximization of classification loss is expressed as: ,in, This represents the data in the i-th domain. This represents the input value and label value of the j-th image in the i-th domain. This represents the classification operation of a domain-specific classifier. The loss function representing classification. Represents the uncertainty loss function. Represents network parameters, Represents the i-th domain. This represents a set of domains with a total number of N. This represents the input image features of the j-th image in the i-th domain. This represents the input image label value for the j-th image in the i-th domain.

Images