Two-stage unsupervised adaptation methods, systems, devices, and media for visual language models

Abstract

This invention discloses a two-stage unsupervised adaptation method, system, device, and medium for visual language models. These are corresponding solutions. The solutions include: constructing a high-quality auxiliary dataset through adaptive retrieval, achieving both cost-effectiveness and versatility, and providing an efficient and reliable new paradigm for cross-domain knowledge transfer; furthermore, decomposing the complex adaptation task into a two-stage optimization process, controlling the distribution of auxiliary data to be similar to the pre-training or target distribution through adjustable data distribution control parameters, significantly reducing the difficulty of single-step adaptation; simultaneously, during category filtering, using weak supervision signals contained in the image-text pair data, matching frequency statistics, and predictive information entropy to filter samples of specific categories from the large-scale image-text dataset, avoiding the introduction of noisy category samples; and furthermore, based on two-stage training, the model can gradually adapt to the target task and target data distribution, improving adaptation performance and achieving higher classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of image classification technology, and in particular to a two-stage unsupervised adaptation method, system, device and medium for visual language models. Background Technology

[0002] In recent years, deep neural networks (DNNs) have made groundbreaking progress in computer vision tasks such as image classification, object detection, and semantic segmentation due to their powerful representation learning capabilities. However, these models face a fundamental challenge in practical deployment: their performance significantly degrades when there is a distributional difference between the training data and the downstream task data (target domain). The rise of large-scale pre-trained visual-language models has provided a new approach to addressing the distributional shift problem. By pre-training on ultra-large-scale datasets (such as LAION-400M) for image-text comparison, these models (such as CLIP and ALIGN) have demonstrated significant robustness to distributional changes. To cost-effectively apply large-scale pre-trained visual-language models to unlabeled downstream datasets, the current mainstream approach is unsupervised domain adaptation. Unsupervised domain adaptation uses source domain auxiliary data that is relevant to the target domain, more readily available, but with domain differences, to aid the adaptation process. CLIP (Contrastive Language–Image Pretraining) and ALIGN (A Large-scale Image and Noisy-text Embedding) are two advanced pre-trained visual language models listed here.

[0003] Chinese invention patent application CN118095281A, entitled "A Method and System for Entity Recognition in Power Grid Fault Contingency Plans for Small Samples," proposes a text processing method for power grid faults based on unsupervised fine-tuning. This method constructs a training set through two-step expansion of historical data (sentence template generation and syntactic replacement), optimizes the power BERT model using unsupervised fine-tuning, vectorizes the fault contingency plan text, and accurately extracts key entities based on a recurrent neural network, solving the challenges of semantic modeling and generalization in domain-specific texts. Chinese invention patent CN116913379B, entitled "Targeted Protein Modification Based on Iterative Optimization of Pre-trained Large Model Sampling," further supports this approach. The paper "A Method for Directed Protein Modification Based on Unsupervised Fine-tuning" proposes a method for targeted protein modification. It utilizes a pre-trained large model to predict key mutation sites and generate candidate mutants. High-potential mutants are screened using experimental data, and the model is fine-tuned in conjunction with natural proteins. Through multiple iterations, the model is continuously optimized to achieve efficient directed evolution. The Chinese invention patent application CN119741468A, "A Method for SAR Target Recognition Based on Unsupervised Domain Adaptation," proposes a similar method. This method generates augmented source domain samples through multi-level wavelet data enhancement, combining instance-prototype distribution alignment and strong / weak augmentation consistency. A feature alignment strategy, under the condition of no target domain annotation, jointly optimizes cross-domain feature alignment and model generalization ability to achieve efficient unsupervised transfer recognition of SAR images from the source domain to the target domain; Chinese invention patent application CN119722485A, "A Method and System for Bone Imaging Based on Dual-View Depth Enhancement CT", proposes an unsupervised domain adaptation module to adapt to different data distributions, without additional annotation, to achieve cross-domain learning and model generalization, ensuring the system's stable performance in various medical scenarios; Chinese invention patent application CN119648985A, "An Unsupervised Domain Adaptation Target Detection Method Based on Dual-Path Structure", proposes a method to achieve cross-domain learning and model generalization, ensuring the system's stable performance in various medical scenarios. The method and system obtained by label filtering and fusion of the results obtained by dual-path detection yielded pseudo-labels with high confidence. The model was trained and fine-tuned using the dataset with pseudo-labels, which improved the recall and precision of the model detection, thereby improving the detection accuracy. The Chinese invention patent application with publication number CN119169451A, entitled "An Unsupervised Domain Adaptive Hyperspectral Image Classification Method Based on Multimodality", proposes an unsupervised domain adaptive multimodal hyperspectral classification method: by generating pseudo-labels and aligning with visual-language cross-modality, it achieves high-precision cross-domain classification without the need for target domain annotation, significantly reducing the dependence on labeled data.

[0004] However, the above unsupervised domain adaptation schemes mainly rely on the given auxiliary source domain data. Unlike traditional small models, in large-scale pre-trained vision-language models, unsuitable auxiliary source domain data can easily lead to transfer difficulties.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a two-stage unsupervised adaptation method, system, device, and medium for visual language models, which can help the model gradually adapt to the target task and target data distribution, thereby improving classification accuracy.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A two-stage unsupervised adaptation method for visual language models includes:

[0009] Obtain the target domain training dataset and the image-text pair dataset;

[0010] For each category in the target domain training dataset, multiple image-text pairs are retrieved from the image-text pair dataset using text embeddings of the category names. These image-text pairs retrieved from all categories form an initial candidate pool, which is then filtered sequentially using matching frequency and predicted information entropy to obtain the corresponding auxiliary dataset. When filtering using predicted information entropy, feature representations of domain-related information from the target domain training dataset are embedded, along with adjustable data distribution control parameters. By configuring these adjustable data distribution control parameters, the first-stage and second-stage auxiliary datasets are obtained. These adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approximate the data distribution of the given pre-training data or the target domain training dataset.

[0011] In the first stage, the pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset from the first stage. In the second stage, the visual language model trained in the first stage is trained using the target domain training dataset and the auxiliary dataset from the second stage.

[0012] A two-stage unsupervised adaptation system for visual language models, used to implement the aforementioned method, includes:

[0013] The dataset acquisition unit is used to acquire the target domain training dataset and the image-text pair dataset;

[0014] An adaptive auxiliary dataset construction unit is used to retrieve multiple image-text pairs from the image-text pair dataset for each category in the target domain training dataset using text embeddings of the category names. The retrieved image-text pairs from all categories are combined to form an initial candidate pool, which is then filtered sequentially using matching frequency and predicted information entropy to obtain the auxiliary dataset for the corresponding stage. When using predicted information entropy for filtering, feature representations of domain-related information from the target domain training dataset are embedded, along with adjustable data distribution control parameters. By configuring these adjustable data distribution control parameters, auxiliary datasets for the first and second stages are obtained. These adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approximate the data distribution of the given pre-training data or the target domain training dataset.

[0015] The two-stage training unit is used to train the pre-trained visual language model in the first stage using the target domain training dataset and the auxiliary dataset of the first stage; and in the second stage, to train the visual language model trained in the first stage using the target domain training dataset and the auxiliary dataset of the second stage.

[0016] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0018] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0019] As can be seen from the technical solutions provided by the present invention above: (1) No manual annotation or predefined source domain is required. A high-quality auxiliary dataset is constructed through adaptive retrieval, which is both cost-effective and universal, providing an efficient and reliable new paradigm for cross-domain knowledge transfer; (2) Complex adaptation tasks are decomposed into a two-stage optimization process. The distribution of auxiliary data is controlled to be similar to the pre-training or target distribution through adjustable data distribution control parameters, which significantly reduces the difficulty of single-step adaptation; (3) When filtering categories, samples of specific categories are selected from large-scale image and text datasets by using weak supervision signals, matching frequency statistics and prediction information entropy contained in the image and text data, avoiding the introduction of noisy category samples; (4) Based on two-stage training, the model is helped to gradually adapt to the target task and target data distribution, which improves the adaptation performance and can obtain higher classification accuracy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a two-stage unsupervised adaptation method for visual language models provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the framework of a two-stage unsupervised adaptation method for visual language models provided in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of a two-stage unsupervised adaptation system for a visual language model provided in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0026] First, the following explanations are provided for the terms that may be used in this article:

[0027] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0028] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0029] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0030] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0031] The following provides a detailed description of a two-stage unsupervised adaptation method, system, device, and medium for visual language models provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0032] Example 1

[0033] This invention provides a two-stage unsupervised adaptation method for visual language models. Unlike existing methods, this invention focuses on constructing an auxiliary dataset most conducive to the unsupervised adaptation of visual language models, replacing the source domain in the unsupervised domain adaptation paradigm, thereby avoiding forgetting and transfer failure problems caused by an unsuitable source domain. Specifically, this invention proposes to adaptively construct an auxiliary dataset based on the target domain, thus decomposing the complex unsupervised adaptation into two simple adaptation processes, reducing the adaptation difficulty. To select the most conducive auxiliary dataset for adaptation, this invention first proposes a category filter, which uses weak supervision signals contained in the image-text pair data, matching frequency statistics, and predictive information entropy to filter samples of specific categories from a large-scale image-text dataset, avoiding the introduction of noisy category samples; secondly, it proposes a data distribution controller, which ensures that the data distribution of the auxiliary dataset is close to the pre-training or target domain data distribution by explicitly extracting domain-related feature representations; finally, this invention proposes a two-stage adaptation algorithm, using the constructed unlabeled auxiliary dataset to help the model gradually adapt to the target task and target data distribution. Figure 1 As shown, the method mainly includes the following:

[0034] 1. Obtain the dataset.

[0035] In this embodiment of the invention, a target domain training dataset and an image-text pair dataset are obtained.

[0036] 2. Adaptively construct auxiliary data.

[0037] In this embodiment of the invention, for each category in the target domain training dataset, multiple image-text pairs are retrieved from the image-text pair dataset using text embedding of the category name. An initial candidate pool is formed by combining the image-text pairs retrieved from all categories, and then filtered sequentially using matching frequency and predicted information entropy to obtain the corresponding auxiliary dataset. When filtering using predicted information entropy, feature representations of domain-related information from the target domain training dataset are embedded, along with adjustable data distribution control parameters. By configuring these adjustable data distribution control parameters, the first-stage auxiliary dataset and the second-stage auxiliary dataset are obtained. The adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approximate the data distribution of the given pre-training data (i.e., the data used during the pre-training of the visual language model) or the target domain training dataset.

[0038] In this embodiment of the invention, a category filter is constructed, which uses text-to-text retrieval, matching frequency statistics, and predicted information entropy triple filtering to accurately construct low-noise auxiliary data for specific categories.

[0039] (1) Text-to-text retrieval.

[0040] In this embodiment of the invention, multiple image-text pairs are retrieved from the image-text pair dataset using the text embedding of category names. The initial candidate pool is formed by combining the image-text pairs retrieved from all categories. This includes: for each category, using text enhancement techniques to generate multiple semantically consistent enhanced texts; extracting the text embedding of the category name and the text embedding of each generated enhanced text; using the text embedding of the category name and the text embedding of each text prompt to perform text-to-text nearest neighbor retrieval in the image-text pair dataset to obtain multiple image-text pairs; and deduplicating the image-text pairs retrieved from all categories, with the deduplicated image-text pairs forming the initial candidate pool.

[0041] (2) Use matching frequency for filtering.

[0042] Based on the matching results of the text description and multiple enhanced texts for each image-text pair in the initial candidate pool, the matching frequency is calculated, and image-text pairs with a matching frequency not exceeding a frequency threshold are filtered out. For category j, filtering using matching frequency is expressed as follows:

[0043] ;

[0044] in, For image With text The resulting image-text pair, This forms the initial candidate pool for category j. To match the frequency, For frequency threshold, This is the candidate pool for category j after filtering using matching frequency.

[0045] (3) Use the predicted information entropy for filtering.

[0046] Calculate the predicted probability of the image in each image-text pair in the candidate pool after filtering using matching frequency. Filter image-text pairs whose predicted probability entropy is not higher than an information entropy threshold. For category j, filtering using predicted information entropy is expressed as:

[0047] ;

[0048] ;

[0049] in, For image With text The resulting image-text pair, p represents the candidate pool for category j after filtering using matching frequencies; p is the image in the image-text pair. The predicted probability, For image encoder, T is the transpose symbol. The softmax function (normalization exponential function) is used, where W represents the parameters of the classifier constructed from the text embeddings. Information entropy for predicting probabilities, The information entropy threshold, This is the candidate pool for category j after filtering using predicted information entropy.

[0050] This invention also designs a data distribution controller, which controls the data distribution of the auxiliary dataset by explicitly modeling the feature representation of domain-related information and then combining it with adjustable data distribution control parameters. Specifically, the feature representation of domain-related information of the target domain training dataset and the adjustable data distribution control parameters are embedded into parameter W. By configuring the adjustable data distribution control parameters corresponding to the first and second stages, the weight parameters corresponding to the first stage are calculated. Weight parameters corresponding to the second stage Use the weight parameters corresponding to the first stage. The candidate pool obtained after replacing parameter W This refers to the auxiliary dataset corresponding to category j in the first stage; the weight parameters corresponding to the second stage are used. The candidate pool obtained after replacing parameter W This is the auxiliary dataset corresponding to category j in the second stage.

[0051] Define s as the domain-related feature representation of the target domain training dataset, and the adjustable data distribution control parameter is: value range The weight parameters for the two stages are calculated using the following formula. :

[0052] ;

[0053] The adjustable data distribution control parameters in the first and second stages are configured as follows: and The weight parameters are calculated using the formula above. and .

[0054] In this embodiment of the invention, the feature representation of domain-related information in the target domain training dataset is calculated as follows: For each image in the target domain training dataset, an image embedding is extracted; for each category, a text embedding of the category name is extracted; using the image embedding and the text embedding of the category name, the predicted probability of each image is calculated, and a pseudo-label is assigned; based on the pseudo-label, the difference between the image embedding and the text embedding of the category name of all images in the same category is calculated and then averaged to obtain the average difference value under the corresponding category; then, the average difference values ​​under all categories are averaged to obtain the feature representation s of the domain-related information, which is expressed as:

[0055] ;

[0056] in, For training the dataset for the target domain, For image pseudo-tags Let j be the category name. For text encoders, Let J be the number of images with pseudo-labels of category j, and J be the number of categories in the target domain training dataset.

[0057] 3. Two-stage training is completed based on the two-stage adaptation algorithm.

[0058] In this embodiment of the invention, in the first stage, a pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset of the first stage; in the second stage, the visual language model trained in the first stage is trained using the target domain training dataset and the auxiliary dataset of the second stage.

[0059] In this embodiment of the invention, a teacher-student framework is configured. The student model in the first stage is a pre-trained visual language model, and the student model in the second stage is the visual language model trained in the first stage. In both stages, the parameters of the student model are optimized through gradient backpropagation, and the parameters of the teacher model are updated by combining the parameters of the student model with the parameters of the student model through exponential moving average. In both stages, the corresponding auxiliary datasets are input into the student model, and the generalized cross-entropy loss is calculated based on the predicted probabilities output by the student model. In both stages, the target domain training dataset is input into the teacher model and the student model respectively, and the instance discrimination contrast loss is calculated based on the predicted probabilities output by the teacher model and the student model. In each stage, the parameters of the student model are optimized through gradient backpropagation by combining the generalized cross-entropy loss and the instance discrimination contrast loss.

[0060] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0061] I. Overall Overview of the Plan

[0062] To address the performance bottleneck caused by static source domain data in unsupervised adaptation of existing large-scale pre-trained visual-language models, this invention proposes a two-stage unsupervised adaptation method for multimodal large models based on adaptive auxiliary data. Current unsupervised adaptation methods primarily rely on given auxiliary source domain data. Unlike traditional small models, inappropriate auxiliary source domain data can easily lead to transfer difficulties in large-scale pre-trained visual-language models. To address this, this invention innovatively proposes constructing an adaptive auxiliary dataset, designs a category filter, a data distribution controller, and a two-stage adaptation algorithm. Based on this invention, the difficulty of unsupervised adaptation for multimodal large models is effectively reduced, and the adaptation performance is improved, providing an innovative and efficient solution for the unsupervised deployment of multimodal large models.

[0063] The core innovations of the solution provided by this invention can be summarized in the following three aspects: 1. Constructing a category filter: Through text-to-text retrieval, matching frequency statistics, and predicted information entropy triple filtering, accurately constructing low-noise auxiliary data for specific categories; 2. Designing a data distribution controller: By calculating the text embedding and average image embedding of a given category, explicitly modeling the feature representation of domain-related information, thereby controlling the data distribution of auxiliary data; 3. Proposing a two-stage adaptation algorithm: First, allowing the model to overcome the differences between pre-training and the target task, then eliminating data distribution differences, applying noise-resistant generalized cross-entropy loss and instance discrimination contrast loss to the auxiliary dataset and the target domain dataset, respectively. Figure 2 The diagram illustrates the overall framework of the present invention.

[0064] II. Detailed introduction of the plan.

[0065] 1. Category Filter.

[0066] The target domain training dataset and the image-text pair dataset are obtained. In order to accurately retrieve data of a specific category from the image-text pair dataset and avoid the influence of a large number of noisy samples (image-text pairs), this invention designs a category filter.

[0067] First, CLIP-based basic retrieval methods primarily use unlabeled target images or class names in text form as queries, utilizing large-scale image-text pair datasets. Using image or text descriptions as the values ​​to be matched, four different retrieval strategies exist in the joint representation space: text-to-text, image-to-image, text-to-image, and image-to-text. Although text-to-text retrieval produces a higher level of noise than other retrieval methods, it demonstrates superior ability to mitigate task differences under unsupervised conditions. Therefore, this invention utilizes text-to-text retrieval from large-scale image-text pair datasets. A large number of samples are recalled as the initial candidate pool, and for class j, its corresponding candidate pool is... for:

[0068] ;

[0069] in, For image With text The resulting image-text pair, Let j be the category name. For text encoder, T is the transpose symbol, and TopK represents the top K most similar samples recalled.

[0070] In addition, to enhance sample diversity, text augmentation techniques are used to generate a large number of semantically consistent augmented texts, which are then added to the initial candidate pool of image-text pairs recalled in the above formula.

[0071] Considering that the initial candidate pool constructed from text-to-text retrieval contains a large amount of noise, this invention designs an advanced category filter for precise semantic category control, such as... Figure 2 As shown. Category filters are mainly designed to address the following two potential noise mechanisms: (1) single-text matching is not effective enough; (2) when the number of noise samples is significantly greater than the number of samples in a specific category, cosine similarity cannot effectively capture semantic relationships.

[0072] For (1), the present invention calculates the matching frequency based on the matching of the text description of each candidate sample with multiple enhanced texts, and extracts candidate samples that match multiple enhanced texts simultaneously:

[0073] ;

[0074] in, Represents the matching frequency. It is the frequency threshold.

[0075] For (2), this invention uses predicted information entropy instead of cosine similarity and utilizes inter-class competition to filter samples with rich information content:

[0076] ;

[0077] ;

[0078] in, The information entropy represents the predicted probability, and W is the weight of the classifier composed of text embeddings. It is the information entropy threshold.

[0079] 2. Data distribution controller.

[0080] To ensure that the retrieved auxiliary dataset is helpful to the adaptation process, it is also necessary to control the data distribution of the auxiliary data. To this end, this invention proposes to explicitly model the feature representation of domain-related information by taking the average of the difference between the image embedding of each class and the text embedding of the class name in the target domain image, and then averaging the difference over all classes.

[0081] ;

[0082] Where J is the number of classes in the target domain training dataset. For image pseudo-tags Let j be the number of images with pseudo-labels of category j. The training dataset is for the target domain.

[0083] Domain-related feature representations can be embedded into the predictive information entropy filter, thereby enabling control over the data distribution:

[0084] ;

[0085] in, Substitute them respectively and We can obtain the weight parameters corresponding to the two stages. and This data is then embedded into the predictive information entropy filtering process to obtain the auxiliary dataset for the corresponding stage.

[0086] For example, when When the retrieved auxiliary dataset corresponds to the pre-training data distribution, the introduced small amount of domain-related information can alleviate textual ambiguity; when hour, By balancing category filtering and data distribution control, the retrieved auxiliary dataset corresponds to the target domain distribution.

[0087] 3. Two-stage adaptation algorithm.

[0088] By combining the retrieved auxiliary dataset with the original target domain dataset, this invention designs a two-stage adaptation algorithm. Specifically, in the first stage, using... The auxiliary dataset, whose distribution is similar to the pre-training dataset, can help the model overcome the differences between pre-training and the target task without considering distribution differences; in the second stage, using The auxiliary dataset, which has a similar distribution to the target dataset, can help the model further eliminate distributional differences. The two-stage adaptation process decouples the difficult adaptation task into two simple steps, thereby improving the final performance of the model.

[0089] In this embodiment of the invention, the large-scale pre-trained visual-language model CLIP is used as an example. During training, a teacher-student framework is adopted. The teacher model updates parameters through exponential moving average, and the student model is optimized through gradient backpropagation.

[0090] In terms of loss design, the pseudo-labels used during retrieval (i.e., the text embeddings used during retrieval corresponding to the categories) are directly used for the auxiliary datasets in both stages, and a noise-robust generalized cross-entropy loss is adopted. For the target domain dataset, based on the teacher-student framework, the training dataset is set up in the target domain. The above calculation example distinguishes the contrast loss. :

[0091] ;

[0092] .

[0093] The logical flow for calculating the contrast loss in this section can refer to conventional techniques, but will be briefly explained here: For any image data in the target domain training dataset... Strong and weak augmentation processing is used to obtain strongly augmented and weakly augmented views, which are then input into the student and teacher models respectively to obtain the prediction probabilities output by the student and teacher models. Within the same batch, the prediction probability of the strongly augmented view output by the student model is... The predicted probability of the corresponding weakly augmented view output by the teacher model is a positive sample. Within the same batch, the predicted probability of other weakly augmented views output by the teacher model is negative. , M is the set of negative samples. Indicates the calculation of prediction probability Similarity score, Reference , Reference and ; Let be the temperature parameter, and exp(.) be the natural exponential function.

[0094] The strong enhancement and weak enhancement processes mentioned above are both general terms in this field. Strong enhancement refers to a severe transformation of the image, such as randomly occluding parts of the image or automatically adjusting the contrast of parts; weak enhancement refers to a slight transformation of the image, such as scaling or randomly flipping the image horizontally.

[0095] Each stage combines generalized cross-entropy loss and instance discrimination contrast loss to construct an overall loss function, which is then used to optimize student model parameters and subsequently update the teacher model; the overall loss function... for:

[0096] ;

[0097] in, For the expectation, To assist the images in the dataset, This is the corresponding pseudo-tag.

[0098] After completing the two-stage training, the student model is used for subsequent image classification tasks.

[0099] III. Specific Examples

[0100] Based on the preceding introduction, the following provides a specific example of the method steps.

[0101] Step 1: Prepare the target domain training dataset, the CLIP vision-language model pre-trained on a large scale, and the large-scale image-text pair dataset LAION-400M. Use the CLIP model to build image and text indexes for efficient retrieval in LAION-400M, based on the Faiss library (an efficient similarity search and clustering library).

[0102] Step 2: Based on the target domain training dataset, extract domain-related feature representations. Use the CLIP model image encoder to extract image embeddings for all image samples; use the CLIP model text encoder to extract text embeddings for all category names. Calculate the predicted probability for each image and assign a pseudo-label. Based on the pseudo-labels, calculate the difference between the image embeddings and the text embeddings of the category names for all image samples of the same category, and then average them to obtain the average difference for the corresponding category. Finally, average the average differences for all categories to obtain the domain-related feature representations.

[0103] Step 3, given category j and Based on category filters and data distribution controllers, samples of specific categories and data distributions are retrieved from LAION-400M. Specifically, for a given category j, the text embedding corresponding to its category name is first extracted, where text enhancement is performed using a large number of existing cue templates. Then, using the constructed LAION-400M text index, text-to-text nearest neighbor retrieval is performed, recalling 2000 image-text pairs for each text embedding. All recalled image-text pairs are deduplicated to construct an initial candidate pool. For all samples in the candidate pool, a matching frequency filter and a predictive information entropy filter are used sequentially for sample filtering. Specifically, using a given... A data distribution controller is computed and embedded into a predictive information entropy filter to control the data distribution. Finally, all retrieved samples are collected to form a given... Auxiliary dataset for time category j.

[0104] Step 4: Repeat step 3 multiple times, iterating through all categories and using two methods. (Corresponding to two stages), construct the complete auxiliary dataset. Finally, given... At that time, the auxiliary dataset contains samples of all categories, and pseudo-labels are assigned to all samples according to the category at the time of retrieval.

[0105] Step 5: Based on a given... The generalized cross-entropy loss is calculated using the auxiliary dataset. Specifically, batches of samples sampled from the auxiliary dataset are fed into the student model to obtain predicted probabilities; the generalized cross-entropy loss is calculated using the labels and predicted probabilities from the auxiliary dataset construction process.

[0106] Step 6: Calculate the instance-discriminative contrast loss based on batch data of the target domain. Specifically, for each image in the batch data, its weakly enhanced view and strongly enhanced view are input into the teacher model and student model respectively to obtain the prediction probability. For any strongly enhanced sample in the batch, the prediction probability of its weakly enhanced sample is taken as a positive sample, and the prediction probabilities of other weakly enhanced samples in the same batch are taken as negative samples, thereby constructing the contrast loss.

[0107] Step 7: Accumulate the loss functions from Steps 5 and 6 above, and minimize the loss function using backpropagation and gradient descent. Update the weights of the student model, and update the parameters of the teacher model using an exponential moving average based on the student model's parameters. The entire training process consists of two phases. The first phase uses... The auxiliary dataset used in the second phase Auxiliary dataset.

[0108] Step 8, Prediction Phase: Input the test dataset of the target domain, scale it to a specified size (e.g., 224×224), and then perform numerical normalization. Input the processed image into the student model image encoder after two-stage training to obtain image embeddings. Input all the textual terms of the categories to be classified into the student model text encoder to obtain text embeddings. Calculate the similarity between the image and text embeddings, and classify the image into the category with the highest similarity. Calculate the classification accuracy.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0110] Example 2

[0111] This invention also provides a two-stage unsupervised adaptation system for visual language models, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes:

[0112] The dataset acquisition unit is used to acquire the target domain training dataset and the image-text pair dataset;

[0113] An adaptive auxiliary dataset construction unit is used to retrieve multiple image-text pairs from the image-text pair dataset for each category in the target domain training dataset using text embeddings of the category names. The retrieved image-text pairs from all categories are combined to form an initial candidate pool, which is then filtered sequentially using matching frequency and predicted information entropy to obtain the auxiliary dataset for the corresponding stage. When using predicted information entropy for filtering, feature representations of domain-related information from the target domain training dataset are embedded, along with adjustable data distribution control parameters. By configuring these adjustable data distribution control parameters, auxiliary datasets for the first and second stages are obtained. These adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approximate the data distribution of the given pre-training data or the target domain training dataset.

[0114] The two-stage training unit is used to train the pre-trained visual language model in the first stage using the target domain training dataset and the auxiliary dataset of the first stage; and in the second stage, to train the visual language model trained in the first stage using the target domain training dataset and the auxiliary dataset of the second stage.

[0115] Since the main technical details of the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0116] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0117] Example 3

[0118] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0119] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0120] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0121] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0122] The output device can be a display terminal;

[0123] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0124] Example 4

[0125] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0126] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0127] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A two-stage unsupervised adaptation method for visual language models, characterized in that, include: Obtain the target domain training dataset and the image-text pair dataset; For each category in the target domain training dataset, multiple image-text pairs are retrieved from the image-text pair dataset using text embeddings of the category names. These image-text pairs retrieved from all categories form an initial candidate pool, which is then filtered sequentially using matching frequency and predicted information entropy to obtain the corresponding auxiliary dataset. When filtering using predicted information entropy, feature representations of domain-related information from the target domain training dataset are embedded, along with adjustable data distribution control parameters. By configuring these adjustable data distribution control parameters, the first-stage and second-stage auxiliary datasets are obtained. These adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approximate the data distribution of the given pre-training data or the target domain training dataset. In the first stage, the pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset from the first stage. In the second stage, the visual language model trained in the first stage is trained using the target domain training dataset and the auxiliary dataset from the second stage.

2. The two-stage unsupervised adaptation method for visual language models according to claim 1, characterized in that, The step of retrieving multiple image-text pairs from the image-text pair dataset using text embeddings of category names, and combining the image-text pairs retrieved from all categories to form an initial candidate pool includes: For each category, text augmentation techniques are used to generate multiple semantically consistent augmented texts; the text embeddings of the category names and each generated augmented text are extracted; using the text embeddings of the category names and each text prompt, text-to-text nearest neighbor retrieval is performed in the image-text pair dataset to obtain multiple image-text pairs. The image-text pairs retrieved from all categories are deduplicated, and the deduplicated image-text pairs form the initial candidate pool.

3. The two-stage unsupervised adaptation method for visual language models according to claim 2, characterized in that, Filtering using matched frequencies includes: Based on the matching results of the text description and multiple enhanced texts for each image-text pair in the initial candidate pool, the matching frequency is calculated, and image-text pairs with a matching frequency not exceeding a frequency threshold are filtered out. For category j, the filtering using matching frequency is expressed as follows: ； in, For image With text The resulting image-text pair, This forms the initial candidate pool for category j. To match the frequency, For frequency threshold, This is the candidate pool for category j after filtering using matching frequency.

4. The two-stage unsupervised adaptation method for visual language models according to claim 1, characterized in that, When using predicted information entropy for filtering, the feature representation embedding domain-related information of the target domain training dataset, and adjustable data distribution control parameters, are used to obtain the first-stage auxiliary dataset and the second-stage auxiliary dataset by configuring the adjustable data distribution control parameters. Calculate the predicted probability of the image in each image-text pair in the candidate pool after filtering using matching frequency, and filter image-text pairs whose predicted probability information entropy is not higher than the information entropy threshold; for category j, filtering using predicted information entropy is expressed as: ； ； in, For image With text The resulting image-text pair, p represents the candidate pool for category j after filtering using matching frequencies; p is the image in the image-text pair. The predicted probability, For image encoder, T is the transpose symbol. Let W be the normalized exponential function, and W be the weights of the classifier composed of text embeddings. Information entropy for predicting probabilities, The information entropy threshold, This is the candidate pool for category j after filtering using predicted information entropy. The domain-related feature representations of the target domain training dataset and adjustable data distribution control parameters are embedded into parameter W. By configuring the adjustable data distribution control parameters corresponding to the first and second stages, the weight parameters corresponding to the first stage are calculated. Weight parameters corresponding to the second stage Use the weight parameters corresponding to the first stage. The candidate pool obtained after replacing parameter W This refers to the auxiliary dataset corresponding to category j in the first stage; the weight parameters corresponding to the second stage are used. The candidate pool obtained after replacing parameter W This is the auxiliary dataset corresponding to category j in the second stage.

5. The two-stage unsupervised adaptation method for visual language models according to claim 4, characterized in that, The feature representation of domain-related information of the target domain training dataset and the adjustable data distribution control parameters are embedded into parameter W. By configuring the adjustable data distribution control parameters corresponding to the first and second stages, the weight parameters corresponding to the first stage are calculated. Weight parameters corresponding to the second stage include: Define s as the domain-related feature representation of the target domain training dataset, and the adjustable data distribution control parameter is: value range ; The weight parameters corresponding to the two stages are calculated using the following formula. : ； The adjustable data distribution control parameters in the first and second stages are configured as follows: and The weight parameters are calculated using the formula above. and .

6. A two-stage unsupervised adaptation method for visual language models according to claim 1, 4, or 5, characterized in that, The domain-related feature representations of the target domain training dataset are calculated as follows: For each image in the target domain training dataset, extract the image embedding; for each category, extract the text embedding of the category name. Using image embeddings and text embeddings of category names, calculate the predicted probability for each image and assign pseudo-labels; Based on the pseudo-labels, the image embeddings and text embeddings of the category names of all images in the same category are subtracted and then averaged to obtain the average difference value under the corresponding category. Then, the average difference values ​​under all categories are averaged to obtain the feature representation s of domain-related information, which is expressed as: ； in, For training the dataset for the target domain, For image pseudo-tags Let j be the category name. For text encoders, Let J be the number of images with pseudo-labels of category j, and J be the number of categories in the target domain training dataset.

7. The two-stage unsupervised adaptation method for visual language models according to claim 1, characterized in that, In the first stage, the pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset from the first stage. The second stage involves training the visual language model trained in the first stage using the target domain training dataset and the auxiliary dataset from the second stage. This includes: Configure the teacher-student framework. The student model in the first stage is a pre-trained visual language model, and the student model in the second stage is the visual language model trained in the first stage. In both stages, the parameters of the student model are optimized through gradient backpropagation, while the parameters of the teacher model are updated by combining the parameters of the student model through exponential moving average. In both stages, the corresponding auxiliary datasets are input into the student model, and the generalized cross-entropy loss is calculated based on the predicted probabilities output by the student model. In both stages, the target domain training datasets are input into the teacher model and the student model, respectively, and the instance discrimination contrast loss is calculated based on the predicted probabilities output by the teacher model and the student model. At each stage, the parameters of the student model are optimized by combining the generalized cross-entropy loss and the instance discrimination contrast loss through gradient backpropagation.

8. A two-stage unsupervised adaptive system for visual language models, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The dataset acquisition unit is used to acquire the target domain training dataset and the image-text pair dataset; An adaptive auxiliary dataset construction unit is used to retrieve multiple image-text pairs from the image-text pair dataset for each category in the target domain training dataset using text embeddings of the category names. The retrieved image-text pairs from all categories are combined to form an initial candidate pool, which is then filtered sequentially using matching frequency and predicted information entropy to obtain the auxiliary dataset for the corresponding stage. When using predicted information entropy for filtering, feature representations of domain-related information from the target domain training dataset are embedded, along with adjustable data distribution control parameters. By configuring these adjustable data distribution control parameters, auxiliary datasets for the first and second stages are obtained. These adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approximate the data distribution of the given pre-training data or the target domain training dataset. The two-stage training unit is used to train the pre-trained visual language model in the first stage using the target domain training dataset and the auxiliary dataset of the first stage; and in the second stage, to train the visual language model trained in the first stage using the target domain training dataset and the auxiliary dataset of the second stage.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images