Visual language model two-stage unsupervised adaptation method, system and device and medium

Abstract

The invention discloses a visual language model two-stage unsupervised adaptation method, system and device and a medium, which are corresponding schemes, in the scheme, a high-quality auxiliary data set is constructed through adaptive retrieval, cost effectiveness and universality are achieved, and an efficient and reliable new normal form is provided for cross-domain knowledge migration; moreover, a complex adaptation task is decomposed into two stages of optimization processes, distribution of auxiliary data is controlled to be close to pre-training or target distribution through adjustable data distribution control parameters, and the single-step adaptation difficulty is remarkably reduced; meanwhile, during category filtering, samples of a specific category are screened from a large-scale image-text data set by utilizing weak supervision signals, matching frequency statistics and prediction information entropy contained in image-text pair data, and noise category samples are prevented from being introduced; in addition, based on two-stage training, the model can be helped to gradually adapt to target tasks and target data distribution, the adaptive performance is improved, and higher classification accuracy can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of image classification technology, and in particular to a two-stage unsupervised adaptation method, system, device and medium for a visual language model. Background Art

[0002] In recent years, deep neural networks (DNNs), leveraging their powerful representation learning capabilities, have achieved breakthrough progress in computer vision tasks such as image classification, object detection, and semantic segmentation. However, these models face a fundamental challenge in practical deployment: model performance degrades significantly when the training data differs from the distribution of downstream task data (target domain). The rise of large-scale pre-trained vision-language models offers a new approach to addressing this distribution shift. By pre-training image-text comparison on ultra-large-scale datasets such as LAION-400M, these models (such as CLIP and ALIGN) demonstrate remarkable robustness to distribution shifts. To cost-effectively apply large-scale pre-trained vision-language models to unlabeled downstream datasets, the current mainstream approach is unsupervised domain adaptation. Unsupervised domain adaptation utilizes auxiliary data from the source domain, which is related to the target domain and more readily available, but has a different domain. The CLIP (Contrastive Language–Image Pretraining) and ALIGN (ALarge-scale Image and Noisy-text embedding) listed here are two advanced pre-trained visual language models.

[0003] The Chinese invention patent application with publication number CN118095281A, "A method and system for identifying entities in power grid fault plans for small samples", proposes a method for processing power grid fault text based on unsupervised fine-tuning. It constructs a training set through two-step expansion of historical data (sentence template generation and syntax replacement), optimizes the power BERT model using unsupervised fine-tuning, vectorizes the fault plan text, and accurately extracts key entities based on recurrent neural networks to solve the problem of semantic modeling and generalization of domain text; the Chinese invention patent with authorization announcement number CN116913379B, "Directed protein modification based on iterative optimization of pre-trained large model sampling", A method for protein directed modification based on unsupervised fine-tuning is proposed in the paper. It uses a pre-trained large model to predict key mutation sites and generate candidate mutants, combines experimental data to screen high-potential mutants, and combines natural proteins to perform unsupervised fine-tuning on the model. Through multiple rounds of iteration, the model is continuously optimized to achieve efficient directed evolution; the Chinese invention patent application with publication number CN119741468A, "A SAR target recognition method based on unsupervised domain adaptation", proposes a SAR target recognition method based on unsupervised domain adaptation, generates augmented source domain samples through multi-level wavelet data enhancement, combines instance-prototype distribution alignment and strong and weak augmentation consistency. The Chinese invention patent application with publication number CN119722485A, “A method and system for bone imaging based on dual-view depth enhancement CT”, proposes an unsupervised domain adaptation module to adapt to different data distributions without additional annotation, realizes cross-domain learning and model generalization, and ensures the stable performance of the system in various medical scenarios. The Chinese invention patent application with publication number CN119648985A, “A method and system for target detection based on dual-path structure”, proposes an unsupervised domain adaptation module to adapt to different data distributions without additional annotation, realizes cross-domain learning and model generalization, and ensures the stable performance of the system in various medical scenarios. The method and system are applied to obtain pseudo labels with high confidence by filtering and fusing the results obtained from dual-path detection. The model is trained and fine-tuned using a dataset with pseudo labels, which can improve the recall and precision of model detection, thereby improving the detection accuracy. The Chinese invention patent application with publication number CN119169451A entitled "A hyperspectral image classification method based on multimodal unsupervised domain adaptation" proposes a multimodal hyperspectral classification method based on unsupervised domain adaptation: through pseudo label generation and visual-language cross-modal alignment, high-precision cross-domain classification is achieved without the need for target domain annotation, significantly reducing the dependence on labeled data.

[0004] However, the above unsupervised domain adaptation solutions mainly rely on given auxiliary source domain data. Unlike traditional small models, inappropriate auxiliary source domain data can easily lead to migration difficulties in large-scale pre-trained vision-language models.

[0005] In view of this, the present invention is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide a two-stage unsupervised adaptation method, system, device and medium for a visual language model, which can help the model gradually adapt to the target task and target data distribution, and improve the classification accuracy.

[0007] The purpose of the present invention is achieved through the following technical solutions: A two-stage unsupervised adaptation method for vision-language models, including: Obtain the target domain training dataset and image-text pair dataset; For each category in the target domain training dataset, multiple image-text pairs are retrieved from the image-text pair dataset using the text embedding of the category name, and an initial candidate pool is formed by integrating the image-text pairs retrieved from all categories. The initial candidate pool is then filtered using the matching frequency and the predicted information entropy in turn to obtain an auxiliary dataset for the corresponding stage. When filtering using the predicted information entropy, the feature representation of the domain-related information of the target domain training dataset and the adjustable data distribution control parameters are embedded. By configuring the adjustable data distribution control parameters, the auxiliary datasets for the first stage and the auxiliary datasets for the second stage are obtained. The adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approach the data distribution of the given pre-training data or the target domain training dataset. In the first stage, the pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset of the first stage; in the second stage, the visual language model trained in the first stage is trained using the target domain training dataset and the auxiliary dataset of the second stage.

[0008] A two-stage unsupervised adaptation system for a visual language model, used to implement the aforementioned method, includes: A dataset acquisition unit, used to acquire a target domain training dataset and an image-text pair dataset; An adaptive auxiliary dataset construction unit is used to retrieve multiple image-text pairs from the image-text pair dataset for each category in the target domain training dataset using the text embedding of the category name, form an initial candidate pool by integrating the image-text pairs retrieved from all categories, and sequentially filter using matching frequency and predicted information entropy to obtain an auxiliary dataset for the corresponding stage; wherein, when filtering using predicted information entropy, a feature representation of domain-related information of the target domain training dataset and an adjustable data distribution control parameter are embedded, and by configuring the adjustable data distribution control parameter, an auxiliary dataset for the first stage and an auxiliary dataset for the second stage are obtained; the adjustable data distribution control parameter is used to control the data distribution of the auxiliary dataset to approach the data distribution of the given pre-training data or the target domain training dataset; The two-stage training unit is used to train the pre-trained visual language model using the target domain training dataset and the auxiliary dataset of the first stage in the first stage; and to train the visual language model trained in the first stage using the target domain training dataset and the auxiliary dataset of the second stage in the second stage.

[0009] A processing device comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0011] It can be seen from the technical solution provided by the present invention that: (1) no manual labeling or pre-defined source domain is required, and a high-quality auxiliary data set is constructed through adaptive retrieval, which is both cost-effective and universal, and provides an efficient and reliable new paradigm for cross-domain knowledge transfer; (2) the complex adaptation task is decomposed into a two-stage optimization process, and the distribution of auxiliary data is controlled to be close to the pre-training or target distribution through adjustable data distribution control parameters, which significantly reduces the difficulty of single-step adaptation; (3) when filtering categories, the weak supervision signals, matching frequency statistics and predicted information entropy contained in the image-text data are used to screen samples of specific categories from large-scale image-text data sets, avoiding the introduction of noise category samples; (4) based on two-stage training, the model is helped to gradually adapt to the target task and target data distribution, which improves the adaptation performance and can obtain higher classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of a two-stage unsupervised adaptation method for a visual language model provided by an embodiment of the present invention.

[0014] Figure 2 A schematic diagram of a framework of a two-stage unsupervised adaptation method for a visual language model provided by an embodiment of the present invention.

[0015] Figure 3 A schematic diagram of a two-stage unsupervised adaptation system for a visual language model provided by an embodiment of the present invention.

[0016] Figure 4A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] First, the following terms may be used in this article: The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y includes both “X” or “Y” and “X and Y”.

[0019] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles)" should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0020] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0021] Unless otherwise specified or limited, the terms "mounted," "connected," "connect," and "fixed" should be interpreted broadly. For example, they can refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in this document based on specific circumstances.

[0022] The following describes in detail the two-stage unsupervised adaptation method, system, device, and medium for a visual language model provided by the present invention. Any content not described in detail in the embodiments of the present invention is prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, the process is performed in accordance with conventional conditions in the art or the conditions recommended by the manufacturer. Instruments used in the embodiments of the present invention, where the manufacturer is not specified, are all commercially available conventional products.

[0023] Example 1 An embodiment of the present invention provides a two-stage unsupervised adaptation method for a visual language model. Unlike existing methods, the present invention focuses on constructing an auxiliary dataset that is most conducive to the unsupervised adaptation of the visual language model, replacing the source domain in the unsupervised domain adaptation paradigm, thereby avoiding the problems of forgetting and migration failure caused by inappropriate source domains. Specifically, the present invention proposes to adaptively construct an auxiliary dataset based on the target domain, thereby decomposing the complex unsupervised adaptation into two simple adaptation processes, reducing the difficulty of adaptation. In order to screen the auxiliary dataset that is most conducive to adaptation, the present invention first proposes a category filter, which uses the weak supervision signals, matching frequency statistics and predicted information entropy contained in the image-text data to screen samples of specific categories from a large-scale image-text dataset, avoiding the introduction of noise category samples; secondly, a data distribution controller is proposed, which ensures that the data distribution of the auxiliary dataset is close to the pre-training or target domain data distribution by explicitly extracting domain-related feature representations; finally, the present invention proposes a two-stage adaptation algorithm, which uses the constructed unlabeled auxiliary dataset to help the model gradually adapt to the target task and target data distribution. As Figure 1 As shown, the method mainly includes the following: 1. Get the dataset.

[0024] In an embodiment of the present invention, a target domain training dataset and an image-text pair dataset are obtained.

[0025] 2. Adaptively construct auxiliary data.

[0026] In an embodiment of the present invention, for each category in the target domain training dataset, the text embedding of the category name is used to retrieve multiple image-text pairs in the image-text pair dataset, and the image-text pairs retrieved from all categories are integrated to form an initial candidate pool, and the matching frequency and the predicted information entropy are used in turn to filter the data to obtain the auxiliary dataset of the corresponding stage; wherein, when filtering is performed using the predicted information entropy, the feature representation of the domain-related information of the target domain training dataset and the adjustable data distribution control parameters are embedded, and by configuring the adjustable data distribution control parameters, the auxiliary dataset of the first stage and the auxiliary dataset of the second stage are obtained; the adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approach the data distribution of the given pre-training data (i.e., the data during the pre-training of the visual language model) or the data distribution of the target domain training dataset.

[0027] In an embodiment of the present invention, a category filter is constructed, and low-noise auxiliary data of a specific category is accurately constructed through triple filtering of text-to-text retrieval, matching frequency statistics, and predicted information entropy.

[0028] (1) Text-to-text retrieval.

[0029] In an embodiment of the present invention, text embeddings of category names are used to retrieve multiple image-text pairs in an image-text pair dataset, and an initial candidate pool is formed by integrating the image-text pairs retrieved from all categories, including: for each category, using text enhancement technology to generate multiple semantically consistent enhanced texts; extracting the text embedding of the category name and the text embedding of each generated enhanced text; using the text embedding of the category name and the text embedding of each text prompt, respectively performing text-to-text nearest neighbor retrieval in the image-text pair dataset to obtain multiple image-text pairs; deduplicating the image-text pairs retrieved from all categories, and forming an initial candidate pool from all the image-text pairs after deduplication.

[0030] (2) Filter using matching frequency.

[0031] According to the matching between the text description of each image-text pair in the initial candidate pool and multiple enhanced texts, the matching frequency is counted and the image-text pairs whose matching frequency is not higher than the frequency threshold are filtered. For category j, the filtering using the matching frequency is expressed as: ; in, For images With text The image-text pairs formed, is the initial candidate pool corresponding to category j, To match the frequency, is the frequency threshold, is the candidate pool corresponding to category j after filtering using matching frequency.

[0032] (3) Use prediction information entropy for filtering.

[0033] Calculate the predicted probability of the image in each image-text pair in the candidate pool after filtering using the matching frequency, and filter the image-text pairs whose information entropy of the predicted probability is not higher than the information entropy threshold. For category j, filtering using the predicted information entropy is expressed as: ; ; in, For images With text The image-text pairs formed, is the candidate pool corresponding to category j after filtering using matching frequency; p is the image in the image-text pair The predicted probability of is the image encoder, T is the transposed symbol, Softmax function (normalized exponential function), W is the parameter of the classifier composed of text embedding, is the information entropy of the predicted probability, is the information entropy threshold, is the candidate pool corresponding to category j after filtering using the predicted information entropy.

[0034] The present invention also designs a data distribution controller that controls the data distribution of the auxiliary dataset by explicitly modeling the feature representation of domain-related information and then combining it with adjustable data distribution control parameters. Specifically, the feature representation of the domain-related information of the target domain training dataset and the adjustable data distribution control parameters are embedded into the parameter W. By configuring the adjustable data distribution control parameters corresponding to the first and second stages, the weight parameters corresponding to the first stage are calculated. Weight parameters corresponding to the second stage ; Use the weight parameters corresponding to the first stage After replacing the parameter W, the candidate pool obtained That is, the auxiliary data set corresponding to category j in the first stage; the weight parameter corresponding to the second stage is used After replacing the parameter W, the candidate pool obtained This is the auxiliary data set corresponding to category j in the second stage.

[0035] The feature representation of domain-related information of the target domain training dataset is defined as s, and the adjustable data distribution control parameter is , value range ; Calculate the weight parameters corresponding to the two stages by the following formula : ; Among them, the configuration corresponding to the adjustable data distribution control parameters in the first and second stages is and , the weight parameter is calculated by the above formula and .

[0036] In the embodiment of the present invention, the feature representation of the domain-related information of the target domain training dataset is calculated as follows: for each image in the target domain training dataset, the image embedding is extracted; for each category, the text embedding of the category name is extracted; the prediction probability of each image is calculated using the image embedding and the text embedding of the category name, and a pseudo-label is assigned; based on the pseudo-label, the image embeddings and the text embedding of the category name of all images of the same category are subtracted and averaged to obtain the average difference value for the corresponding category, and the average difference value for all categories is then averaged to obtain the feature representation s of the domain-related information, which is expressed as: ; in, training dataset for the target domain, For images The pseudo labels of is the category name of category j, For text encoder, is the number of images with pseudo labels of category j, and J is the number of categories in the target domain training dataset.

[0037] 3. Complete two-stage training based on the two-stage adaptation algorithm.

[0038] In an embodiment of the present invention, in the first stage, the pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset of the first stage; in the second stage, the visual language model trained in the first stage is trained using the target domain training dataset and the auxiliary dataset of the second stage.

[0039] In an embodiment of the present invention, a teacher-student framework is configured, and the student model in the first stage is a pre-trained visual language model, and the student model in the second stage is the visual language model trained in the first stage; the parameters of the student model in both stages are optimized by gradient back propagation, and the parameters of the teacher model are updated by exponential sliding average in combination with the parameters of the student model; in both stages, the corresponding auxiliary data set is input into the student model, and the generalized cross entropy loss is calculated based on the predicted probability output by the student model; in both stages, the target domain training data set is input into the teacher model and the student model respectively, and the instance discrimination contrast loss is calculated based on the predicted probability output by the teacher model and the student model; in each stage, the parameters of the student model are optimized by gradient back propagation in combination with the generalized cross entropy loss and the instance discrimination contrast loss.

[0040] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail below with reference to specific embodiments.

[0041] 1. Overall overview of the plan.

[0042] In order to solve the performance bottleneck problem caused by static source domain data faced by existing large-scale pre-trained visual language models in unsupervised domain adaptation, the present invention proposes a two-stage unsupervised adaptation method for multimodal large models based on adaptive auxiliary data. Current unsupervised domain adaptation methods mainly rely on given auxiliary source domain data. Unlike traditional small models, inappropriate auxiliary source domain data can easily lead to migration difficulties in large-scale pre-trained visual-language models. To this end, the present invention innovatively proposes to construct an adaptive auxiliary dataset, designs a category filter, a data distribution controller and a two-stage adaptation algorithm. Based on the present invention, the difficulty of unsupervised adaptation of multimodal large models is effectively reduced, the adaptation performance is improved, and an innovative and efficient solution is provided for the unsupervised deployment of multimodal large models.

[0043] The core innovations of the solution provided by the present invention can be summarized into the following three aspects: 1. Constructing a category filter, through the triple filtering of text-to-text retrieval, matching frequency statistics and predicted information entropy, accurately constructing low-noise auxiliary data of a specific category; 2. Designing a data distribution controller, by calculating the text embedding and average image embedding of a given category, explicitly modeling the feature representation of domain-related information, thereby controlling the data distribution of the auxiliary data; 3. Proposing a two-stage adaptation algorithm, first allowing the model to overcome the differences between pre-training and target tasks, and then eliminating the differences in data distribution, using noise-resistant generalized cross entropy loss and instance-discriminative contrast loss for the auxiliary dataset and the target domain dataset respectively. Figure 2 As shown, the overall framework of the present invention is presented.

[0044] 2. Detailed introduction of the plan.

[0045] 1. Category filter.

[0046] Obtain a target domain training dataset and an image-text pair dataset. In order to accurately retrieve data of a specific category from the image-text pair dataset and avoid the influence of a large number of noise samples (image-text pairs), the present invention designs a category filter.

[0047] First, the basic retrieval method based on CLIP mainly uses unlabeled target images or class names in text form as queries, using large-scale image-text pair datasets. There are four different retrieval strategies in the joint representation space: text-to-text, image-to-image, text-to-image, and image-to-text. Although text-to-text retrieval produces a higher noise level than other retrieval methods, it has a superior ability to reduce task differences under unsupervised conditions. Therefore, the present invention uses text-to-text retrieval to extract the image-text pair data from a large-scale image-text pair dataset. Recall a large number of samples as the initial candidate pool, and for category j, its corresponding candidate pool for: ; in, For images With text The image-text pairs formed, is the category name of category j, is the text encoder, T is the transpose symbol, and TopK means recalling the top K most similar samples.

[0048] In addition, in order to enhance sample diversity, text enhancement technology is used to generate a large amount of semantically consistent enhanced text, which is then brought into the image-text pairs recalled in the above formula and added to the initial candidate pool.

[0049] Considering that the initial candidate pool constructed by text-to-text retrieval contains a lot of noise, the present invention designs advanced category filters to perform precise semantic category control, such as Figure 2 The category filter mainly targets the following two potential noise mechanisms: (1) the matching of single text to single text is not effective enough; (2) when the number of noise samples is significantly larger than the number of samples of a specific category, the cosine similarity cannot effectively capture the semantic relationship.

[0050] For (1), the present invention counts the matching frequency based on the text description of each candidate sample and multiple enhanced texts, and extracts candidate samples that match multiple enhanced texts at the same time: ; in, represents the matching frequency, is the frequency threshold.

[0051] For (2), the present invention uses predicted information entropy instead of cosine similarity and utilizes inter-class competition to screen samples with rich information content: ; ; in, represents the information entropy of the prediction probability, W is the weight of the classifier composed of text embedding, is the information entropy threshold.

[0052] 2. Data distribution controller.

[0053] To ensure that the retrieved auxiliary dataset is helpful for the adaptation process, it is also necessary to control the data distribution of the auxiliary data. To this end, the present invention proposes to take the average of the difference between the image embedding of each category in the target domain image and the text embedding of the category name, and then average it across all categories to explicitly model the feature representation of the domain-related information: ; Among them, J is the number of categories in the target domain training dataset, For images The pseudo labels of is the number of images with pseudo labels of category j, Training dataset for the target domain.

[0054] The feature representation of domain-related information can be embedded into the prediction information entropy filter to achieve control over the data distribution: ; in, , respectively bring in and The weight parameters corresponding to the two stages can be obtained and , and then embedded into the prediction information entropy filtering process to obtain the auxiliary data set of the corresponding stage.

[0055] For example, when When , the retrieved auxiliary dataset will correspond to the pre-training data distribution, and the small amount of domain-related information introduced can alleviate the text ambiguity problem; when hour, The trade-off between category filtering and data distribution control is made, and the retrieved auxiliary dataset corresponds to the target domain distribution.

[0056] 3. Two-stage adaptation algorithm.

[0057] Combining the retrieved auxiliary dataset with the original target domain dataset, the present invention designs a two-stage adaptation algorithm. Specifically, in the first stage, the The auxiliary dataset is similar to the pre-training dataset in distribution, which can help the model overcome the difference between pre-training and target tasks without considering the distribution difference; in the second stage, use The auxiliary dataset, which has a similar distribution to the target dataset, can help the model further eliminate distribution differences. The two-stage adaptation process decouples the difficult adaptation task into two simple steps, thereby improving the final performance of the model.

[0058] In an embodiment of the present invention, a large-scale pre-trained vision-language model CLIP is taken as an example for introduction. During training, a teacher-student framework is adopted. The teacher model updates parameters through exponential sliding average, and the student model is optimized through gradient backpropagation.

[0059] In terms of loss design, the pseudo labels used during retrieval (that is, the corresponding categories of the text embeddings used during retrieval) are directly used for the auxiliary datasets of the two stages, and a generalized cross entropy loss that is robust to noise is adopted. ; For the target domain dataset, based on the teacher-student framework, the dataset is trained in the target domain Compute instance discrimination contrast loss on : ; .

[0060] The logical process of calculating the contrast loss of this example can refer to conventional technology. Here is a brief explanation: For any image data in the target domain training dataset Strong enhancement and weak enhancement processing are used to obtain strong enhancement views and weak enhancement views, which are then input to the student model and the teacher model respectively to obtain the prediction probabilities of the student model and the teacher model outputs. For the same batch, the prediction probability of the strong enhancement view output by the student model is , the predicted probability of the corresponding weakly enhanced view output by the teacher model is a positive sample , the predicted probabilities of other weakly enhanced views output by the teacher model in the same batch are negative samples , , M is the negative sample set. Indicates the calculation of predicted probability The similarity score of Reference , Reference and ; is the temperature parameter, and exp(.) is the natural exponential function.

[0061] The strong enhancement and weak enhancement processing mentioned above are both general terms in this field. Strong enhancement refers to severe transformation of the image, such as randomly blocking some areas, automatically adjusting the contrast, etc.; the latter weak enhancement refers to slight transformation of the image, such as scaling, random horizontal flipping, etc.

[0062] At each stage, the overall loss function is constructed by combining the generalized cross entropy loss and the instance discrimination contrast loss, and the student model parameters are optimized to update the teacher model; the overall loss function for: ; in, For expectations, are images in the auxiliary dataset, is the corresponding pseudo label.

[0063] After completing the two-stage training, the student model is used for subsequent image classification tasks.

[0064] 3. Introduction of specific examples.

[0065] Based on the above introduction, specific examples of the method steps are provided below.

[0066] Step 1: Prepare the target domain training dataset, the large-scale pre-trained CLIP vision-language model, and the large-scale image-text pair dataset LAION-400M. Use the CLIP model to construct image and text indexes for efficient retrieval for LAION-400M, based on the Faiss library (an efficient similarity search and clustering library).

[0067] Step 2: Based on the target domain training dataset, extract domain-related feature representations. Use the CLIP model image encoder to extract image embeddings for all image samples; use the CLIP model text encoder to extract text embeddings for all category names. Calculate the prediction probability for each image and assign a pseudo-label. Based on the pseudo-labels, subtract the image embeddings and the text embeddings of the category names for all image samples of the same category and average them. This average difference is then averaged across all categories to obtain the domain-related feature representation.

[0068] Step 3: Given category j and , based on the category filter and data distribution controller, samples of specific categories and specific data distribution are retrieved from LAION-400M. Specifically, for a given category j, the text embedding corresponding to its category name is first extracted, wherein a large number of existing prompt templates are used for text enhancement. Then, using the constructed LAION-400M text index, text-to-text nearest neighbor retrieval is performed, 2000 image-text pair data are recalled for each text embedding, and all recalled image-text pair data are deduplicated to construct an initial candidate pool. For all samples in the candidate pool, the matching frequency filter and the predicted information entropy filter are used in turn for sample filtering. Among them, using the given Calculate the data distribution controller and embed it into the prediction information entropy filter to control the data distribution. Finally, collect all the retrieved samples together to form a given The auxiliary dataset of category j is .

[0069] Step 4: Repeat step 3 multiple times to traverse all categories and use two (corresponding to two stages), build a complete auxiliary dataset. Finally, given When , the auxiliary dataset contains samples of all categories, and pseudo labels are assigned to all samples according to the categories at retrieval time.

[0070] Step 5: Based on a given The generalized cross entropy loss is calculated for the auxiliary dataset. Specifically, for the batch samples sampled from the auxiliary dataset, they are input into the student model to obtain the predicted probability; the generalized cross entropy loss is calculated using the labels and predicted probabilities during the construction of the auxiliary dataset.

[0071] Step 6: Calculate the instance-discriminative contrastive loss based on the target domain batch data. Specifically, for each image in the batch, its weakly enhanced view and strongly enhanced view are input to the teacher model and student model, respectively, to obtain prediction probabilities. For any strongly enhanced sample in the batch, the predicted probability of its weakly enhanced sample is used as a positive sample, and the predicted probabilities of other weakly enhanced samples in the same batch are used as negative samples, thereby constructing a contrastive loss.

[0072] Step 7: Accumulate the loss functions of steps 5 and 6 above, minimize the loss function through the back propagation algorithm and gradient descent strategy, update the weight of the student model, and update the parameters of the teacher model through the parameters of the student model in an exponential sliding average manner. The whole training process consists of two stages. The first stage uses The auxiliary data set is used in the second stage. Auxiliary dataset.

[0073] In step 8, during the prediction phase, the target domain test dataset is input, scaled to a specified size (e.g., 224×224), and then numerically normalized. The processed images are fed into the image encoder of the student model trained in the two-stage process to obtain image embeddings. All the category nouns to be classified in text form are fed into the student model's text encoder to obtain text embeddings. The similarity between the image and text embeddings is calculated, and the image is assigned to the category with the highest similarity. The classification accuracy is then calculated.

[0074] Through the above description of the embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented via software or by utilizing software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) and includes a number of instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments of the present invention.

[0075] Example 2 The present invention also provides a two-stage unsupervised adaptation system for a visual language model, which is mainly used to implement the method provided in the above embodiment, such as Figure 3 As shown, the system mainly includes: A dataset acquisition unit, used to acquire a target domain training dataset and an image-text pair dataset; An adaptive auxiliary dataset construction unit is used to retrieve multiple image-text pairs from the image-text pair dataset for each category in the target domain training dataset using the text embedding of the category name, form an initial candidate pool by integrating the image-text pairs retrieved from all categories, and sequentially filter using matching frequency and predicted information entropy to obtain an auxiliary dataset for the corresponding stage; wherein, when filtering using predicted information entropy, a feature representation of domain-related information of the target domain training dataset and an adjustable data distribution control parameter are embedded, and by configuring the adjustable data distribution control parameter, an auxiliary dataset for the first stage and an auxiliary dataset for the second stage are obtained; the adjustable data distribution control parameter is used to control the data distribution of the auxiliary dataset to approach the data distribution of the given pre-training data or the target domain training dataset; The two-stage training unit is used to train the pre-trained visual language model using the target domain training dataset and the auxiliary dataset of the first stage in the first stage; and to train the visual language model trained in the first stage using the target domain training dataset and the auxiliary dataset of the second stage in the second stage.

[0076] Considering that the main technical details involved in the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0077] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0078] Example 3 The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0079] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0080] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example: The input device can be a touch screen, image acquisition device, physical button or mouse; The output device may be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0081] Example 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0082] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0083] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A two-stage unsupervised adaptation method for a visual language model, characterized by: include: Obtain the target domain training dataset and image-text pair dataset; For each category in the target domain training dataset, multiple image-text pairs are retrieved from the image-text pair dataset using the text embedding of the category name, and an initial candidate pool is formed by integrating the image-text pairs retrieved from all categories. The initial candidate pool is then filtered using the matching frequency and the predicted information entropy in turn to obtain an auxiliary dataset for the corresponding stage. When filtering using the predicted information entropy, the feature representation of the domain-related information of the target domain training dataset and the adjustable data distribution control parameters are embedded. By configuring the adjustable data distribution control parameters, the auxiliary datasets for the first stage and the auxiliary datasets for the second stage are obtained. The adjustable data distribution control parameters are used to control the data distribution of the auxiliary dataset to approach the data distribution of the given pre-training data or the target domain training dataset. In the first stage, the pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset of the first stage; in the second stage, the visual language model trained in the first stage is trained using the target domain training dataset and the auxiliary dataset of the second stage.

2. A two-stage unsupervised adaptation method for a visual language model according to claim 1, characterized in that: The text embedding of the category name is used to retrieve multiple image-text pairs from the image-text pair dataset. The image-text pairs retrieved from all categories are combined to form an initial candidate pool including: For each category, text augmentation technology is used to generate multiple semantically consistent augmented texts. The text embedding of the category name and the text embedding of each generated augmented text are extracted. The text embedding of the category name and the text embedding of each text prompt are used to perform text-to-text nearest neighbor retrieval in the image-text pair dataset to obtain multiple image-text pairs. The image-text pairs retrieved from all categories are deduplicated, and the initial candidate pool is formed from all the deduplicated image-text pairs.

3. The two-stage unsupervised adaptation method of a visual language model according to claim 2, characterized in that: Filtering using matching frequencies includes: According to the matching between the text description of each image-text pair in the initial candidate pool and multiple enhanced texts, the matching frequency is counted and the image-text pairs whose matching frequency is not higher than the frequency threshold are filtered. For category j, the filtering using the matching frequency is expressed as: ； in, For images With text The image-text pairs formed, is the initial candidate pool corresponding to category j, To match the frequency, is the frequency threshold, is the candidate pool corresponding to category j after filtering using matching frequency.

4. The two-stage unsupervised adaptation method of a visual language model according to claim 1, characterized in that: When using the predicted information entropy for filtering, the feature representation of the domain-related information of the target domain training dataset is embedded, and the adjustable data distribution control parameters are configured. By configuring the adjustable data distribution control parameters, the auxiliary datasets of the first stage and the auxiliary datasets of the second stage are obtained, including: Calculate the predicted probability of the image in each image-text pair in the candidate pool after filtering using the matching frequency, and filter the image-text pairs whose information entropy of the predicted probability is not higher than the information entropy threshold; for category j, filtering using the predicted information entropy is expressed as: ； ； in, For images With text The image-text pairs formed, is the candidate pool corresponding to category j after filtering using matching frequency; p is the image in the image-text pair The predicted probability of is the image encoder, T is the transposed symbol, is the normalized exponential function, W is the weight of the classifier composed of text embedding, is the information entropy of the predicted probability, is the information entropy threshold, is the candidate pool corresponding to category j after filtering using the predicted information entropy; The feature representation of the domain-related information of the target domain training dataset and the adjustable data distribution control parameters are embedded into the parameter W. By configuring the adjustable data distribution control parameters corresponding to the first and second stages, the weight parameters corresponding to the first stage are calculated. Weight parameters corresponding to the second stage ; Use the weight parameters corresponding to the first stage After replacing the parameter W, the candidate pool obtained That is, the auxiliary data set corresponding to category j in the first stage; the weight parameter corresponding to the second stage is used After replacing the parameter W, the candidate pool obtained This is the auxiliary data set corresponding to category j in the second stage.

5. The two-stage unsupervised adaptation method of a visual language model according to claim 4, characterized in that: The feature representation of the domain-related information of the target domain training data set and the adjustable data distribution control parameters are embedded into the parameter W. By configuring the adjustable data distribution control parameters corresponding to the first and second stages, the weight parameters corresponding to the first stage are calculated. Weight parameters corresponding to the second stage include: The feature representation of domain-related information of the target domain training dataset is defined as s, and the adjustable data distribution control parameter is , value range ; The weight parameters corresponding to the two stages are calculated by the following formula : ； Among them, the configuration corresponding to the adjustable data distribution control parameters in the first and second stages is and , the weight parameter is calculated by the above formula and .

6. A two-stage unsupervised adaptation method for a visual language model according to claim 1, 4 or 5, characterized in that: The feature representation of the domain-related information of the target domain training dataset is calculated as follows: For each image in the target domain training dataset, extract the image embedding; for each category, extract the text embedding of the category name; Using the image embedding and the text embedding of the category name, we calculate the predicted probability for each image and assign a pseudo label. According to the pseudo-labels, the image embeddings of all images of the same category and the text embeddings of the category names are subtracted and averaged to obtain the average difference under the corresponding category. The average difference under all categories is then averaged to obtain the feature representation s of the domain-related information, which is expressed as: ； in, training dataset for the target domain, For images The pseudo labels of is the category name of category j, For text encoder, is the number of images with pseudo labels of category j, and J is the number of categories in the target domain training dataset.

7. The two-stage unsupervised adaptation method of a visual language model according to claim 1, characterized in that: In the first stage, the pre-trained visual language model is trained using the target domain training dataset and the auxiliary dataset from the first stage; In the second stage, the visual language model trained in the first stage is trained using the target domain training dataset and the auxiliary dataset of the second stage. The training includes: Configure the teacher-student framework. The student model in the first stage is the pre-trained visual language model, and the student model in the second stage is the visual language model trained in the first stage. In both stages, the parameters of the student model are optimized by gradient backpropagation, and the parameters of the teacher model are combined with the parameters of the student model and updated by exponential sliding average; In both stages, the corresponding auxiliary datasets are input into the student model, and the generalized cross entropy loss is calculated based on the predicted probabilities output by the student model. In both stages, the target domain training datasets are input into the teacher model and the student model respectively, and the instance discrimination contrast loss is calculated based on the predicted probabilities output by the teacher model and the student model. At each stage, the parameters of the student model are optimized by gradient backpropagation using a combination of generalized cross entropy loss and instance-discriminative contrast loss.

8. A two-stage unsupervised adaptation system for a visual language model, characterized by: The method for implementing any one of claims 1 to 7 comprises: A dataset acquisition unit, used to acquire a target domain training dataset and an image-text pair dataset; An adaptive auxiliary dataset construction unit is used to retrieve multiple image-text pairs from the image-text pair dataset for each category in the target domain training dataset using the text embedding of the category name, form an initial candidate pool by integrating the image-text pairs retrieved from all categories, and sequentially filter using matching frequency and predicted information entropy to obtain an auxiliary dataset for the corresponding stage; wherein, when filtering using predicted information entropy, a feature representation of domain-related information of the target domain training dataset and an adjustable data distribution control parameter are embedded, and by configuring the adjustable data distribution control parameter, an auxiliary dataset for the first stage and an auxiliary dataset for the second stage are obtained; the adjustable data distribution control parameter is used to control the data distribution of the auxiliary dataset to approach the data distribution of the given pre-training data or the target domain training dataset; The two-stage training unit is used to train the pre-trained visual language model using the target domain training dataset and the auxiliary dataset of the first stage in the first stage; and to train the visual language model trained in the first stage using the target domain training dataset and the auxiliary dataset of the second stage in the second stage.

9. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images