An unsupervised domain adaptation method and system based on multimodal category centers

Abstract

This application provides an unsupervised domain adaptation method and system based on multimodal category centers. The method includes: acquiring a multi-source unsupervised domain adaptation dataset; preprocessing the dataset and dividing it into a training set and a test set; wherein the training set is source domain data and the test set is target domain data; generating multimodal category centers based on the training set; training a pre-constructed visual-language CLIP model based on the training set, test set, and multimodal category centers, and optimizing the CLIP model using a total loss function to obtain the final domain adaptation model; and training and evaluating the domain adaptation model using the test set. The unsupervised domain adaptation method based on cue affinity and multimodal category centers (PMCC) proposed in this application effectively measures the semantic similarity between source and target domain data without requiring target domain labeled data.

Description

Technical Field

[0001] This invention relates to the field of domain adaptation technology, and in particular to an unsupervised domain adaptation method and system based on multimodal category centers. Background Technology

[0002] Unsupervised domain adaptation (UDA) refers to a method of transferring a model from the source domain to the target domain when there is no labeled data in the target domain. In the field of unsupervised domain adaptation (UDA), existing techniques mainly focus on reducing the distributional differences between the source and target domains through various strategies, including feature alignment, generative adversarial networks (GANs), and leveraging large vision-language models (VLMs). However, these methods often encounter challenges when dealing with complex, high-dimensional data, especially when domain bias is significant. Furthermore, existing methods may struggle to fully utilize the semantic information in multimodal data, leading to limitations in knowledge transfer between different domains.

[0003] For example, some methods rely on feature alignment techniques, such as maximum mean difference (MMD) and maximum density divergence (MDD), to minimize the distributional differences between the source and target domains. While these methods have achieved some success in certain situations, they may struggle to capture complex semantic variations, especially when there are significant semantic differences between the source and target domains.

[0004] In addition, Generative Adversarial Networks (GANs) have also been used in UDAs to learn domain-invariant features through adversarial training between the generator and the discriminator. However, the training of GANs may be unstable and may be difficult to effectively transfer semantic information between different domains.

[0005] Recently, large-scale vision-language models, such as the multimodal pre-trained model CLIP, have attracted attention due to their powerful performance on both image and text modalities. These models, trained through contrastive learning, are able to learn joint representations between images and text, which has potential value for maintaining semantic consistency in UDA. However, how to effectively utilize these models for UDA, especially in the absence of labeled data in the target domain, remains an open research question. Summary of the Invention

[0006] The purpose of this invention is to provide an unsupervised domain adaptation method and system based on multimodal category centers, aiming to solve the above-mentioned problems in the prior art.

[0007] This invention provides an unsupervised domain adaptation method based on multimodal category centers, comprising:

[0008] A multi-source unsupervised domain adaptation dataset is obtained, the unsupervised domain adaptation dataset is preprocessed, and the processed data is divided into a training set and a test set; wherein, the training set is source domain data, and the test set is target domain data;

[0009] Generate multimodal category center points based on the training set;

[0010] The pre-constructed visual-language CLIP model is trained based on the training set, test set, and multimodal category centroids, and then optimized using the total loss function to obtain the final domain adaptation model; and

[0011] The domain adaptation model is trained and evaluated using the test set.

[0012] This invention provides an unsupervised domain adaptation system based on multimodal category centers, comprising:

[0013] The data module is used to acquire a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data and the test set is target domain data.

[0014] The center point module is used to generate multimodal category center points based on the training set;

[0015] The training module is used to train a pre-constructed vision-language CLIP model based on the training set, test set, and multimodal category centroids, and to optimize the CLIP model using the total loss function to obtain the final domain adaptation model; and

[0016] An evaluation module is used to train and evaluate the domain adaptation model using the test set.

[0017] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the above-described unsupervised domain adaptation method based on multimodal category centers.

[0018] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described unsupervised domain adaptation method based on multimodal category centers.

[0019] The embodiments of this invention can include the following beneficial effects: This invention proposes an unsupervised domain adaptation method based on prompt-affinity multi-modal class centroids (PMCC). This method fully utilizes the semantic information of images and text in large-scale vision-language models, maintains semantic consistency between different domains by constructing multi-modal class centroids, and effectively measures the semantic similarity between source and target domain data by fusing text and image class centroids without requiring target domain labeled data. Furthermore, this invention also introduces a multi-modal prompt learning mechanism to accurately predict the true category of each image in the source and target domains. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the unsupervised domain adaptation method based on multimodal category centers according to an embodiment of the present invention;

[0022] Figure 2 This is a general flowchart of an embodiment of the present invention;

[0023] Figure 3 This is a PMCC framework diagram according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of an unsupervised domain adaptation system based on multimodal category centers according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0026] Method Implementation Examples

[0027] According to embodiments of the present invention, an unsupervised domain adaptation method based on multimodal category centers is provided. Figure 1 This is a flowchart of the unsupervised domain adaptation method based on multimodal category centers according to an embodiment of the present invention, as follows: Figure 1 As shown, the unsupervised domain adaptation method based on multimodal category centers according to an embodiment of the present invention specifically includes:

[0028] Step S101: Obtain a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein, the training set is source domain data, and the test set is target domain data;

[0029] Step S102, generating multimodal category center points based on the training set, specifically includes:

[0030] The CLIP model is used to predict the center points of each image in the source domain data. Several samples with the highest prediction probability are selected to calculate the center points of the image class. The manually created template in the CLIP model is used as the center points of the text class. The image class center points and the text class center points are linearly fused to generate multimodal class center points.

[0031] Step S103 involves training the pre-constructed visual-language CLIP model based on the training set, test set, and multimodal category centroids, and optimizing the CLIP model using the total loss function to obtain the final domain adaptation model. Specifically, this includes:

[0032] The source domain data and target domain data are input into the CLIP model for training. The text input and learnable text prompts are concatenated and then processed by the CLIP text encoder to obtain the prompt affinity text features. The learnable visual prompts are concatenated with the source domain image and the target domain image respectively, and then processed by the CLIP image encoder to obtain the prompt affinity source domain image features and prompt affinity target domain image features.

[0033] Calculate the cosine similarity between the source domain image features and the source domain text features of cue affinity to obtain the source domain cosine similarity matrix, and calculate the cross-entropy loss between the source domain cosine similarity matrix and the true label.

[0034] The target domain image is processed through the CLIP model to obtain target domain image features. The cosine similarity between the target domain image features and the cue affinity text features is calculated, and the one with the largest cosine similarity is taken as the pseudo label of the target domain image. The cosine similarity between the cue affinity target domain image features and the cue affinity text features is calculated to obtain the target domain cosine similarity matrix. The cross-entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated.

[0035] The learnable visual cue includes a front part and a back part;

[0036] The first part is generated by learning text prompts through a linear projection layer;

[0037] The latter part is generated independently;

[0038] Attention features are calculated between the source domain image features and target domain image features of cue affinity and the multimodal category centroids to obtain enhanced source domain image features and target domain image features. The enhanced source domain image features and target domain image features are then compared with the cue affinity text features to calculate cosine similarity, resulting in enhanced source domain cosine similarity matrices and enhanced target domain cosine similarity matrices. The cross-entropy loss between the enhanced source domain cosine similarity matrix and the real label, as well as the cross-entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label, are also calculated.

[0039] The CLIP model is optimized based on the total loss function shown in Equation 1 for each cross-entropy loss, resulting in the final domain adaptation model.

[0040]

[0041] in, The cross-entropy loss represents the cross-entropy loss between the source domain cosine similarity matrix and the true label. The cross-entropy loss represents the cross-entropy between the target domain cosine similarity matrix and the pseudo-label. , representing the cross-entropy losses calculated between the enhanced source domain data and the real label, and between the target domain data and the pseudo label after attention calculation, respectively, with β being the balance factor;

[0042] Step S104: Train and evaluate the domain adaptation model using the test set.

[0043] The following describes the specific details of the unsupervised domain adaptation method based on multimodal category centers according to embodiments of the present invention, such as... Figure 2 As shown, the above technical solutions of the embodiments of the present invention will be described in detail.

[0044] To address the problems and shortcomings of existing technologies, the PMCC method proposed in this invention mainly solves the following technical problems:

[0045] 1. Redundant and unreliable target class centers: Existing methods rely on pseudo-labels when generating class centers for the target domain. This means that new class centers need to be generated whenever the target domain changes, and the reliability of these class centers depends on the confidence level of the pseudo-labels. PMCC addresses this issue by discarding unreliable target class centers and directly using hand-crafted prompts as text centers.

[0046] 2. Lack of Modal Class Centers: Traditional methods only consider image features when generating class centers, ignoring textual feature information from Visual Language Models (VLMs). This leads to the loss of textual modality and fails to fully utilize the multimodal advantages of VLMs. The PMCC method proposed in this embodiment of the invention, based on the CLIP architecture, solves this problem by fusing the textual and image semantic information of each class to construct multimodal class centers.

[0047] 3. Excessive and unnecessary cross-attention computational overhead: Existing solutions require two similarity measurements between different types of class centers for each input image, resulting in significant computational overhead. PMCC devises a novel method to remove target domain center points, as these are unbelievable, thereby reducing unnecessary computational overhead on target domain center points and allowing the input to be closer to the multimodal center.

[0048] With the above improvements, PMCC can better preserve the semantic consistency between the source and target domains in unsupervised domain adaptation tasks, and demonstrates performance that surpasses the current state-of-the-art methods on three popular UDA datasets.

[0049] The embodiments of the present invention specifically include the following steps:

[0050] The first step is to obtain three publicly available unsupervised domain adaptation datasets: OfficeHome, Office31, and VisDA17. After preprocessing these datasets, they are divided into training and testing sets.

[0051] 1.1 Download the OfficeHome, Office31, and VisDA17 datasets from OfficeHomeDataset-10072016.zip-Google Drive, Domain Adaptation-UC Berkeley, and VisDA2017: Visual Domain Adaptation Challenge, respectively.

[0052] 1.2 The dataset is divided into a training set and a test set, that is, the source domain data is used as the training set and the target domain data is used as the test set.

[0053] The second step is to construct the multimodal class centroids for each dataset.

[0054] 2.1 Setting up a deep learning environment. Install Anaconda on the server and create a virtual environment. Set up PyTorch in the virtual environment, and then install the necessary packages such as scipy, numpy, dassl, matplotlib, and clip.

[0055] 2.2 Multimodal center points are generated by constructing image class center points and text class center points.

[0056] 2.3 Construction of Image Class Centers: To avoid erroneous classification due to the uncertainty of pseudo-labels, this embodiment of the invention only uses labeled source domain data to calculate image class centers. CLIP is used to predict each image, and the top-K samples with the highest probability for each class are selected to calculate the center points, resulting in a image class centers, denoted as S = {s1, s2, ..., s...} a}

[0057] 2.4 Construction of Text Class Center Points: In this embodiment of the invention, a manually created "a photo of a [CLASS]" is directly used as the text class center point, thus obtaining b text class center points, denoted as T = {t1, t2, ..., t...} b}

[0058] 2.5 Linearly fuse the obtained image and text class center points using 0.5*S + 0.5*T to obtain k multimodal class center points M = {m1, m2, ..., m}. k The multimodal class center points obtained through linear fusion contain both image and semantic information, which can better align the source and target domains.

[0059] The third step is to construct an unsupervised domain adaptation method based on cue affinity multimodal category centers.

[0060] 3.1 Feature Extraction of Affinity Text: Learnable cues P = {p1, p2, ..., p...} of length L are concatenated and combined using the concat operation. L} and text input T = {t1, t2, ..., t b That is, concat(T,P)={t1,t2,...,t b ,p1,p2,...,p L The text features are learned to enhance the transferability of the model. This can improve the semantic information in the text features by learning only a small number of parameters P without changing the CLIP text encoder parameters, and obtain affinity text features. For c classes, the text input is set to "a photo of a[CLASS]", where CLASS is the class name of each class.

[0061] 3.2 Next, a series of learnable cues and manual cues are introduced and combined, that is, a learnable text cues P = {p1, p2, ..., p...} of length L are concatenated and combined using the concat operation. L} and text input T = {t1, t2, ..., tb That is, concat(T,P)={t1,t2,...,t b ,p1,p2,...,p L}, which is then fed into the frozen CLIP text encoder to obtain text features of c classes.

[0062] 3.3 Hint: Affinity Image Feature Extraction: For each input source domain image X s ={x1,x2,...,x N ,x N+1} and target domain image X t ={x1,x2,...,x N+1}, where N is the number of patches in the image segmentation, and the first x1 is the additional cls token vector. This embodiment of the invention uses a learnable visual cue V = {v1, v2, ..., v} of length L. L} and input image X s X t We use concat to learn representational information in images.

[0063] 3.4 Input the image into V = {v1, v2, ..., v} L The concat operator obtains {x1, x2, ..., x}. N ,x N+1 ,v1,v2,...,v L The input to the frozen CLIP image encoder ultimately yields the affinity image features. In the first few layers of the encoder, the visual cues V = {v1, v2, ..., v} L} is indicated by the text prompt P = {p1, p2, ..., p L The image is generated through a linear projection layer Proj(P), which helps to incorporate more textual semantic information into the image, and in the later layers of the encoder, visual cues V = {v1, v2, ..., v}. L The} is generated independently, which is beneficial for obtaining richer and more independent prompts.

[0064] 3.5 Cosine similarity over the source domain, for each input labeled source domain image data X s Extract its affinity image features (where P represents multimodal cues and s represents the source domain) and c class text features. Calculate the cosine similarity between image features and text features. Where logit_scale is the similarity scaling factor, the similarity matrix logit is obtained, and the cross-entropy loss is calculated with the true label.

[0065] 3.6 Calculate pseudo-labels for target domain image classification, enabling unlabeled images to participate in training and making better use of the data. Input X using an unlabeled target domain image. t ={x1,x2,...,x N+1 Image features are obtained by freezing the image encoder and text encoder in CLIP. and text features Calculate cosine similarity The class with the highest probability is selected as the pseudo-label_p of the target domain image.

[0066] 3.7 Cosine Similarity over the Target Domain: For each input unlabeled target domain image data X t Extract its affinity image features (where P represents multimodal cues and t represents the target domain) and c class text features. Calculate the cosine similarity between image features and text features. Where logit_scale is the similarity scaling factor, the cross-entropy loss is calculated after obtaining the similarity matrix and the pseudo-label.

[0067] 3.8 The above image features For the multimodal class center points M = {m1, m2, ..., m k Perform cross-attention calculation:

[0068]

[0069] K,V=f pre (M) (3);

[0070] Among them, f pre (·) denotes a linear projection layer, Q s Q t Let K and V represent the query vectors of the source sequence and the target sequence, respectively, and let K and V represent the key vector and value vector obtained by projecting the multimodal class centroid M.

[0071] Calculate attention score:

[0072]

[0073] In the formula, softmax() is the softmax function, and attn_score s The attention score, attn_score, represents the source domain features. t K represents the attention score for the target domain features. Trepresents the transpose of the key vector, and ∈ represents the temperature parameter used to control the smoothness of the softmax function.

[0074] Calculate attention features:

[0075] z s =f post (attn_score s · V) (6);

[0076] z t =f post (attn_score t · V) (7);

[0077] In the formula, z s z t These represent the enhanced source domain features and target domain features, respectively, after attention computation.

[0078] Finally, the similarity matrix of the attention features between the source and target domains is calculated:

[0079] logit s = logit_scale·z s ·W (8);

[0080] logit t = logit_scale·z t ·W (9);

[0081] In the formula, logit s ,logit t These represent the similarity matrices calculated from the enhanced source domain features and target domain features with the text features, respectively. logit_scale represents the scaling factor used to adjust the values ​​of the similarity matrices of the attention features.

[0082] And calculate attention loss:

[0083]

[0084] In the formula, cross_entropy() represents the cross-entropy function. These represent the cross-entropy loss values ​​calculated after attention calculation, obtained by comparing the source domain data with the real label and the target domain data with the pseudo label.

[0085] 3.9 Calculate the total loss function

[0086]

[0087] Where β is a balancing factor used to control the impact of the attention loss obtained from the attention operation on the multimodal class center on training.

[0088] The fourth step is to train the PMCC visual language large model and evaluate the model, such as... Figure 3 As shown

[0089] 4.1 Training was performed on four NVIDIA A40 GPUs. The image feature extractor backbone of CLIP was selected from ResNet50, ResNet101, and ViT / B-16, respectively, and the text encoder used a transformer.

[0090] 4.2 The batch size was set to 32. The pseudo-label thresholds for the three datasets OfficeHome, Office31 and VisDA17 were set to 0.9, 0.8 and 0.6 respectively, and the epoch sizes were 10, 20 and 10 respectively.

[0091] 4.3 Set the Top-K of the multimodal center point construction, select the number K as 5, and the balance factor β as 0.1.

[0092] 4.4 Train the model, save the model with the highest accuracy, and finally train it on the test set to finish.

[0093] In summary, the PMCC method proposed in this invention aims to overcome the limitations of existing technologies by constructing multimodal class centers and more effectively measure the similarity between the source and target domains by fusing semantic information from text and images. This method not only leverages the multimodal advantages of VLMs but also improves the model's adaptability and generalization ability across different domains through cross-attention mechanisms and multimodal cue learning. Experiments on popular UDA datasets such as Office-31, Office-Home, and VisDA-2017 demonstrate PMCC's superior performance and low model complexity, proving its effectiveness and practicality in UDA tasks. The main components of this invention include the following:

[0094] 1. Construction of Multimodal Class Centroids: A novel method for constructing multimodal class centroids is proposed. This method obtains multimodal class centroids with rich semantic information by fusing text class centroids (generated directly by CLIP text encoder based on manual prompts for each category) and image class centroids (generated by CLIP image encoder based on source domain image data).

[0095] 2. Cross-attention mechanism: The cross-attention mechanism is innovatively introduced, which allows each image in the source or target domain to interact with the above-mentioned multimodal class center, thus serving as a bridge to effectively measure the semantic similarity between different domains.

[0096] 3. Multi-modal prompt learning mechanism: A multi-modal prompt learning mechanism is proposed to accurately predict the true categories of source and target domain images, thereby achieving better performance in unsupervised domain adaptation tasks.

[0097] The above content aims to address problems encountered in existing technologies, such as limitations in quantifying the similarity between source and target domain data, and challenges in maintaining semantic consistency in unsupervised domain adaptation tasks. Using the methods described above, PMCC demonstrates performance exceeding state-of-the-art results on three popular unsupervised domain adaptation datasets.

[0098] System Implementation Examples

[0099] According to embodiments of the present invention, an unsupervised domain adaptation system based on multimodal category centers is provided. Figure 4 This is a schematic diagram of an unsupervised domain adaptation system based on multimodal category centers according to an embodiment of the present invention, as shown below. Figure 4 As shown, the unsupervised domain adaptation system based on multimodal category centers according to an embodiment of the present invention specifically includes:

[0100] Data module 40 is used to acquire a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data and the test set is target domain data;

[0101] Center point module 42 is used to generate multimodal category center points based on the training set, specifically for:

[0102] The CLIP model is used to predict the center points of each image in the source domain data. Several samples with the highest prediction probability are selected to calculate the center points of the image class. The manually created template in the CLIP model is used as the center points of the text class. The image class center points and the text class center points are linearly fused to generate multimodal class center points.

[0103] Training module 44 is used to train the pre-constructed visual-language CLIP model based on the training set, test set, and multimodal category centroids, and to optimize the CLIP model using the total loss function to obtain the final domain adaptation model. Specifically, it is used for:

[0104] The source domain data and target domain data are input into the CLIP model for training. The text input and learnable text prompts are concatenated and then processed by the CLIP text encoder to obtain the prompt affinity text features. The learnable visual prompts are concatenated with the source domain image and the target domain image respectively, and then processed by the CLIP image encoder to obtain the prompt affinity source domain image features and prompt affinity target domain image features.

[0105] Calculate the cosine similarity between the source domain image features and the source domain text features of cue affinity to obtain the source domain cosine similarity matrix, and calculate the cross-entropy loss between the source domain cosine similarity matrix and the true label.

[0106] The target domain image is processed through the CLIP model to obtain target domain image features. The cosine similarity between the target domain image features and the cue affinity text features is calculated, and the one with the largest cosine similarity is taken as the pseudo label of the target domain image. The cosine similarity between the cue affinity target domain image features and the cue affinity text features is calculated to obtain the target domain cosine similarity matrix. The cross-entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated.

[0107] Attention features are calculated between the source domain image features and target domain image features of cue affinity and the multimodal category centroids to obtain enhanced source domain image features and target domain image features. The enhanced source domain image features and target domain image features are then compared with the cue affinity text features to calculate cosine similarity, resulting in enhanced source domain cosine similarity matrices and enhanced target domain cosine similarity matrices. The cross-entropy loss between the enhanced source domain cosine similarity matrix and the real label, as well as the cross-entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label, are also calculated.

[0108] The CLIP model is optimized based on the total loss function shown in Equation 1 for each cross-entropy loss, resulting in the final domain adaptation model.

[0109]

[0110] in, The cross-entropy loss represents the cross-entropy loss between the source domain cosine similarity matrix and the true label. The cross-entropy loss represents the cross-entropy between the target domain cosine similarity matrix and the pseudo-label. , representing the cross-entropy losses calculated between the enhanced source domain data and the real label, and between the target domain data and the pseudo label after attention calculation, respectively, with β being the balance factor;

[0111] Evaluation module 46 is used to train and evaluate the domain adaptation model using the test set.

[0112] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0113] This invention discloses an unsupervised domain adaptation method based on cue affinity multimodal category centroids. First, three publicly available domain adaptation datasets are acquired, and a large visual-language model is trained on the source domains of these three datasets. Finally, the test results on the test set show improvements over state-of-the-art unsupervised domain adaptation methods, while reducing model complexity and enhancing generalization ability and robustness. The specific benefits of using this invention include the following:

[0114] 1. Reduced computational costs: By avoiding unnecessary cross-attention computational overhead, PMCC can reduce the demand for computing resources, thereby saving hardware costs;

[0115] 2. Reduced need for manual annotation: Since PMCC is an unsupervised domain adaptation method, it does not require labeled data of the target domain, thus reducing the cost and time consumption of manual annotation;

[0116] 3. Promotes the effective use of resources: PMCC can effectively transfer knowledge between different fields, which helps to promote the use of machine learning models in application scenarios where it is difficult to obtain labeled data;

[0117] 4. Promoted technological progress: As an innovative method in the field of unsupervised domain adaptation, PMCC has provided new ideas for related research and promoted technological progress and development in this field.

[0118] In conclusion, PMCC not only improves existing methods from a technical perspective, but also has a positive impact on economic and social aspects, demonstrating high application value and development potential.

[0119] Device Example 1

[0120] This invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps described in the method embodiment.

[0121] Device Example 2

[0122] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, performs the steps described in the method embodiment.

[0123] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised domain adaptation method based on multimodal category centers, characterized in that... include: A multi-source unsupervised domain adaptation dataset is obtained, the unsupervised domain adaptation dataset is preprocessed, and the processed data is divided into a training set and a test set; wherein, the training set is source domain data, and the test set is target domain data; Generate multimodal category center points based on the training set; The pre-built visual-language CLIP model is trained based on the training set, test set, and multimodal category centroids, and then optimized using the total loss function to obtain the final domain adaptation model, specifically including: The source domain data and target domain data are input into the CLIP model for training. The input text and learnable text cues are concatenated and then processed by the CLIP text encoder to obtain cue affinity text features. The learnable visual cues are concatenated with the source domain image and the target domain image respectively, and then processed by the CLIP image encoder to obtain cue affinity source domain image features and cue affinity target domain image features. Calculate the cosine similarity between the source domain image features and the source domain text features of cue affinity to obtain the source domain cosine similarity matrix, and calculate the cross-entropy loss between the source domain cosine similarity matrix and the true label. The target domain image is processed through the CLIP model to obtain target domain image features. The cosine similarity between the target domain image features and the cue affinity text features is calculated, and the one with the largest cosine similarity is taken as the pseudo label of the target domain image. The cosine similarity between the cue affinity target domain image features and the cue affinity text features is calculated to obtain the target domain cosine similarity matrix. The cross-entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated. The domain adaptation model is trained and evaluated using the test set.

2. The method according to claim 1, characterized in that, Generating multimodal category centroids based on the training set specifically includes: The CLIP model is used to predict the center point of each image in the source domain data. The center point of each image is calculated by selecting the samples with the highest prediction probability. The manually created template in the CLIP model is used as the center point of the text class. The center points of the image class and the center points of the text class are linearly fused to generate multimodal class center points.

3. The method according to claim 1, characterized in that, The learnable visual cue includes a front part and a back part; The first part is generated by learning text prompts through a linear projection layer; The latter part is generated independently.

4. The method according to claim 1, characterized in that, The pre-constructed visual-language CLIP model is trained based on the training set, test set, and multimodal category centroids, and the CLIP model is optimized using the total loss function to obtain the final domain adaptation model, which specifically includes: Attention features are calculated between the source domain image features and target domain image features of cue affinity and the multimodal category centroids to obtain enhanced source domain image features and target domain image features. The enhanced source domain image features and target domain image features are then compared with the cue affinity text features to calculate cosine similarity, resulting in enhanced source domain cosine similarity matrices and enhanced target domain cosine similarity matrices. The cross-entropy loss between the enhanced source domain cosine similarity matrix and the real label, as well as the cross-entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label, are also calculated. The CLIP model is optimized based on the total loss function shown in Equation 1 for each cross-entropy loss, resulting in the final domain adaptation model. Formula 1: in, The cross-entropy loss represents the cross-entropy loss between the source domain cosine similarity matrix and the true label. The cross-entropy loss represents the cross-entropy between the target domain cosine similarity matrix and the pseudo-label. , represents the cross-entropy loss calculated between the enhanced source domain data and the real label, and between the target domain data and the pseudo label, respectively, after attention calculation; β is the balance factor.

5. An unsupervised domain adaptation system based on multimodal category centers, characterized in that... include: The data module is used to acquire a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data and the test set is target domain data. The center point module is used to generate multimodal category center points based on the training set; The training module is used to train the pre-built vision-language CLIP model based on the training set, test set, and multimodal category centroids, and to optimize the CLIP model using the total loss function to obtain the final domain adaptation model. Specifically, it is used for: The source domain data and target domain data are input into the CLIP model for training. The input text and learnable text cues are concatenated and then processed by the CLIP text encoder to obtain cue affinity text features. The learnable visual cues are concatenated with the source domain image and the target domain image respectively, and then processed by the CLIP image encoder to obtain cue affinity source domain image features and cue affinity target domain image features. Calculate the cosine similarity between the source domain image features and the source domain text features of cue affinity to obtain the source domain cosine similarity matrix, and calculate the cross-entropy loss between the source domain cosine similarity matrix and the true label. The target domain image is processed through the CLIP model to obtain target domain image features. The cosine similarity between the target domain image features and the cue affinity text features is calculated, and the one with the largest cosine similarity is taken as the pseudo label of the target domain image. The cosine similarity between the cue affinity target domain image features and the cue affinity text features is calculated to obtain the target domain cosine similarity matrix. The cross-entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated. An evaluation module is used to train and evaluate the domain adaptation model using the test set.

6. The system according to claim 5, characterized in that, The center point module is specifically used for: The CLIP model is used to predict the center point of each image in the source domain data. The center point of each image is calculated by selecting the samples with the highest prediction probability. The manually created template in the CLIP model is used as the center point of the text class. The center points of the image class and the center points of the text class are linearly fused to generate multimodal class center points.

7. The system according to claim 5, characterized in that, The training module is also specifically used for: Attention features are calculated between the source domain image features and target domain image features of cue affinity and the multimodal category centroids to obtain enhanced source domain image features and target domain image features. The enhanced source domain image features and target domain image features are then compared with the cue affinity text features to calculate cosine similarity, resulting in enhanced source domain cosine similarity matrices and enhanced target domain cosine similarity matrices. The cross-entropy loss between the enhanced source domain cosine similarity matrix and the real label, as well as the cross-entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label, are also calculated. The CLIP model is optimized based on the total loss function shown in Equation 1 for each cross-entropy loss, resulting in the final domain adaptation model. Formula 1: in, The cross-entropy loss represents the cross-entropy loss between the source domain cosine similarity matrix and the true label. The cross-entropy loss represents the cross-entropy between the target domain cosine similarity matrix and the pseudo-label. , represents the cross-entropy loss calculated between the enhanced source domain data and the real label, and between the target domain data and the pseudo label, respectively, after attention calculation; β is the balance factor.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the unsupervised domain adaptation method based on multimodal category centers as described in any one of claims 1-4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the unsupervised domain adaptation method based on multimodal category centers as described in any one of claims 1-4.

Images