Unsupervised domain adaptation method and system based on multi-modal category center

Abstract

The invention provides an unsupervised domain adaptation method and system based on a multi-modal category center, and the method comprises the steps: obtaining a multi-source unsupervised domain adaptation data set, carrying out the preprocessing of the unsupervised domain adaptation data set, and dividing the processed data into a training set and a test set; wherein the training set is source domain data, and the test set is target domain data; generating a multi-modal category center point according to the training set; training a pre-constructed vision-language CLIP model based on the training set, the test set and the multi-modal category center point, and optimizing the CLIP model by using a total loss function to obtain a final domain adaptation model; and training and evaluating the domain adaptation model through the test set. According to the unsupervised domain adaptation method based on the prompt affinity and the multi-modal category center (PMCC), the semantic similarity between the source domain data and the target domain data is effectively measured under the condition that the target domain does not need to label the data.

Description

Technical Field

[0001] The present invention relates to the field of domain adaptation technology, and in particular to an unsupervised domain adaptation method and system based on multimodal category centers. Background Art

[0002] Unsupervised Domain Adaptation (UDA) refers to a method of migrating a model from a source domain to a target domain when there is no labeled data in the target domain. In the research field of Unsupervised Domain Adaptation (UDA), existing techniques mainly focus on reducing the distribution difference between the source and target domains through various strategies, including feature alignment, generative adversarial networks (GANs), and the use of large visual-language models (VLMs). However, these methods often encounter challenges when dealing with complex and high-dimensional data, especially when the domain shift is significant. In addition, existing methods may find it difficult to fully utilize the semantic information in multimodal data, resulting in limitations in transferring knowledge between different domains.

[0003] For example, some methods rely on feature alignment techniques, such as Maximum Mean Difference (MMD) and Maximum Density Difference (MDD), to minimize the distribution difference between the source and target domains. Although these methods have achieved some success in some cases, they may have difficulty in capturing complex semantic changes, especially when there are significant semantic differences between the source and target domains.

[0004] In addition, generative adversarial networks are also used in UDA to learn domain-invariant features through adversarial training between the generator and the discriminator. However, the training of GANs may be unstable and it may be difficult to effectively transfer semantic information between different domains.

[0005] Recently, large visual-language models, such as the multimodal pre-trained model CLIP, have attracted attention due to their strong performance on both image and text modalities. These models are trained through contrastive learning and are able to learn joint representations between images and text, which is potentially valuable for maintaining semantic consistency in UDA. However, how to effectively utilize these models for UDA, especially in the absence of labeled data in the target domain, remains an open research problem. Summary of the invention

[0006] The purpose of the present invention is to provide an unsupervised domain adaptation method and system based on multimodal category centers, aiming to solve the above-mentioned problems in the prior art.

[0007] The embodiment of the present invention provides an unsupervised domain adaptation method based on multimodal category centers, comprising:

[0008] Obtain a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data, and the test set is target domain data;

[0009] Generating multimodal category center points according to the training set;

[0010] Training a pre-built visual-language CLIP model based on the training set, the test set, and the multimodal category center points, and optimizing the CLIP model using a total loss function to obtain a final domain adaptation model; and

[0011] The domain adaptation model is trained and evaluated using the test set.

[0012] An embodiment of the present invention provides an unsupervised domain adaptation system based on multimodal category centers, including:

[0013] A data module, used to obtain a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data, and the test set is target domain data;

[0014] A center point module, used to generate a multimodal category center point according to the training set;

[0015] A training module, for training a pre-built visual-language CLIP model based on the training set, the test set and the multimodal category center points, and optimizing the CLIP model using a total loss function to obtain a final domain adaptation model; and

[0016] An evaluation module is used to train and evaluate the domain adaptation model through the test set.

[0017] An embodiment of the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned unsupervised domain adaptation method based on multimodal category centers when executed by the processor.

[0018] An embodiment of the present invention further provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the above-mentioned unsupervised domain adaptation method based on multimodal category centers are implemented.

[0019] The use of the embodiments of the present invention can include the following beneficial effects: the embodiments of the present invention propose an unsupervised domain adaptation method based on prompt affinity and multimodal category center (PMCC), which makes full use of the image and text semantic information in a large visual-language model, and maintains the semantic consistency between different domains by constructing multimodal class centers. By fusing text class centers and image class centers, PMCC effectively measures the semantic similarity between source domain and target domain data without the need for target domain labeled data; in addition, the embodiments of the present invention also introduce a multimodal prompt learning mechanism to accurately predict the true category of each image in the source domain and the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 is a flow chart of an unsupervised domain adaptation method based on multimodal category centers according to an embodiment of the present invention;

[0022] Figure 2 is an overall flow chart of an embodiment of the present invention;

[0023] Figure 3 is a PMCC framework diagram of an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of an unsupervised domain adaptation system based on multimodal category centers according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0026] Method Embodiment

[0027] According to an embodiment of the present invention, an unsupervised domain adaptation method based on multimodal category centers is provided. Figure 1 is a flow chart of an unsupervised domain adaptation method based on multimodal category centers according to an embodiment of the present invention. Figure 1 As shown, the unsupervised domain adaptation method based on multimodal category centers according to an embodiment of the present invention specifically includes:

[0028] Step S101, obtaining a multi-source unsupervised domain adaptation dataset, preprocessing the unsupervised domain adaptation dataset, and dividing the processed data into a training set and a test set; wherein the training set is source domain data, and the test set is target domain data;

[0029] Step S102, generating a multimodal category center point according to the training set, specifically includes:

[0030] The CLIP model is used to predict each image in the source domain data, and several samples with the largest prediction probability are selected to calculate the center point to obtain the image class center point; the manually made template in the CLIP model is used as the text class center point; the image class center point and the text class center point are linearly fused to generate a multimodal class center point;

[0031] Step S103, training the pre-built visual-language CLIP model based on the training set, the test set and the multimodal category center point, and optimizing the CLIP model using the total loss function to obtain the final domain adaptation model, specifically including:

[0032] The source domain data and the target domain data are input into the CLIP model for training, and the text input and the learnable text prompt are concatenated and then the prompt affinity text features are obtained through the CLIP text encoder; the learnable visual prompt is concatenated with the source domain image and the target domain image respectively, and the prompt affinity source domain image features and the prompt affinity target domain image features are obtained through the CLIP image encoder;

[0033] Calculate the cosine similarity of the prompt affinity source domain image features and the prompt affinity text features to obtain the source domain cosine similarity matrix, and calculate the cross entropy loss between the source domain cosine similarity matrix and the true label;

[0034] The target domain image is passed through the CLIP model to obtain the target domain image features, and the cosine similarity between the target domain image features and the prompt affinity text features is calculated. The one with the largest cosine similarity is used as the pseudo label of the target domain image; the cosine similarity between the prompt affinity target domain image features and the prompt affinity text features is calculated to obtain the target domain cosine similarity matrix, and the cross entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated;

[0035] The learnable visual cue includes a front portion and a rear portion;

[0036] The first part is generated by a linear projection layer using learnable text prompts;

[0037] The latter part is generated independently;

[0038] The prompt affinity source domain image features and the prompt affinity target domain image features are respectively calculated with the multimodal category center point for attention features to obtain enhanced source domain image features and target domain image features, the enhanced source domain image features and the target domain image features are respectively calculated for cosine similarity with the prompt affinity text features to obtain an enhanced source domain cosine similarity matrix and an enhanced target domain cosine similarity matrix, and the cross entropy loss between the enhanced source domain cosine similarity matrix and the true label is calculated, and the cross entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label is calculated;

[0039] Based on each cross entropy loss, the CLIP model is optimized using the total loss function shown in Formula 1 to obtain the final domain adaptation model;

[0040]

[0041] in, represents the cross entropy loss between the source domain cosine similarity matrix and the true label, represents the cross entropy loss between the target domain cosine similarity matrix and the pseudo label, They represent the cross entropy losses calculated between the enhanced source domain data and the true label and the target domain data and the pseudo label after attention calculation, and β is the balancing factor;

[0042] Step S104: training and evaluating the domain adaptation model using the test set.

[0043] The following is a specific example of the unsupervised domain adaptation method based on multimodal category centers according to an embodiment of the present invention. Figure 2 As shown, the above technical solution of the embodiment of the present invention is described in detail.

[0044] In view of the problems and shortcomings in the prior art, the PMCC method proposed in the embodiment of the present invention mainly solves the following technical problems:

[0045] 1. Redundant and unreliable target class centers: Existing methods rely on pseudo-labels when generating class centers of the target domain, which results in the need to regenerate new class centers once the target domain changes, and the reliability of these class centers depends on the confidence of the pseudo-labels. PMCC solves this problem by discarding unreliable target class centers and directly using manual hints as text centers.

[0046] 2. Lack of modal class centers: Traditional methods only consider image features when generating class centers and ignore text feature information from visual language models (VLMs), which leads to the loss of text modality and fails to fully utilize the multimodal advantages of VLMs. The PMCC method proposed in the embodiment of the present invention on the CLIP architecture can solve this problem by fusing the text and image semantic information of each class to construct a multimodal class center.

[0047] 3. Excessive and unnecessary cross-attention computational overhead: Existing solutions require two similarity measurements between different types of class centers for each input image, which brings a lot of computational overhead. PMCC removes the target domain center point by designing a novel method because the target domain center point is untrustworthy, thereby reducing unnecessary computational overhead on the target domain center point, allowing the input to be closer to the multimodal center.

[0048] Through the above improvements, PMCC is able to better preserve the semantic consistency between the source and target domains in unsupervised domain adaptation tasks, and demonstrates performance that surpasses existing state-of-the-art methods on three popular UDA datasets.

[0049] The embodiment of the present invention specifically includes the following steps:

[0050] The first step is to obtain three public unsupervised domain adaptation datasets, namely OfficeHome, Office31, and VisDA17. After preprocessing, these data are divided into training sets and test sets.

[0051] 1.1 Download the OfficeHome, Office31, and VisDA17 datasets from OfficeHomeDataset-10072016.zip-Google Drive, Domain Adaptation-UC Berkeley, and VisDA2017: Visual Domain Adaptation Challenge, respectively.

[0052] 1.2 Divide the data set into a training set and a test set, that is, use the source domain data as the training set and the target domain data as the test set.

[0053] The second step is to construct the multimodal class center points for each data set

[0054] 2.1 Build a deep learning environment. Install anaconda on the server and create a virtual environment. Build Pytorch in the virtual environment and install the required packages such as scipy, numpy, dassl, matplotlib, clip, etc.

[0055] 2.2 Generate multimodal center points by constructing image-class center points and text-class center points.

[0056] 2.3 Construction of image class center points: In order to avoid incorrect classification knowledge caused by the uncertainty of pseudo-labels, the embodiment of the present invention only uses labeled source domain data to calculate the image class center points, uses CLIP to predict each image, selects the Top-K samples with the largest probability for each class to calculate the center points, and obtains a image class center points, denoted as S = {s1, s2, ..., s a}.

[0057] 2.4 Construction of text class center points: The embodiment of the present invention directly uses the manually produced “a photo of a[CLASS]” as the text class center point, thereby obtaining b text class center points, denoted as T = {t1, t2, ..., t b}.

[0058] 2.5 Linearly fuse the image center points and text center points obtained above, 0.5*S+0.5*T, and we can get k multimodal center points M={m1,m2,...,m k}, the multimodal class center points through linear fusion contain both image information and semantic information, which can better align the source domain and the target domain.

[0059] The third step is to construct an unsupervised domain adaptation method based on multimodal category centers based on cue affinity.

[0060] 3.1 Prompt affinity text feature extraction: The learnable prompt P = {p1, p2, ..., p L} and text input T = {t1, t2, ..., t b}, that is, concat(T,P)={t1,t2,...,t b ,p1,p2,...,p L}To learn text features and enhance the transferability of the model, this can improve the semantic information in the text features by learning only a small number of parameters P without changing the parameters of the CLIP text encoder, and obtain affinity text features. For c classes, the embodiment of the present invention sets the text input to "a photo of a[CLASS]", where CLASS is the class name of each class.

[0061] 3.2 Next, we introduce a series of learnable prompts to combine with manual prompts, that is, we use the concat operation to combine the learnable text prompts P = {p1, p2, ..., p L} and text input T = {t1, t2, ..., tb}, that is, concat(T,P)={t1,t2,...,t b ,p1,p2,...,p L}, and sent to the frozen CLIP text encoder to obtain the text features of c classes.

[0062] 3.3 Prompt affinity image feature extraction: For each input source domain image X s ={x1,x2,...,x N ,x N+1} and the target domain image X t ={x1,x2,...,x N+1}, where N is the number of patches cut into the image, and the first x1 is the additional cls token vector. The embodiment of the present invention uses a learnable visual cue V with a length of L = {v1,v2,...,v L} with input image X s , X t Through concat combination, we can learn the representation information in the image.

[0063] 3.4 Combine the image input and V = {v1,v2,...,v L} Through concat, we can get {x1,x2,...,x N ,x N+1 ,v1,v2,...,v L}, input into the frozen CLIP image encoder to finally obtain the affinity image feature where the visual cues V={v1,v2,...,v L} is a text prompt P = {p1, p2, ..., p L} is generated through a linear projection layer Proj(P), which helps to integrate more text semantic information into the image. In the encoder’s later layers, the visual prompts V = {v1,v2,...,v L} is generated independently, which is conducive to obtaining richer prompt information independent of text information.

[0064] 3.5 Cosine similarity on the source domain, for each input labeled source domain image data X s , extract its affinity image features (where P is the multimodal prompt and s represents the source domain) and c text-like features Calculate the cosine similarity between image features and text features Where logit_scale is the similarity scaling factor, and the similarity matrix logit is obtained, and the cross entropy loss is calculated with the true label label.

[0065] 3.6 Calculate the pseudo labels for the target domain image classification, so that unlabeled images can participate in training and make better use of data. Use the target domain image without prompts as input X t ={x1,x2,...,x N+1}, get image features by freezing CLIP's image encoder and text encoder and text features Calculate cosine similarity Select the class with the highest probability as the pseudo label label_p of the target domain image.

[0066] 3.7 Cosine similarity on the target domain: For each input unlabeled target domain image data X t , extract its affinity image features (where P is the multimodal prompt and t represents the target domain) and c class text features Calculate the cosine similarity between image features and text features Where logit_scale is the similarity scaling factor, and after obtaining the similarity matrix, the cross entropy loss is calculated with the pseudo label.

[0067] 3.8 The above image features For multi-modal center points M = {m1,m2,...,m k}Perform cross attention calculation:

[0068]

[0069] K,V=f pre (M) (3);

[0070] Among them, f pre (·) represents a linear projection layer, Q s , Q t They represent the query vectors of the source sequence and the target sequence respectively, and K and V represent the key vector and value vector obtained by projecting the multimodal class center point M.

[0071] Calculate the attention score:

[0072]

[0073] In the formula, softmax() is the softmax function, attn_score s Represents the attention score of the source domain features, attn_score t represents the attention score of the target domain feature, K Trepresents the transpose of the key vector, ∈ represents the temperature parameter, which is used to control the smoothness of the softmax function.

[0074] Compute attention features:

[0075] z s =f post (attn_score s · V) (6);

[0076] z t =f post (attn_score t · V) (7);

[0077] In the formula, z s 、z t They respectively represent the source domain features and target domain features enhanced after attention calculation.

[0078] Finally, the similarity matrix of the attention features of the source domain and the target domain is calculated:

[0079] logit s = logit_scale·z s ·W (8);

[0080] logit t = logit_scale·z t ·W (9);

[0081] In the formula, logit s , logit t They represent the similarity matrices calculated from the enhanced source domain features, target domain features and text features respectively, and logit_scale represents the scaling factor, which is used to adjust the value of the similarity matrix of the attention feature.

[0082] And calculate the attention loss:

[0083]

[0084] Where cross_entropy() represents the cross entropy function, They represent the cross entropy loss values ​​calculated between the source domain data and the true label and between the target domain data and the pseudo label after attention calculation.

[0085] 3.9 Calculate the total loss function

[0086]

[0087] Where β is a balancing factor, which is used to control the influence of the attention loss obtained by the attention operation of the center point of the multimodal class on the training.

[0088] The fourth step is to train the PMCC visual language model and evaluate the model, such as Figure 3 Shown

[0089] 4.1 Training was performed on 4 NVIDIAA40 GPUs. The image feature extractor backbone of CLIP selected ResNet50, ResNet101, and ViT / B-16 respectively, and the text encoder used transformer.

[0090] 4.2 The batch size is set to 32. The pseudo-label thresholds for the three datasets OfficeHome, Office31, and VisDA17 are set to 0.9, 0.8, and 0.6, respectively, and the epoch sizes are 10, 20, and 10, respectively.

[0091] 4.3 Set the Top-K constructed by multimodal center points, the selected number K is 5, and the balance factor β is 0.1.

[0092] 4.4 Perform training, save the model with the highest accuracy, and finally train on the test set to end.

[0093] In summary, the PMCC method proposed in the embodiment of the present invention aims to overcome the limitations of the existing technology by constructing a multimodal class center, and more effectively measure the similarity between the source domain and the target domain by fusing the semantic information of text and image. This method not only takes advantage of the multimodal advantages of VLMs, but also improves the adaptability and generalization ability of the model across different domains through cross-attention mechanisms and multimodal cue learning. Through experiments on popular UDA datasets such as Office-31, Office-Home, and VisDA-2017, PMCC demonstrated its superior performance and low model complexity, proving its effectiveness and practicality in UDA tasks. The main parts of the embodiment of the present invention include the following points:

[0094] 1. Construction of multi-modal class centroids: We propose a new method for constructing multi-modal class centroids, which fuses text class centroids (generated directly by the CLIP text encoder based on manual hints for each category) and image class centroids (generated by the CLIP image encoder based on source domain image data) to obtain multi-modal class centroids with rich semantic information.

[0095] 2. Cross-attention mechanism: The cross-attention operation is innovatively introduced, which allows each image in the source domain or target domain to interact with the above-mentioned multimodal class center, serving as a bridge to effectively measure the semantic similarity between different domains.

[0096] 3. Multi-modal prompt learning mechanism: A multi-modal prompt learning mechanism is proposed to accurately predict the true categories of source and target domain images, thereby achieving better performance in unsupervised domain adaptation tasks.

[0097] The above aims to address the problems encountered in existing technologies, such as the limitations in quantifying the similarity of source and target domain data, and the challenges in maintaining semantic consistency in unsupervised domain adaptation tasks. Through the above methods, PMCC has demonstrated performance that exceeds the existing state-of-the-art technologies on three popular unsupervised domain adaptation datasets.

[0098] System Example

[0099] According to an embodiment of the present invention, an unsupervised domain adaptation system based on multimodal category centers is provided. Figure 4 is a schematic diagram of an unsupervised domain adaptation system based on multimodal category centers according to an embodiment of the present invention. Figure 4 As shown, the unsupervised domain adaptation system based on multimodal category centers according to an embodiment of the present invention specifically includes:

[0100] A data module 40 is used to obtain a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data, and the test set is target domain data;

[0101] The center point module 42 is used to generate a multimodal category center point according to the training set, specifically for:

[0102] The CLIP model is used to predict each image in the source domain data, and several samples with the largest prediction probability are selected to calculate the center point to obtain the image class center point; the manually made template in the CLIP model is used as the text class center point; the image class center point and the text class center point are linearly fused to generate a multimodal class center point;

[0103] The training module 44 is used to train the pre-built visual-language CLIP model based on the training set, the test set and the multimodal category center point, and optimize the CLIP model using the total loss function to obtain the final domain adaptation model, specifically for:

[0104] The source domain data and the target domain data are input into the CLIP model for training, and the text input and the learnable text prompt are concatenated and then the prompt affinity text features are obtained through the CLIP text encoder; the learnable visual prompt is concatenated with the source domain image and the target domain image respectively, and the prompt affinity source domain image features and the prompt affinity target domain image features are obtained through the CLIP image encoder;

[0105] Calculate the cosine similarity of the prompt affinity source domain image features and the prompt affinity text features to obtain the source domain cosine similarity matrix, and calculate the cross entropy loss between the source domain cosine similarity matrix and the true label;

[0106] The target domain image is passed through the CLIP model to obtain the target domain image features, and the cosine similarity between the target domain image features and the prompt affinity text features is calculated. The one with the largest cosine similarity is used as the pseudo label of the target domain image; the cosine similarity between the prompt affinity target domain image features and the prompt affinity text features is calculated to obtain the target domain cosine similarity matrix, and the cross entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated;

[0107] The prompt affinity source domain image features and the prompt affinity target domain image features are respectively calculated with the multimodal category center point for attention features to obtain enhanced source domain image features and target domain image features, the enhanced source domain image features and the target domain image features are respectively calculated for cosine similarity with the prompt affinity text features to obtain an enhanced source domain cosine similarity matrix and an enhanced target domain cosine similarity matrix, and the cross entropy loss between the enhanced source domain cosine similarity matrix and the true label is calculated, and the cross entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label is calculated;

[0108] Based on each cross entropy loss, the CLIP model is optimized using the total loss function shown in Formula 1 to obtain the final domain adaptation model;

[0109]

[0110] in, represents the cross entropy loss between the source domain cosine similarity matrix and the true label, represents the cross entropy loss between the target domain cosine similarity matrix and the pseudo label, They represent the cross entropy losses calculated between the enhanced source domain data and the true label and the target domain data and the pseudo label after attention calculation, and β is the balancing factor;

[0111] The evaluation module 46 is used to train and evaluate the domain adaptation model using the test set.

[0112] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0113] The embodiment of the present invention discloses an unsupervised domain adaptation method based on the center point of the multimodal category of prompt affinity. First, by obtaining three public domain adaptation datasets, a large visual language model is trained on the source domains of the three datasets. Finally, the test results on the test set are improved compared with the most advanced unsupervised domain adaptation methods, and the complexity of the model is reduced, and the generalization ability and robustness of the model are enhanced. The embodiment of the present invention specifically includes the following beneficial effects:

[0114] 1. Reduced computational cost: By avoiding unnecessary cross-attention computational overhead, PMCC can reduce the demand for computing resources, thereby saving hardware costs;

[0115] 2. Reduced need for manual labeling: Since PMCC is an unsupervised domain adaptation method, it does not require labeled data in the target domain, thus reducing the cost and time consumption of manual labeling;

[0116] 3. Promotes the effective use of resources: PMCC can effectively transfer knowledge between different fields, which helps to promote the use of machine learning models in application scenarios where it is difficult to obtain labeled data;

[0117] 4. Promoted technological progress: As an innovative method in the field of unsupervised domain adaptation, PMCC provides new ideas for related research and promotes technological progress and development in this field.

[0118] In summary, PMCC not only improves existing methods from a technical level, but also has a positive impact on the economic and social levels, and has high application value and development potential.

[0119] Device Example 1

[0120] An embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps described in the method embodiment when executed by the processor.

[0121] Device Example 2

[0122] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps described in the method embodiment are implemented.

[0123] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised domain adaptation method based on multimodal category centers, characterized in that include: Obtain a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data, and the test set is target domain data; Generating multimodal category center points according to the training set; Training a pre-built visual-language CLIP model based on the training set, the test set, and the multimodal category center points, and optimizing the CLIP model using a total loss function to obtain a final domain adaptation model; and The domain adaptation model is trained and evaluated using the test set.

2. The method according to claim 1, characterized in that Generating a multimodal category center point according to the training set specifically includes: The CLIP model is used to predict each image in the source domain data, and several samples with the largest prediction probability are selected for center point calculation to obtain the image class center point; the hand-crafted template in the CLIP model is used as the text class center point; the image class center point and the text class center point are linearly fused to generate a multimodal category center point.

3. The method according to claim 1, characterized in that The pre-built visual-language CLIP model is trained based on the training set, the test set and the multimodal category center points, and the CLIP model is optimized using the total loss function to obtain the final domain adaptation model, which specifically includes: The source domain data and the target domain data are input into the CLIP model for training, and the text input and the learnable text prompt are concatenated and then the prompt affinity text features are obtained through the CLIP text encoder; the learnable visual prompt is concatenated with the source domain image and the target domain image respectively, and the prompt affinity source domain image features and the prompt affinity target domain image features are obtained through the CLIP image encoder; Calculate the cosine similarity of the prompt affinity source domain image features and the prompt affinity text features to obtain the source domain cosine similarity matrix, and calculate the cross entropy loss between the source domain cosine similarity matrix and the true label; The target domain image is passed through the CLIP model to obtain the target domain image features, and the cosine similarity between the target domain image features and the prompt affinity text features is calculated. The one with the largest cosine similarity is taken as the pseudo label of the target domain image; the cosine similarity between the prompt affinity target domain image features and the prompt affinity text features is calculated to obtain the target domain cosine similarity matrix, and the cross entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated.

4. The method according to claim 3, characterized in that The learnable visual cue includes a front portion and a rear portion; The first part is generated by a linear projection layer using learnable text prompts; The latter part is generated independently.

5. The method according to claim 3, characterized in that: The pre-built visual-language CLIP model is trained based on the training set, the test set and the multimodal category center points, and the CLIP model is optimized using the total loss function to obtain the final domain adaptation model, which specifically includes: The prompt affinity source domain image features and the prompt affinity target domain image features are respectively calculated with the multimodal category center point for attention features to obtain enhanced source domain image features and target domain image features, the enhanced source domain image features and the target domain image features are respectively calculated for cosine similarity with the prompt affinity text features to obtain an enhanced source domain cosine similarity matrix and an enhanced target domain cosine similarity matrix, and the cross entropy loss between the enhanced source domain cosine similarity matrix and the true label is calculated, and the cross entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label is calculated; Based on each cross entropy loss, the CLIP model is optimized using the total loss function shown in Formula 1 to obtain the final domain adaptation model; in, represents the cross entropy loss between the source domain cosine similarity matrix and the true label, represents the cross entropy loss between the target domain cosine similarity matrix and the pseudo label, They represent the cross entropy losses calculated between the enhanced source domain data and the true label and the target domain data and the pseudo label after attention calculation, and β is the balancing factor.

6. An unsupervised domain adaptation system based on multimodal category centers, characterized in that include: A data module, used to obtain a multi-source unsupervised domain adaptation dataset, preprocess the unsupervised domain adaptation dataset, and divide the processed data into a training set and a test set; wherein the training set is source domain data, and the test set is target domain data; A center point module, used to generate a multimodal category center point according to the training set; A training module, for training a pre-built visual-language CLIP model based on the training set, the test set and the multimodal category center points, and optimizing the CLIP model using a total loss function to obtain a final domain adaptation model; and An evaluation module is used to train and evaluate the domain adaptation model through the test set.

7. The system according to claim 6, characterized in that The central point module is specifically used for: The CLIP model is used to predict each image in the source domain data, and several samples with the largest prediction probability are selected for center point calculation to obtain the image class center point; the hand-crafted template in the CLIP model is used as the text class center point; the image class center point and the text class center point are linearly fused to generate a multimodal category center point.

8. The system according to claim 6, characterized in that The training module is specifically used for: The source domain data and the target domain data are input into the CLIP model for training, and the text input and the learnable text prompt are concatenated and then the prompt affinity text features are obtained through the CLIP text encoder; the learnable visual prompt is concatenated with the source domain image and the target domain image respectively, and the prompt affinity source domain image features and the prompt affinity target domain image features are obtained through the CLIP image encoder; Calculate the cosine similarity of the prompt affinity source domain image features and the prompt affinity text features to obtain the source domain cosine similarity matrix, and calculate the cross entropy loss between the source domain cosine similarity matrix and the true label; The target domain image is passed through the CLIP model to obtain the target domain image features, and the cosine similarity between the target domain image features and the prompt affinity text features is calculated. The one with the largest cosine similarity is used as the pseudo label of the target domain image; the cosine similarity between the prompt affinity target domain image features and the prompt affinity text features is calculated to obtain the target domain cosine similarity matrix, and the cross entropy loss between the target domain cosine similarity matrix and the pseudo label is calculated; The prompt affinity source domain image features and the prompt affinity target domain image features are respectively calculated with the multimodal category center point for attention features to obtain enhanced source domain image features and target domain image features, the enhanced source domain image features and the target domain image features are respectively calculated for cosine similarity with the prompt affinity text features to obtain an enhanced source domain cosine similarity matrix and an enhanced target domain cosine similarity matrix, and the cross entropy loss between the enhanced source domain cosine similarity matrix and the true label is calculated, and the cross entropy loss between the enhanced target domain cosine similarity matrix and the pseudo label is calculated; Based on each cross entropy loss, the CLIP model is optimized using the total loss function shown in Formula 1 to obtain the final domain adaptation model; in, represents the cross entropy loss between the source domain cosine similarity matrix and the true label, represents the cross entropy loss between the target domain cosine similarity matrix and the pseudo label, They represent the cross entropy losses calculated between the enhanced source domain data and the true label and the target domain data and the pseudo label after attention calculation, and β is the balancing factor.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the unsupervised domain adaptation method based on multimodal category centers are implemented as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the unsupervised domain adaptation method based on multimodal category centers as described in any one of claims 1 to 5 are implemented.

Images