Unsupervised domain adaptation method and device, equipment and storage medium

By generating prompt words and data enhancement, the parameters of the pre-trained model are optimized, and the problems of semantic information loss and training overhead in unsupervised domain adaptation are solved, and the generalization ability and robustness of the model are improved.

CN119067191APending Publication Date: 2024-12-03PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411072471.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing unsupervised domain adaptation methods are insufficient in the problems of semantic information loss, large training overhead and poor generalization capabilities, and are difficult to effectively apply in resource-limited environments.

Method used

By selecting the pre-trained model, generate prompt words, and generate target data based on the input data and prompt words, input it into the text encoder and image encoder, generate text representation and image representation, freeze some parameters, update only the target parameters, and optimize cosine Similarity to improve model performance.

Benefits of technology

The performance of pre-trained models in new domain tasks is improved, the diversity and richness of training data is increased, the robustness and generalization ability of the model is optimized, and the training overhead is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067191A_ABST
    Figure CN119067191A_ABST
Patent Text Reader

Abstract

The invention relates to an unsupervised domain adaptation method. The method comprises the steps of selecting a pre-training model, obtaining input data, generating cue words according to the pre-training model, and generating target data of the pre-training model according to the input data and the cue words. The pre-training model comprises a text encoder and an image encoder, inputting the target data into the text encoder and the image encoder of the pre-training model, and generating text representation and image representation. And training a pre-training model according to the text representation and the image representation to obtain a target model. Through the steps, the performance of the pre-training model in a new domain task is improved, the diversity and richness of training data are increased through prompt word generation and data enhancement, meanwhile, the parameters of the model are optimized, and the robustness and generalization ability of the model are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an unsupervised domain adaptation method, device, equipment and storage medium. Background Art

[0002] In recent years, deep learning has made remarkable progress in fields such as natural language processing (NLP) and computer vision (CV). Pre-trained models such as BERT, GPT-3, ResNet, and Vision Transformer have performed well in these tasks. However, these pre-trained models are usually trained on large-scale annotated datasets and adapted to data in specific domains. Although pre-trained models can capture rich semantic and visual features, their performance may drop significantly when these models are applied to new domains or tasks. This is mainly because pre-trained models fail to fully consider the specific characteristics and data distribution of the target domain during training.

[0003] Domain adaptation is a technique to solve this problem by adapting the pre-trained model to new domain data to improve its performance in the target task. Traditional domain adaptation methods usually require a large amount of labeled data in the target domain, which may be difficult to obtain in practical applications. For this reason, unsupervised domain adaptation methods have attracted widespread attention. Unsupervised domain adaptation methods do not rely on labeled data in the target domain, but instead enable the model to adapt to the data distribution and characteristics of the target domain through techniques such as contrastive learning and self-supervised learning.

[0004] Existing unsupervised domain adaptation methods have the following main problems:

[0005] Semantic information loss: When mapping data to feature space, some semantic information may be lost, which affects the performance of the model.

[0006] High training cost: Traditional methods usually require fine-tuning the entire model, resulting in huge training costs and difficult to apply in resource-limited environments.

[0007] Poor generalization ability: When processing data from different fields, the model’s generalization ability is limited and it is difficult to adapt to a variety of different tasks and data distributions. Summary of the invention

[0008] The main purpose of the present invention is to provide an unsupervised domain adaptation method, apparatus, device and storage medium, aiming to solve the technical problems that the unsupervised domain adaptation system has poor generalization ability and loses semantic information.

[0009] To achieve the above object, the present invention provides an unsupervised domain adaptation method, which comprises the following steps:

[0010] Select a pre-trained model, obtain input data, generate prompt words according to the pre-trained model, and generate target data of the pre-trained model according to the input data and the prompt words;

[0011] Inputting the target data into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation;

[0012] The pre-trained model is trained according to the text representation and the image representation to obtain a target model.

[0013] Optionally, the step of training the pre-trained model according to the text representation and the image representation includes:

[0014] Selecting a target parameter to be trained in the pre-trained model, and freezing other parameters in the pre-trained model except the target parameter;

[0015] The pre-trained model is trained according to the text representation and the image representation, and the target parameters are updated.

[0016] Optionally, the step of training the pre-trained model according to the text representation and the image representation includes:

[0017] Calculating the cosine similarity between the text representation and the image representation;

[0018] The text encoder and the image encoder of the pre-trained model are trained using the cosine similarity to maximize the cosine similarity to optimize the pre-trained model.

[0019] Optionally, the step of calculating the cosine similarity between the text representation and the image representation includes:

[0020] Expanding the dimensions of the text representation and the image representation so that the expanded text representation and the image features can be subjected to matrix operations;

[0021] The cosine similarity between the text representation and the image representation is calculated through the matrix operation.

[0022] Optionally, the step of generating prompt words according to the pre-trained model includes:

[0023] Constructing a template containing prompt words related to the domain of the pre-trained model and those unrelated to the domain;

[0024] Determine general text that is not related to the domain of the pre-trained model, and determine specific text that is related to the domain of the pre-trained model;

[0025] The prompt word template is used to combine the general text and the specific text to generate the prompt word.

[0026] Optionally, the step of inputting the target data into the text encoder and the image encoder of the pre-trained model includes:

[0027] Extracting text information and image information from the target data;

[0028] Using the embedding layer of the pre-trained model, converting the text information into an embedding vector, and inputting the embedding vector into the text editor;

[0029] The image information is subjected to a data enhancement operation, and the enhanced image information is input into the image encoder.

[0030] Optionally, the step of selecting a pre-trained model includes:

[0031] Determine the type of the target task, and select the pre-trained model from a preset model library according to the type of the target task.

[0032] Furthermore, to achieve the above object, the present invention also provides an unsupervised domain adaptation device, the unsupervised domain adaptation device comprising:

[0033] A selection module selects a pre-trained model, obtains input data, generates prompt words according to the pre-trained model, and generates target data of the pre-trained model according to the input data and the prompt words;

[0034] A generation module, inputting the target data into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation;

[0035] A training module trains the pre-trained model according to the text representation and the image representation to obtain a target model.

[0036] Furthermore, to achieve the above-mentioned purpose, the present invention also provides an unsupervised domain adaptation device, which includes a memory, a processor, and an unsupervised domain adaptation program stored in the memory and executable on the processor, and the unsupervised domain adaptation program, when executed by the processor, implements the steps of the unsupervised domain adaptation method described above.

[0037] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which an unsupervised domain adaptation program is stored, and when the unsupervised domain adaptation program is executed by a processor, the steps of the unsupervised domain adaptation method as described above are implemented.

[0038] The present invention relates to an unsupervised domain adaptation method, which selects a pre-trained model, obtains input data, generates prompt words according to the pre-trained model, and then generates target data of the pre-trained model according to the input data and the prompt words. The pre-trained model includes a text encoder and an image encoder, and the target data is input into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation. The pre-trained model is trained according to the text representation and the image representation to obtain a target model. Through the above steps, the present invention improves the performance of the pre-trained model in new domain tasks, and increases the diversity and richness of the training data by generating prompt words and data enhancement, while also optimizing the parameters of the model and improving the robustness and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the structure of the device hardware operating environment involved in the unsupervised domain adaptation device embodiment of the present invention;

[0040] Figure 2 It is a flowchart of the first embodiment of the unsupervised domain adaptation method of the present invention;

[0041] Figure 3 Schematic diagram of functional modules of a preferred embodiment of the unsupervised domain adaptation device of the present invention.

[0042] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0044] It should be noted that in recent years, deep learning has made remarkable progress in fields such as natural language processing (NLP) and computer vision (CV). Pre-trained models such as BERT, GPT-3, ResNet, and Vision Transformer have performed well in these tasks. However, these pre-trained models are usually trained on large-scale annotated datasets and adapted to data in specific domains. Although pre-trained models can capture rich semantic and visual features, their performance may drop significantly when these models are applied to new domains or tasks. This is mainly because the pre-trained models fail to fully consider the specific characteristics and data distribution of the target domain during the training process.

[0045] Domain adaptation is a technique to solve this problem by adapting the pre-trained model to new domain data to improve its performance in the target task. Traditional domain adaptation methods usually require a large amount of labeled data in the target domain, which may be difficult to obtain in practical applications. For this reason, unsupervised domain adaptation methods have attracted widespread attention. Unsupervised domain adaptation methods do not rely on labeled data in the target domain, but instead enable the model to adapt to the data distribution and characteristics of the target domain through techniques such as contrastive learning and self-supervised learning.

[0046] Existing unsupervised domain adaptation methods have the following main problems:

[0047] Semantic information loss: When mapping data to feature space, some semantic information may be lost, which affects the performance of the model.

[0048] High training cost: Traditional methods usually require fine-tuning the entire model, resulting in huge training costs and difficult to apply in resource-limited environments.

[0049] Poor generalization ability: When processing data from different fields, the model’s generalization ability is limited and it is difficult to adapt to a variety of different tasks and data distributions.

[0050] Based on the above-mentioned defects, the present invention provides an unsupervised domain adaptation device, referring to Figure 1 , Figure 1 It is a schematic diagram of the structure of the device hardware operating environment involved in the unsupervised domain adaptation device embodiment of the present invention.

[0051] like Figure 1 As shown, the unsupervised domain adaptation device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0052] Those skilled in the art will understand that Figure 1 The hardware structure of the unsupervised domain adaptation device shown in the figure does not constitute a limitation of the unsupervised domain adaptation device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0053] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and an unsupervised domain adaptation program. The operating system is a program for managing and controlling unsupervised domain adaptation devices and software resources, and supports the operation of the network communication module, the user interface module, the unsupervised domain adaptation program, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0054] exist Figure 1 In the hardware structure of the unsupervised domain adaptation device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the unsupervised domain adaptation program stored in the memory 1005 and perform the following operations:

[0055] Select a pre-trained model, obtain input data, generate prompt words according to the pre-trained model, and generate target data of the pre-trained model according to the input data and the prompt words;

[0056] Inputting the target data into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation;

[0057] The pre-trained model is trained according to the text representation and the image representation to obtain a target model.

[0058] Furthermore, the step of training the pre-trained model according to the text representation and the image representation includes:

[0059] Selecting a target parameter to be trained in the pre-trained model, and freezing other parameters in the pre-trained model except the target parameter;

[0060] The pre-trained model is trained according to the text representation and the image representation, and the target parameters are updated.

[0061] Furthermore, the step of training the pre-trained model according to the text representation and the image representation includes:

[0062] Calculating the cosine similarity between the text representation and the image representation;

[0063] The text encoder and the image encoder of the pre-trained model are trained using the cosine similarity to maximize the cosine similarity to optimize the pre-trained model.

[0064] Furthermore, the step of calculating the cosine similarity between the text representation and the image representation includes:

[0065] Expanding the dimensions of the text representation and the image representation so that the expanded text representation and the image features can be subjected to matrix operations;

[0066] The cosine similarity between the text representation and the image representation is calculated through the matrix operation.

[0067] Furthermore, the step of generating prompt words according to the pre-trained model includes:

[0068] Constructing a template containing prompt words related to the domain of the pre-trained model and those unrelated to the domain;

[0069] Determine general text that is not related to the domain of the pre-trained model, and determine specific text that is related to the domain of the pre-trained model;

[0070] The prompt word template is used to combine the general text and the specific text to generate the prompt word.

[0071] Furthermore, the step of inputting the target data into the text encoder and the image encoder of the pre-trained model includes:

[0072] Extracting text information and image information from the target data;

[0073] Using the embedding layer of the pre-trained model, converting the text information into an embedding vector, and inputting the embedding vector into the text editor;

[0074] The image information is subjected to a data enhancement operation, and the enhanced image information is input into the image encoder.

[0075] Furthermore, the step of selecting a pre-trained model includes:

[0076] Determine the type of the target task, and select the pre-trained model from a preset model library according to the type of the target task.

[0077] The specific implementation of the unsupervised domain adaptation device of the present invention is basically the same as the following embodiments of the unsupervised domain adaptation method, and will not be repeated here.

[0078] The present invention also provides an unsupervised domain adaptation method based on the unsupervised domain adaptation device.

[0079] Reference Figure 2 , Figure 2 Schematic diagram of the process of the first embodiment of the unsupervised domain adaptation method of the present invention.

[0080] The embodiments of the present invention provide embodiments of the unsupervised domain adaptation method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0081] In various embodiments of the unsupervised domain adaptation method, the execution subject is an unsupervised domain adaptation device.

[0082] The unsupervised domain adaptation method is applied to an unsupervised domain adaptation device, and the unsupervised domain adaptation method includes:

[0083] Step S10, selecting a pre-trained model, acquiring input data, generating prompt words according to the pre-trained model, and generating target data of the pre-trained model according to the input data and the prompt words;

[0084] In recent years, deep learning has made remarkable progress in fields such as natural language processing (NLP) and computer vision (CV). Pre-trained models such as BERT, GPT-3, ResNet, and Vision Transformer have performed well in these tasks. However, these pre-trained models are usually trained on large-scale annotated datasets and adapted to data in specific domains. Although pre-trained models can capture rich semantic and visual features, their performance may drop significantly when these models are applied to new domains or tasks. This is mainly because pre-trained models fail to fully consider the specific characteristics and data distribution of the target domain during training.

[0085] Domain adaptation is a technique to solve this problem by adapting the pre-trained model to new domain data to improve its performance in the target task. Traditional domain adaptation methods usually require a large amount of labeled data in the target domain, which may be difficult to obtain in practical applications. For this reason, unsupervised domain adaptation methods have attracted widespread attention. Unsupervised domain adaptation methods do not rely on labeled data in the target domain, but instead enable the model to adapt to the data distribution and characteristics of the target domain through techniques such as contrastive learning and self-supervised learning.

[0086] Existing unsupervised domain adaptation methods have the following main problems:

[0087] Semantic information loss: When mapping data to feature space, some semantic information may be lost, which affects the performance of the model.

[0088] High training cost: Traditional methods usually require fine-tuning the entire model, resulting in huge training costs and difficult to apply in resource-limited environments.

[0089] Poor generalization ability: When processing data from different fields, the model’s generalization ability is limited and it is difficult to adapt to a variety of different tasks and data distributions.

[0090] The unsupervised domain adaptation method in this embodiment is applicable to a device having the function of the unsupervised domain adaptation method.

[0091] In this embodiment, a pre-trained model is first selected, and the selection method can be selected according to the type of target task.

[0092] For example, determine the type of target task.

[0093] Text tasks: such as text classification, sentiment analysis, question-answering systems, etc. For these tasks, you can choose pre-trained models such as BERT and GPT-3.

[0094] Image tasks: such as image classification, object detection, image generation, etc. For these tasks, you can choose pre-trained models such as ResNet and Vision Transformer (ViT).

[0095] Multimodal tasks: such as image-text matching, image description generation, etc. For these tasks, you can choose multimodal pre-trained models such as CLIP and DALL-E.

[0096] Then select a pre-trained model from the preset model library. In the preset model library, select the most suitable pre-trained model according to the requirements of the target task.

[0097] For example, for text tasks: select the BERT model for text classification and the GPT-3 model for generation tasks.

[0098] Image tasks: Select the ResNet model for image classification and the Vision Transformer for complex image analysis tasks.

[0099] Multimodal tasks: Select the CLIP model for image-text matching tasks, and select the DALL-E model for image generation tasks.

[0100] At the same time, you can also evaluate the model performance: evaluate the performance of the selected pre-trained model on the source data to ensure that it performs well on the data in the source domain. Based on the evaluation results, select the model with the best performance as the final pre-trained model.

[0101] After determining the pre-trained model, obtain input data. The acquisition method can be:

[0102] First, collect the raw data:

[0103] Text data: Collect large amounts of text data from public text datasets, web crawlers, internal corporate databases, etc.

[0104] Image data: Image data is collected from public image datasets, Internet image libraries, corporate internal image resources, etc.

[0105] Multimodal data: Collect data sets containing correspondences between text and images, such as picture-text comparison sets, image description data sets, etc.

[0106] Second, preprocess the input data:

[0107] Text data preprocessing: Perform operations such as word segmentation, stop word removal, stemming, and regularization on text data to ensure that the text data format is unified.

[0108] Image data preprocessing: Standardize image data, including resizing, cropping, noise removal, and other operations.

[0109] Multimodal data preprocessing: Ensure the accurate correspondence between text and image, and preprocess the text and image separately.

[0110] Then, generate prompt words based on the pre-trained model, which can be done by:

[0111] Construct a prompt word template, for example:

[0112] General templates: Build general prompt word templates, such as "This is a category picture."

[0113] Domain-related templates: Build prompt word templates related to specific domains, such as "This is a sketch picture" or "This is an oil painting picture."

[0114] Identify generic and specific text, such as:

[0115] Generic text: Identify generic text that is not domain-related, such as “This is a picture.”

[0116] Specific text: Identify texts related to specific fields, such as "sketch", "oil painting", "landscape", "character", etc.

[0117] Combined prompt words: Use the prompt word template to combine general text and specific text to generate the final prompt word.

[0118] For example, the template “This is a category picture” combines the general text “This is a” and the specific text “sketch” to generate the prompt word “This is a sketch picture”.

[0119] Finally, the target data of the pre-trained model is generated based on the input data and the prompt words. The method can be:

[0120] First, combine the text data with the prompt word: combine the preprocessed text data with the generated prompt word to form the final input text, i.e., the target data. For example, if the original text data is "the cat is sleeping" and the prompt word is "this is a sketch picture", then the final input text is "this is a sketch picture of a cat sleeping".

[0121] Second, combine the image data with the prompt word: combine the preprocessed image data with the generated prompt word to form the final input image, i.e., the target data. For example, if the original image data is a picture of a cat and the prompt word is "This is a sketch picture", then when inputting the image, the prompt word information can be added to the metadata of the image, or the prompt word information can be attached when the image is displayed.

[0122] Third, combine multimodal data with prompt words: combine the preprocessed text and image data with the generated prompt words to form the final multimodal input data, i.e., target data. For example, if the text data is "This is a cat", the image data is a picture of a cat, and the prompt word is "This is a sketch picture", the multimodal input data formed is "This is a sketch picture, this is a cat" and the corresponding cat picture.

[0123] Step S20, inputting the target data into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation;

[0124] In this embodiment, the target data is input into the text encoder and the image encoder of the pre-trained model.

[0125] The method can be:

[0126] Text encoder input method: Get the input text after combining the prompt word, use the embedding layer of the pre-trained model to convert the input text into an embedding vector, input the embedding vector into the text encoder of the pre-trained model to generate a text representation.

[0127] Image encoder input method: obtain the input image combined with the prompt word, use data enhancement technology (such as random cropping, rotation, color jitter, etc.) to enhance the image, and input the enhanced image into the image encoder of the pre-trained model to generate an image representation.

[0128] Multimodal encoder input method: obtain multimodal input data after combining the prompt words, process the text and image data separately, generate their respective representations, use a multimodal pre-trained model (such as CLIP) to fuse the text and image representations to generate a multimodal representation.

[0129] Step S30: training the pre-trained model according to the text representation and the image representation to obtain a target model.

[0130] In this embodiment, the pre-trained model is trained according to the text representation and the image representation to obtain a target model. The steps may include:

[0131] Calculate similarity: Match the generated text representation and image representation to calculate similarity. For example, use cosine similarity to calculate the similarity between the text representation and the image representation.

[0132] Define loss function: Define the loss function based on the calculated similarity. For example, use contrastive loss to maximize the similarity between matching text and image representations and minimize the similarity between mismatched representations.

[0133] Freeze some parameters: During the training process, freeze most of the parameters of the pre-trained model and only fine-tune a small number of target parameters to reduce training overhead.

[0134] Model optimization: Use optimization algorithms (such as Adam or SGD) to optimize the loss function, gradually update the model parameters, reduce the loss function value, and thus improve model performance.

[0135] Iterative training: Perform multiple iterative trainings to continuously optimize and update model parameters and eventually obtain the target model.

[0136] Through the above steps, the pre-trained model can be trained according to the text representation and image representation, the model parameters can be optimized, and the target model with excellent performance in the target task can be obtained.

[0137] The present invention relates to an unsupervised domain adaptation method, which selects a pre-trained model, obtains input data, generates prompt words according to the pre-trained model, and then generates target data of the pre-trained model according to the input data and the prompt words. The pre-trained model includes a text encoder and an image encoder, and the target data is input into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation. The pre-trained model is trained according to the text representation and the image representation to obtain a target model. Through the above steps, the present invention improves the performance of the pre-trained model in new domain tasks, and increases the diversity and richness of the training data by generating prompt words and data enhancement, while also optimizing the parameters of the model and improving the robustness and generalization ability of the model.

[0138] Furthermore, a second embodiment of the unsupervised domain adaptation method of the present invention is proposed. In the above step S30, the step of training the pre-trained model according to the text representation and the image representation includes:

[0139] Step a1, selecting a target parameter to be trained in the pre-trained model, and freezing other parameters in the pre-trained model except the target parameter;

[0140] Step a2: training the pre-trained model according to the text representation and the image representation, and updating the target parameters.

[0141] In this embodiment, first, the target parameters that need to be trained in the pre-trained model are determined. These parameters are usually located in the last few layers of the model or in the layers directly related to the specific task. For example, in the BERT model, the parameters of the last few layers can be selected for training; in the ResNet model, the last fully connected layer can be selected for training.

[0142] In addition, target parameters can also be selected by analyzing the structure of the pre-trained model to understand the function and parameter distribution of each layer. For example, the BERT model contains an embedding layer, a multi-head self-attention layer, and a feedforward neural network layer; the ResNet model contains multiple convolutional layers and residual blocks, etc.

[0143] Then determine the layers that are relevant to the task: Based on the requirements of the target task, determine which layers are most closely related to the task. Usually, the parameters of the last few layers of the model are directly related to the specific task and can be used as target parameters. For example: in image classification tasks, the parameters of the last fully connected layer of the ResNet model are the most important. In text classification tasks, the parameters of the last Transformer layer of the BERT model are the most important.

[0144] After selecting the target parameters, you need to freeze the other parameters in the model except the target parameters to ensure that these parameters remain unchanged during the training process. Freezing methods include:

[0145] Set parameter freezing: Set non-target parameters to a non-trainable state to ensure that they will not be updated during training. For example, for the BERT model, you can select the parameters of the last Transformer layer for training and freeze the parameters of other layers. For the ResNet model, you can select the parameters of the last fully connected layer for training and freeze the parameters of other layers.

[0146] Freeze all parameters except the target parameters in the pre-trained model to ensure that they remain unchanged during training. This can be achieved by setting the gradients of these parameters to zero, thus preventing them from being updated during back-propagation.

[0147] In addition, the target parameters can be selected in the same way as the parameters of a specific layer. For example:

[0148] Target parameter selection for text models: In the BERT model, the parameters of the last Transformer layer can be selected for training. This allows the model to adapt to the needs of new tasks while retaining the original pre-training knowledge.

[0149] Target parameter selection for image models: In the ResNet model, the parameters of the last fully connected layer can be selected for training. By training these parameters, the model can be better adapted to the image classification task.

[0150] Targeted parameter selection for multimodal models: In multimodal models such as CLIP, the parameters of the last few layers of the text encoder and image encoder can be selected for training. This allows the model to perform better when processing text and image data.

[0151] Finally, the pre-trained model is trained based on the text representation and image representation, and the target parameters are updated.

[0152] This embodiment can effectively select the target parameters that need to be trained in the pre-trained model through the above method, and freeze other parameters, thereby reducing training overhead and improving training efficiency. At the same time, by selecting target parameters related to the task, it can better adapt to the task requirements of the new field and improve the performance of the model in the target task.

[0153] Furthermore, a third embodiment of the unsupervised domain adaptation method of the present invention is proposed. In the above step S30, the step of training the pre-trained model according to the text representation and the image representation includes:

[0154] Step b1, calculating the cosine similarity between the text representation and the image representation;

[0155] Step b2: Use the cosine similarity to train the text encoder and image encoder of the pre-trained model to maximize the cosine similarity to optimize the pre-trained model.

[0156] In this embodiment, the cosine similarity between the text representation and the image representation is calculated:

[0157] Get representation: Pass the input data through the text encoder and image encoder respectively to obtain the text representation g(t) and image representation f(x).

[0158] First, calculate the cosine similarity: Use cosine similarity to calculate the similarity between each pair of text representation and image representation. The formula for calculating cosine similarity is:

[0159]

[0160] Among them, g(t) and f(x) represent text representation and image representation respectively, and ∥g(t)∥ and ∥f(x)∥ represent the norms of g(t) and f(x) respectively.

[0161] Then, define the loss function:

[0162] Classification loss function: In the classification task, the cross entropy loss function is used to optimize the model so that the similarity of correctly matched text and image pairs is maximized and the similarity of incorrectly matched pairs is minimized. The cross entropy loss function is defined as follows:

[0163]

[0164] Among them, N s is the sample size, is the true label of the i-th sample, is the label predicted by the model, is the probability that the model predicts the correct label.

[0165] The predicted probability is calculated by softmax, and the softmax function is used to calculate the probability of each category:

[0166]

[0167] Where <·,·> represents cosine similarity, T is a user-defined hyperparameter, represents the text representation of the kth category, represents the image representation of the i-th sample, and K is the total number of categories.

[0168] Optimize the model: According to the specific requirements of the model, select a suitable optimization algorithm, such as Adam, SGD, etc. The optimization algorithm gradually reduces the value of the loss function through the gradient descent method. Through the optimization algorithm, adjust the parameters of the model to minimize the loss function value, thereby improving the performance of the model.

[0169] Update target parameters: In each iteration, the gradient is calculated and the model parameters are updated. The goal is to maximize the similarity between correctly matched text and image representations and minimize the similarity between incorrectly matched representations. Through multiple iterations, the model parameters are continuously optimized, and the model's predictive ability is gradually improved.

[0170] Iterative training: Perform multiple iterative training, each of which includes the steps of forward propagation, loss calculation, back propagation, and parameter update. By continuously optimizing and updating the target parameters, the performance of the model will gradually improve.

[0171] Evaluate the model: After each iteration, use the validation set to evaluate the performance of the model to ensure that the model has not been overfitted or underfitted during training.

[0172] Final model: After training, the optimized target model is obtained. This model can perform well in the target task and has good generalization ability.

[0173] Through the above steps, this embodiment can effectively use text representation and image representation to train the pre-trained model, optimize the model's text encoder and image encoder, maximize the cosine similarity between the text representation and the image representation, and thus improve the performance of the model in new domain tasks. The present invention does not require images to be aligned at the feature level, but combines their domain information by using prompt words, and uses cosine similarity to merge them, which can effectively avoid the performance degradation caused by feature alignment, and the prompt word method of the present invention is more explanatory.

[0174] Furthermore, a fourth embodiment of the unsupervised domain adaptation method of the present invention is proposed, wherein the above step b1 includes:

[0175] Step c1, expanding the dimensions of the text representation and the image representation so that the expanded text representation and the image features can be subjected to matrix operations;

[0176] Step c2: calculating the cosine similarity between the text representation and the image representation through the matrix operation.

[0177] In this embodiment, the input data is passed through a text encoder and an image encoder respectively to obtain a text representation g(t) and an image representation f(x). The text representation and the image representation are expanded in dimension so that they can be used for matrix operations. The specific method is as follows:

[0178] Assume that the dimension of the text representation g(t) is (n, d), where n is the number of samples and d is the dimension of the representation. The dimension of the image representation f(x) is (m, d), where m is the number of samples and d is the dimension of the representation. Expand the text representation g(t) to (n, 1, d) and the image representation f(x) to (1, m, d), so that matrix operations can be performed through the broadcast mechanism.

[0179] The cosine similarity between the text representation and the image representation is calculated through matrix operations. The specific steps are as follows: First, the inner product of the text representation and the image representation is calculated. Assuming that the expanded text representation is G and the expanded image representation is F, the inner product calculation formula is:

[0180] Inner product = G·F T

[0181] Then, the norms of the text representation and the image representation are calculated. Assuming that the norm of the text representation is ∥g(t)∥ and the norm of the image representation is ∥f(x)∥, the norm calculation formula is

[0182]

[0183] Finally, calculate the cosine similarity. Assuming the inner product is the inner product (G, F), the formula for calculating the cosine similarity is:

[0184]

[0185] Through the above steps, the cosine similarity matrix between each pair of text representation and image representation can be obtained.

[0186] Through the above steps, this embodiment can effectively use text representation and image representation to train the pre-trained model, optimize the text encoder and image encoder of the model, maximize the cosine similarity between the text representation and the image representation, and thus improve the performance of the model in new domain tasks.

[0187] Furthermore, a fifth embodiment of the unsupervised domain adaptation method of the present invention is proposed. In the above step S10, the step of generating prompt words according to the pre-trained model includes:

[0188] Step d1, constructing a template containing prompt words related to the domain of the pre-trained model and those unrelated to the domain;

[0189] Step d2, determining general texts that are unrelated to the field of the pre-trained model, and determining specific texts that are related to the field of the pre-trained model;

[0190] Step d3, using the prompt word template to combine the general text and the specific text to generate the prompt word.

[0191] In this embodiment, a prompt word template is constructed: the prompt word template contains parts related to the domain of the pre-trained model and parts not related to the domain. The template structure should be flexible to meet the task requirements of different domains. For example, the example structure of the prompt word template can be "This is a picture of 'category'", where 'category' can be filled with specific content in different domains. Specific methods may include:

[0192] Identify domain-independent text, i.e., general text that is not related to a domain: for example, “This is a *** picture”. These texts are not related to a specific domain and can be shared across all categories, using symbols Indicates that m 1 is the number of corresponding texts in the prompt word.

[0193] Identify domain-related texts, i.e., specific texts related to a domain: for example, "sketch", these texts are related to a specific domain and use the symbol Indicates that m 2 ∈{1,2,……,M 2}, which have the same dimensions as the domain-independent text.

[0194] Composition of prompt word template: The definition format of prompt words can be as follows:

[0195]

[0196] in, Indicates the prompt word Prompt, [CLASS] K Indicates the category token, Represents general text that is not related to the field. Represents specific text related to the domain.

[0197] The general text and the specific text are combined using the prompt word template. The combination method may be: using the prompt word template, combining the general text and the specific text to generate a final prompt word.

[0198] For example: prompt word template: "This is a picture of 'category'";

[0199] Generic text: "This is a";

[0200] Specific text: "Sketch";

[0201] Generated prompt word: "This is a picture of a sketch."

[0202] By inputting these prompt words into a text encoder, the corresponding text representation g is generated.

[0203] Through the above steps, this embodiment can generate prompt words suitable for tasks in different fields, effectively combining field-related and field-irrelevant information; at the same time, using prompt words in natural language form makes the operation of the system more interpretable, avoiding the uninterpretable feature space in traditional feature alignment methods.

[0204] Furthermore, a sixth embodiment of the unsupervised domain adaptation method of the present invention is proposed. In the above step S20, the step of inputting the target data into the text encoder and the image encoder of the pre-trained model includes:

[0205] Step e1, extracting text information and image information from the target data;

[0206] Step e2, using the embedding layer of the pre-trained model to convert the text information into an embedding vector, and inputting the embedding vector into the text editor;

[0207] In this embodiment, text information and image information are extracted from the target data:

[0208] Text information extraction: Extract all text information from the target dataset. For example, in a multimodal dataset, the descriptive text part can be extracted separately.

[0209] Image information extraction: Extract all image information from the target dataset. For example, in a multimodal dataset, the image part can be extracted separately.

[0210] Then use the embedding layer of the pre-trained model to convert the text information into an embedding vector:

[0211] Text information embedding: The embedding layer of the pre-trained model is responsible for converting text information into embedding vectors, which can represent the semantic information of the text.

[0212] Conversion process: Input text information to the embedding layer of the pre-trained model. The embedding layer maps text information into high-dimensional vectors, which are called embedding vectors. Example: Using the embedding layer of the BERT model, the sentence "This is a picture of a sketch" can be converted into an embedding vector.

[0213] The embedding vector is then fed into the text encoder: The text encoder receives the embedding vector and further processes it to generate text representations. These representations can be used for subsequent similarity calculations and model optimization.

[0214] Input process: The embedding vector generated from the embedding layer is input into the text encoder. The text encoder processes the embedding vector to generate a text representation. Example: The text encoder of the BERT model can further process the embedding vector to obtain a deep representation of the text.

[0215] Step e3, performing data enhancement operation on the image information, and inputting the enhanced image information into the image encoder.

[0216] At the same time, the image information is subjected to data enhancement operations, which can increase the diversity of training data and improve the generalization ability of the model.

[0217] Common data enhancement techniques:

[0218] Random Cropping: Randomly crop different parts of the image to generate different training samples.

[0219] Rotation: Randomly rotate the image by a certain angle to increase the diversity of the image.

[0220] Color jittering: Randomly adjust the color, brightness, contrast, etc. of the image to improve the robustness of the model to different image conditions.

[0221] Flip: Randomly flip images horizontally or vertically to increase the diversity of training data.

[0222] Input the enhanced image information into the image encoder: The image encoder receives the enhanced image information and converts it into image representations. These representations can be used for subsequent similarity calculations and model optimization.

[0223] Input process: The image after data augmentation is input into the image encoder. The image encoder processes the image information and generates an image representation. Example: The image encoder of the ResNet model can process the augmented image to obtain a deep representation of the image.

[0224] This embodiment uses the embedding layer of the pre-trained model to convert text information into high-dimensional embedding vectors, which can capture the deep semantic information of the text. These embedding vectors can better represent the semantic content of the text, thereby improving the model's ability to understand text information; at the same time, by performing data enhancement operations on image information (such as random cropping, rotation, color jitter, etc.), the diversity of training data can be increased, allowing the model to see more variant data during training. This helps to improve the generalization ability of the model, allowing it to perform better when processing unseen data.

[0225] Furthermore, a seventh embodiment of the unsupervised domain adaptation method of the present invention is proposed. In the above step S10, the step of selecting a pre-trained model includes:

[0226] Step f1, determining the type of the target task, and selecting the pre-trained model from a preset model library according to the type of the target task.

[0227] In this embodiment, the type of the target task is first determined, for example:

[0228] Text tasks: Text tasks mainly involve natural language processing (NLP), such as text classification, sentiment analysis, machine translation, text generation, question-answering systems, etc. Pre-trained model selection: Commonly used pre-trained models include BERT, GPT-3, RoBERTa, T5, etc.

[0229] Image tasks: Image tasks mainly involve computer vision (CV), such as image classification, object detection, image generation, image segmentation, etc. Pre-trained model selection: Commonly used pre-trained models include ResNet, VGG, Inception, EfficientNet, Vision Transformer (ViT), etc.

[0230] Multimodal tasks: Multimodal tasks involve processing and fusing multiple data types, such as image-text matching, image description generation, video understanding, etc. Pre-trained model selection: Commonly used pre-trained models include CLIP, DALL-E, VisualBERT, etc.

[0231] The preset model library should contain a variety of pre-trained models suitable for different tasks. These models have been trained on large-scale data and have good initial performance. Example: The model library can contain different types of pre-trained models such as BERT, GPT-3, ResNet, ViT, CLIP, etc.

[0232] The process of selecting a pre-trained model depending on the type of target task can be:

[0233] Text task model selection:

[0234] Text classification task: Choose BERT or RoBERTa models, which perform well on text classification tasks.

[0235] Text generation task: Choose GPT-3 or T5 models, which have strong generation capabilities in generation tasks.

[0236] Image task model selection:

[0237] Image classification task: Choose ResNet or EfficientNet models, which have high accuracy and efficiency in image classification tasks.

[0238] Object detection task: Choose YOLO or Faster R-CNN models, which have good detection effects on object detection tasks.

[0239] Multimodal task model selection:

[0240] Image-text matching task: Select the CLIP model, which has high matching accuracy in image-text matching tasks.

[0241] Image description generation task: Select the DALL-E model, which performs well in generating high-quality image descriptions.

[0242] Finally, it is also possible to evaluate and validate the chosen model:

[0243] Model performance evaluation: After selecting a pre-trained model, you need to evaluate its performance on the target task. You can use the validation set to test the model and evaluate its performance on the target task.

[0244] Adaptability verification: Make sure that the selected pre-trained model can adapt to the data and requirements of the target task. If the model does not perform well on the target task, you can try to fine-tune the model parameters or choose other models for comparison.

[0245] For example, suppose the target task is text classification, which involves classifying input text. In the preset model library, select a pre-trained model suitable for text classification tasks, such as BERT or RoBERTa. Use the validation set to test the selected pre-trained model and evaluate its performance on the text classification task. Based on the evaluation results, determine whether you need to fine-tune the model or select other models for comparison.

[0246] Through the above steps, this embodiment can select a suitable pre-training model according to the type of target task, thereby improving the performance and adaptability of the model on the target task.

[0247] The present invention also provides an unsupervised domain adaptation device.

[0248] Reference Figure 3 , Figure 3 Schematic diagram of functional modules of the first embodiment of the unsupervised domain adaptation device of the present invention. The unsupervised domain adaptation device comprises:

[0249] A selection module selects a pre-trained model, obtains input data, generates prompt words according to the pre-trained model, and generates target data of the pre-trained model according to the input data and the prompt words;

[0250] A generation module, inputting the target data into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation;

[0251] A training module trains the pre-trained model according to the text representation and the image representation to obtain a target model.

[0252] In addition, an embodiment of the present invention further provides a storage medium.

[0253] An unsupervised domain adaptation program is stored on the storage medium, and when the unsupervised domain adaptation program is executed by the processor, the steps of the unsupervised domain adaptation method described above are implemented.

[0254] The specific implementation of the storage medium of the present invention is basically the same as the above-mentioned embodiments of the unsupervised domain adaptation method, and will not be repeated here.

[0255] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

[0256] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0257] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0258] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0259] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An unsupervised domain adaptation method, characterized in that: The unsupervised domain adaptation method comprises the following steps: Select a pre-trained model, obtain input data, generate prompt words according to the pre-trained model, and generate target data of the pre-trained model according to the input data and the prompt words; Inputting the target data into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation; The pre-trained model is trained according to the text representation and the image representation to obtain a target model.

2. The unsupervised domain adaptation method according to claim 1, characterized in that The step of training the pre-trained model according to the text representation and the image representation comprises: Selecting a target parameter to be trained in the pre-trained model, and freezing other parameters in the pre-trained model except the target parameter; The pre-trained model is trained according to the text representation and the image representation, and the target parameters are updated.

3. The unsupervised domain adaptation method as claimed in claim 1, characterized in that The step of training the pre-trained model according to the text representation and the image representation comprises: Calculating the cosine similarity between the text representation and the image representation; The text encoder and the image encoder of the pre-trained model are trained using the cosine similarity to maximize the cosine similarity to optimize the pre-trained model.

4. The unsupervised domain adaptation method according to claim 3, characterized in that The step of calculating the cosine similarity between the text representation and the image representation comprises: Expanding the dimensions of the text representation and the image representation so that the expanded text representation and the image features can be subjected to matrix operations; The cosine similarity between the text representation and the image representation is calculated through the matrix operation.

5. The unsupervised domain adaptation method according to claim 1, characterized in that The step of generating prompt words according to the pre-trained model comprises: Constructing a template containing prompt words related to the domain of the pre-trained model and those unrelated to the domain; Determine general text that is not related to the domain of the pre-trained model, and determine specific text that is related to the domain of the pre-trained model; The prompt word template is used to combine the general text and the specific text to generate the prompt word.

6. The unsupervised domain adaptation method according to claim 1, characterized in that The step of inputting the target data into the text encoder and the image encoder of the pre-trained model comprises: Extracting text information and image information from the target data; Using the embedding layer of the pre-trained model, converting the text information into an embedding vector, and inputting the embedding vector into the text editor; The image information is subjected to a data enhancement operation, and the enhanced image information is input into the image encoder.

7. The unsupervised domain adaptation method according to claim 1, characterized in that The step of selecting a pre-trained model comprises: Determine the type of the target task, and select the pre-trained model from a preset model library according to the type of the target task.

8. An unsupervised domain adaptation device, characterized in that The unsupervised domain adaptation device comprises: A selection module selects a pre-trained model, obtains input data, generates prompt words according to the pre-trained model, and generates target data of the pre-trained model according to the input data and the prompt words; A generation module, inputting the target data into the text encoder and the image encoder of the pre-trained model to generate text representation and image representation; A training module trains the pre-trained model according to the text representation and the image representation to obtain a target model.

9. An unsupervised domain adaptation device, characterized in that The unsupervised domain adaptation device includes a memory, a processor, and an unsupervised domain adaptation program stored in the memory and executable on the processor. When the unsupervised domain adaptation program is executed by the processor, the steps of the unsupervised domain adaptation method according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores an unsupervised domain adaptation program, which, when executed by a processor, implements the steps of the unsupervised domain adaptation method according to any one of claims 1 to 7.