Multi-modal data field self-adaption method and device and computer equipment

Abstract

The invention relates to a multi-modal data field self-adaption method and device and computer equipment. The method comprises the following steps: acquiring a multi-modal sample set of a source domain and a multi-modal sample set of a target domain, and respectively mapping the multi-modal sample set of the source domain and the multi-modal sample set of the target domain to a public feature space in each round of adaptive process, different self-supervised learning strategies are adopted to carry out self-supervised learning on the first multi-modal sample set and the second multi-modal sample set of the target domain, and the optimal target self-supervised learning strategy in the current round is screened out from the different self-supervised learning strategies; and according to a target self-supervised learning strategy, pseudo labels are added to samples in the second multi-modal sample set without labels to update the multi-modal sample set of the target domain, and field self-adaptive processing is performed on the to-be-trained model based on the updated multi-modal sample set of the target domain and the multi-modal sample set of the source domain. By adopting the method, the precision of the field adaptive process can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a multi-modal data domain adaptation method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] At present, deep learning algorithms are increasingly applied in remote sensing, medical treatment and other multi-modal data processing fields. However, the premise of the effectiveness of deep learning algorithms is to have a large amount of labeled supervised data as training data, which is one of the disadvantages of deep learning. It is time-consuming, labor-intensive and extremely costly to obtain a large amount of labeled training data, which poses a great challenge to the effectiveness of deep learning. To address this challenge, a domain adaptation algorithm can be used to transfer knowledge from a domain with a large amount of supervised information to assist the training of a target domain with similar but scarce supervised information, so as to alleviate the challenge of data scarcity to deep learning. In this process, self-supervised learning is an important auxiliary technology. Self-supervised learning can assign pseudo-labels to target domain samples with scarce supervised information, so that target domain samples can more fully participate in the process of domain adaptation.

[0003] However, in actual application, different self-supervised learning strategies may produce different auxiliary effects at different stages of training. If an inaccurate self-supervised learning strategy is used, it not only cannot assist domain adaptation, but also misleads the entire training process. SUMMARY

[0004] Therefore, it is necessary to provide a multi-modal data domain adaptation method, device, computer equipment, computer readable storage medium and computer program product with higher accuracy to solve the above technical problems.

[0005] In a first aspect, the present application provides a multi-modal data domain adaptation method, comprising:

[0006] obtaining a multi-modal sample set of a source domain and a multi-modal sample set of a target domain, wherein the multi-modal sample set of the target domain includes a first multi-modal sample set with labels and a second multi-modal sample set without labels;

[0007] In each adaptation process, the multi-modal sample set of the source domain and the multi-modal sample set of the target domain are respectively mapped to a common feature space;

[0008] different self-supervised learning strategies are used for self-supervised learning of the first multi-modal sample set and the second multi-modal sample set of the target domain, and the optimal target self-supervised learning strategy in the current round is selected from the different self-supervised learning strategies;

[0009] adding pseudo labels to the samples in the unlabeled second multi-modal sample set according to the target self-supervised learning strategy to update the multi-modal sample set of the target domain;

[0010] performing domain adaptation processing on the to-be-trained model based on the updated multi-modal sample set of the target domain and the multi-modal sample set of the source domain.

[0011] In some example embodiments, the self-supervised learning of the first multi-modal sample set and the second multi-modal sample set of the target domain by using different self-supervised learning strategies includes:

[0012] adding pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain by using different self-supervised learning strategies, the different self-supervised learning strategies being self-supervised learning strategies obtained by combining at least three self-supervised learning strategies;

[0013] determining label accuracy rates of the different self-supervised learning strategies for adding pseudo labels to the first multi-modal sample set based on the labeled first multi-modal sample set;

[0014] determining the self-supervised learning strategy with the highest label accuracy rate as the target self-supervised learning strategy.

[0015] In some example embodiments, the adding of pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain by using different self-supervised learning strategies includes:

[0016] in the case where the self-supervised learning strategy is a single strategy, adding pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain by using a classification prediction algorithm corresponding to the self-supervised learning strategy;

[0017] in the case where the self-supervised learning strategy is a mixed strategy, adding pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain by using classification prediction algorithms corresponding to single strategies in the mixed strategy, and if the pseudo labels added by each single strategy are consistent for each multi-modal sample, determining the pseudo labels as the final pseudo labels of the multi-modal sample.

[0018] In some example embodiments, after the mapping of the multi-modal sample set of the source domain and the multi-modal sample set of the target domain to the common feature space, the method further includes:

[0019] training at least three data prediction models based on the multi-modal sample set of the source domain, the data prediction models including at least a predictor neural network, an extreme gradient boosting model, and a search model.

[0020] Each data prediction model is assigned to a different single self-supervised learning strategy, and the data prediction model corresponds to the self-supervised learning strategy one-to-one;

[0021] The plurality of different self-supervised learning strategies are combined to obtain different self-supervised learning strategies.

[0022] In some exemplary embodiments, the self-supervised learning of the first multi-modal sample set and the second multi-modal sample set of the target domain by using different self-supervised learning strategies includes:

[0023] Each of the different self-supervised learning strategies corresponds to a classification prediction algorithm, and the first multi-modal sample set and the second multi-modal sample set of the target domain are predicted respectively to obtain prediction results;

[0024] Based on the prediction results, the uncertainty and the label accuracy of the different self-supervised learning strategies are determined;

[0025] Based on the uncertainty and the accuracy, the confidence of the pseudo-label added to the first multi-modal sample set by the different self-supervised learning strategies is determined;

[0026] Based on the confidence of the pseudo-label, the optimal target self-supervised learning strategy in the current round is selected from the different self-supervised learning strategies.

[0027] In some exemplary embodiments, the mapping of the multi-modal sample set of the source domain and the multi-modal sample set of the target domain to the common feature space includes:

[0028] The source domain features are extracted from the multi-modal sample set of the source domain by a source domain feature extractor;

[0029] The target domain features are extracted from the multi-modal sample set of the target domain by a target domain feature extractor;

[0030] The source domain features and the target domain features are mapped to the common feature space.

[0031] In a second aspect, the present application further provides a multi-modal data field adaptation device, comprising:

[0032] A data acquisition module is configured to acquire a multi-modal sample set of a source domain and a multi-modal sample set of a target domain, wherein the multi-modal sample set of the target domain includes a first multi-modal sample set with labels and a second multi-modal sample set without labels;

[0033] a data mapping module, configured to map the multi-modal sample set of the source domain and the multi-modal sample set of the target domain to a common feature space respectively in each round of the adaptation process;

[0034] a strategy screening module, configured to perform self-supervised learning on the first multi-modal sample set and the second multi-modal sample set of the target domain by using different self-supervised learning strategies, and screen an optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies;

[0035] a self-learning module, configured to add pseudo labels to samples in the unlabeled second multi-modal sample set according to the target self-supervised learning strategy, so as to update the multi-modal sample set of the target domain;

[0036] a domain adaptation module, configured to perform domain adaptation processing based on the updated multi-modal sample set of the target domain and the multi-modal sample set of the source domain.

[0037] In a third aspect, the present application also provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps in any one of the multi-modal data domain adaptation method embodiments above when executing the computer program.

[0038] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps in any one of the multi-modal data domain adaptation embodiments above when executed by a processor.

[0039] In a fifth aspect, the present application also provides a computer program product, including a computer program, and the computer program implements the steps in any one of the multi-modal data domain adaptation method embodiments above when executed by a processor.

[0040] The multi-modal data domain adaptation method, device, computer equipment, computer readable storage medium and computer program product have the advantages that, different from the method of assigning pseudo labels to the samples of the target domain by using a single or fixed index as a target in a traditional domain adaptation algorithm, the multi-modal samples of the source domain and the target domain are uniformly mapped to a common feature space in each adaptation process, which helps to eliminate the representation difference between the modalities, and then different self-supervised learning strategies are used for self-supervised learning on the first multi-modal sample set and the second multi-modal sample set of the target domain, the performance of the multi-modal sample set of the target domain, especially the labeled target domain sample, is evaluated, and the target self-supervised learning strategy with the best performance in the current round is selected, and then the unlabeled second multi-modal sample set is self-supervised learned according to the target self-supervised learning strategy to determine the pseudo labels of the unlabeled second multi-modal sample set. In this way, the most suitable strategy can be dynamically selected at different training stages, the accuracy of self-supervised learning is improved, the model can better adapt to the changes of the target domain, the problem that a single strategy fails in the training process due to changes in data distribution is reduced, the overall precision and quality of the pseudo labels are improved, and in this way, the unlabeled target domain sample can more fully and accurately participate in the domain adaptation process, and the precision of the domain adaptation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0042] Figure 1 An application environment diagram of the multi-modal data domain adaptation method in an embodiment;

[0043] Figure 2 A flowchart of the multi-modal data domain adaptation method in an embodiment;

[0044] Figure 3 A flowchart of the step of selecting the target self-supervised learning strategy in an embodiment;

[0045] Figure 4 A detailed flowchart of the step of selecting the target self-supervised learning strategy in an embodiment;

[0046] Figure 5 A flowchart of the step of selecting the target self-supervised learning strategy in another embodiment;

[0047] Figure 6A structural block diagram of a multi-modal data domain adaptation device in an embodiment;

[0048] Figure 7 An internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0049] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0050] The multi-modal data domain adaptation method provided by the embodiments of the present application can be applied in an application environment as shown in the figure. Figure 1 As shown in the figure, the terminal 102 communicates with the server 104 through a network. The data storage system can store the data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0051] Specifically, the operator can upload the collected multi-modal sample set of the source domain and the multi-modal sample set of the target domain to the server 104 through the terminal 102, and then send an adaptation processing message to the server 104 through the terminal 102. The server 104 responds to the message, obtains the multi-modal sample set of the source domain and the multi-modal sample set of the target domain, and then starts the domain adaptation process. In each round of the adaptation process, the multi-modal sample set of the source domain and the multi-modal sample set of the target domain are respectively mapped to a common feature space, the first multi-modal sample set and the second multi-modal sample set of the target domain are subjected to self-supervised learning by using different self-supervised learning strategies, the optimal target self-supervised learning strategy in the current round is selected from the different self-supervised learning strategies, the samples in the unlabeled second multi-modal sample set are added with pseudo labels according to the target self-supervised learning strategy, the multi-modal sample set of the target domain is updated, and the domain adaptation processing is performed based on the updated multi-modal sample set of the target domain and the multi-modal sample set of the source domain.

[0052] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0053] In an exemplary embodiment, as shown in Figure 2 , a multi-modal data domain adaptation method is provided. The method is applied to the server 104 in Figure 1 , and includes the following steps (hereinafter referred to as S) S202 to S210. Wherein:

[0054] S202, obtaining a multi-modal sample set of a source domain and a multi-modal sample set of a target domain, wherein the multi-modal sample set of the target domain includes a first multi-modal sample set with labels and a second multi-modal sample set without labels.

[0055] A multi-modal sample refers to a sample containing information of multiple modalities, such as image + text, video + audio, etc. The multi-modal sample of the source domain refers to a multi-modal sample with rich annotation information, which is related to the target task. The multi-modal sample of the target domain refers to a specific domain data set on which the model is expected to perform well. Such data may have less or even no annotation, but its data distribution is the focus. Generally, the multi-modal sample data of the source domain is relatively sufficient in data volume, complete and accurate in label information, and the data distribution may be different from that of the target domain. The multi-modal sample data of the target domain may have only a small amount of labeled data or no label at all, and the data distribution may be different from that of the source domain, but it is the domain where the model is finally applied. In this embodiment, the knowledge learned from the multi-modal sample of the source domain is used to adjust the model to adapt to the data distribution of the target domain, thereby improving the performance of the model on the target domain.

[0056] In specific implementation, the server can load the multi-modal samples of the source domain and the target domain from a database or a file system in response to the adaptive processing message sent by the terminal, then parse each sample to extract data of each modality (such as images, texts, audios, etc.), divide the target domain samples into a labeled part (a first sample set) and an unlabeled part (a second sample set), and then construct a multi-modal sample set of the source domain and a multi-modal sample set of the target domain, wherein the multi-modal sample set of the target domain includes the first multi-modal sample set with labels and the second multi-modal sample set without labels.

[0057] In S204, in each round of the adaptive process, the multi-modal sample set of the source domain and the multi-modal sample set of the target domain are respectively mapped to a common feature space.

[0058] The common feature space is a unified representation space, so that data from different domains and different modalities can be compared and aligned in this space.

[0059] In specific implementation, a pre-trained multi-modal encoder can be used to extract source domain features and target domain features from the multi-modal samples of the source domain and the target domain respectively, and the extracted source domain features and target domain features are mapped to the common feature space. For example, for each sample of the source domain and the target domain, a feature vector of each modality can be extracted, the feature vectors of the modalities can be fused (such as splicing, weighted average, cross-attention, etc.), a unified feature vector can be obtained, and the mapped feature vector can be output for subsequent domain alignment and pseudo-label generation.

[0060] In S206, different self-supervised learning strategies are used for self-supervised learning of the first multi-modal sample set and the second multi-modal sample set of the target domain, and the optimal target self-supervised learning strategy in the current round is selected from the different self-supervised learning strategies.

[0061] The self-supervised learning strategy is a method for generating pseudo-labels without manual annotation, such as prediction consistency, contrast learning, and mask reconstruction strategies. The pseudo-label is a label automatically generated by a model or a self-supervised mechanism, which is used to guide subsequent model training.

[0062] In specific implementation, a plurality of self-supervised strategies such as A, B, and C can be defined, each strategy is independently run on the first multi-modal sample set and the second multi-modal sample set of the target domain to generate pseudo-labels. Since the samples in the first multi-modal sample set carry true labels, the true labels and the pseudo-labels of each first multi-modal sample can be compared, so that the stability and accuracy of the self-supervised learning strategy can be evaluated, and the optimal target self-supervised learning strategy in the current round can be selected from the different self-supervised learning strategies.

[0063] S208, adding pseudo-labels to samples in the unlabeled second multi-modal sample set in the target domain according to the target self-supervised learning strategy to update the multi-modal sample set in the target domain.

[0064] After the optimal target self-supervised learning strategy in the current round is selected, pseudo-labels can be added to samples in the unlabeled second multi-modal sample set in the target domain according to the target self-supervised learning strategy, and the multi-modal samples with added pseudo-labels are combined with the original labeled multi-modal samples in the target domain to form a new multi-modal sample set in the target domain.

[0065] S210, performing domain adaptation on the to-be-trained model based on the updated multi-modal sample set in the target domain and the multi-modal sample set in the source domain.

[0066] Domain adaptation is a process of making the source domain and the target domain as close as possible in the feature space to reduce the performance decline caused by domain difference. The to-be-trained model can include but is not limited to a deep learning model in a certain specific domain, such as a deep learning model in the field of remote sensing data processing.

[0067] After adding pseudo-labels to samples in the unlabeled second multi-modal sample set in the target domain according to the target self-supervised learning strategy, some unlabeled samples in the target domain will obtain self-supervised labels with good accuracy. Based on this, in the process of domain adaptation, the amount of labeled data in the target domain increases, thereby increasing the participation of the target domain data in the process of domain adaptation. After obtaining the updated multi-modal sample set in the target domain, the data in the multi-modal sample sets in the source domain and the target domain can be mapped to the regenerated Hilbert space with intra-class cohesion and inter-class dispersion through a pre-set domain adaptation algorithm, the model parameters are optimized to minimize the domain difference, thereby enhancing the sufficiency and accuracy of domain adaptation. After completing the adaptation process in this round, return to S204 until the pre-set end condition is reached, and a model with good performance in the target domain is obtained.

[0068] In the multi-modal data field adaptation method, unlike the traditional field adaptation algorithm which uses a single or fixed index to assign pseudo labels to the target domain samples, the multi-modal samples of the source domain and the target domain are mapped to a common feature space in each adaptation process, which helps to eliminate the representation difference between modalities. Then, different self-supervised learning strategies are used for self-supervised learning of the first multi-modal sample set and the second multi-modal sample set of the target domain. The performance of various self-supervised learning strategies on the target domain samples, especially the labeled target domain samples, is evaluated, and the target self-supervised learning strategy with the best performance in the current round is selected. Then, the unlabeled second multi-modal sample set is self-supervised learned according to the target self-supervised learning strategy to determine the pseudo labels of the unlabeled second multi-modal sample set. In this way, the most suitable strategy can be selected at different training stages, the model can better adapt to the changes of the target domain, the problem of single strategy failure due to data distribution changes in the training process is reduced, and the overall precision and quality of the pseudo labels are improved. In this way, the subsequent data processing precision can be effectively improved.

[0069] In one exemplary embodiment, as shown in Figure 3 S206 includes:

[0070] S216, different self-supervised learning strategies are used to add pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain. The different self-supervised learning strategies are self-supervised learning strategies obtained by combining at least three self-supervised learning strategies.

[0071] S226, based on the labeled first multi-modal sample set, the label accuracy of the different self-supervised learning strategies for adding pseudo labels to the first multi-modal sample set is determined.

[0072] S236, the self-supervised learning strategy with the highest label accuracy is determined as the target self-supervised learning strategy.

[0073] The label accuracy is a measure of the consistency between the pseudo labels generated by the self-supervised learning strategy and the true labels. In this embodiment, the self-supervised learning strategy includes both single self-supervised learning strategy and combined strategy with multiple single strategies. For example, at least three basic self-supervised learning strategies can be combined to obtain multiple different self-supervised learning strategies, such as combining self-supervised learning strategies A, B, and C to obtain multiple different self-supervised learning strategies A, B, C, AB, AC, BC, and ABC. Then, each self-supervised learning strategy or its combination is applied to the first multi-modal sample set and the second multi-modal sample set of the target domain to generate preliminary pseudo labels. In this process, consistency checking needs to be performed. For example, in the combined strategy AB, only when the predictions of both the image and the text point to the same class, a pseudo label is assigned to the sample.

[0074] Next, using the known true labels as a benchmark, the pseudo-labels generated by each strategy are compared. The proportion of pseudo-labels that agree with the true labels under each strategy is calculated to obtain the label accuracy of each strategy. For example, if strategy A correctly assigns pseudo-labels to 85 out of 100 samples, the label accuracy of strategy A is 85%. Then, based on the label accuracy of each strategy calculated in the previous step, the strategy with the highest label accuracy is selected as the final target self-supervised learning strategy for adding pseudo-labels to unlabeled samples in the target domain. The target self-supervised learning strategy is then used to add pseudo-labels to unlabeled samples in the target domain.

[0075] In this embodiment, by combining multiple self-supervised learning strategies, the essential characteristics of the data can be captured from different perspectives. By directly evaluating the effectiveness of each strategy, it is helpful to screen out the strategy that best suits the current data distribution, thereby improving the quality of pseudo labels.

[0076] like Figure 4 As shown, in an exemplary embodiment, S216 includes:

[0077] S2162: When the self-supervised learning strategy is a single strategy, pseudo labels are added to the first multimodal sample set and the second multimodal sample set of the target domain by using a classification prediction algorithm corresponding to the self-supervised learning strategy.

[0078] S2164, when the self-supervised learning strategy is a hybrid strategy, pseudo labels are added to the first multimodal sample set and the second multimodal sample set of the target domain through the classification prediction algorithm corresponding to a single strategy in the hybrid strategy. If the pseudo labels added by each single strategy are consistent for each multimodal sample, the pseudo label is determined as the final pseudo label of the multimodal sample.

[0079] A single strategy refers to a strategy that uses only one self-supervised learning method to add pseudo-labels to samples, such as using only a semantic search-based prediction model. A hybrid strategy refers to a strategy that combines multiple single strategies and their combinations to jointly participate in pseudo-label generation, such as image prediction + text matching + contrastive learning. A classification prediction algorithm refers to an algorithm used to map input samples to a category space, such as a neural network or classifier. In this embodiment, each self-supervised learning strategy is configured with an independent classification prediction algorithm.

[0080] In a specific implementation, in the process of adding pseudo labels to samples respectively by using the self-supervised learning strategy, if the self-supervised learning strategy is a single strategy, pseudo labels of samples in the first multi-modal sample set and the second multi-modal sample set of the target domain are directly added respectively by using the classification prediction algorithm corresponding to the self-supervised learning strategy, so as to obtain the pseudo labels of the samples. If the self-supervised learning strategy is a mixed strategy such as AB, pseudo labels are added to the first multi-modal sample set and the second multi-modal sample set of the target domain by using the classification prediction algorithm corresponding to strategy A and strategy B in the mixed strategy respectively. Subsequently, it is checked whether the pseudo labels generated by strategy A and strategy B for the same multi-modal sample are consistent. For a multi-modal sample, if the pseudo labels predicted by all strategies are consistent, the pseudo labels are taken as the pseudo labels of the multi-modal sample. If the pseudo labels are inconsistent, the multi-modal sample does not add pseudo labels. Exemplarily, strategy A considers that sample 1 is a cat, and strategy B also considers that sample 1 is a cat, so the self-supervised label of sample 1 is “cat”. If strategy A considers that sample 1 is a cat, and strategy B considers that sample 1 is a dog, then sample 1 does not add a self-supervised label.

[0081] In the embodiment, the advantages of multi-modal information and different learning strategies are comprehensively utilized to improve the overall domain adaptation performance. Only when multiple self-supervised learning strategies reach a consensus and all point to the same class label, the sample is given a pseudo label, thereby avoiding the adverse effects of misjudgment by a single self-supervised learning strategy. Moreover, any number of strategy combinations are supported.

[0082] In an exemplary embodiment, after the multi-modal sample set of the source domain and the multi-modal sample set of the target domain are respectively mapped to the public feature space, the method further comprises:

[0083] At least three data prediction models including a predictor neural network, an extreme gradient boosting model and a search model are trained based on the multi-modal sample set of the source domain respectively.

[0084] Each data prediction model is respectively assigned to a different single self-supervised learning strategy, and the data prediction model and the self-supervised learning strategy correspond to each other.

[0085] The multiple different self-supervised learning strategies are combined to obtain different self-supervised learning strategies.

[0086] In this embodiment, the predictor neural network includes but is not limited to the predictor of the model to be trained, a CNN (Convolutional Neural Network), a deep learning model such as a Transformer, and is good at capturing high-dimensional nonlinear features. The extreme gradient boosting model refers to an ensemble learning model based on decision trees, which is suitable for structured data and low-dimensional features, including but not limited to an XGBoost model and a Gradient Boosting Model. The search model refers to a model based on similarity matching and retrieval mechanism to predict categories or labels, including but not limited to an index search model and a semantic search model. The search model can search the most similar vector from a plurality of vectors through semantic similarity.

[0087] In specific implementation, after mapping the multi-modal sample set of the source domain and the multi-modal sample set of the target domain to the public feature space respectively, at least three data prediction models can be trained based on the multi-modal sample set of the source domain respectively, including at least a predictor neural network, an extreme gradient boosting model, and a search model. Subsequently, each model is respectively assigned (bound) to a specific self-supervised learning strategy, for example, the predictor neural network, the extreme gradient boosting model, and the search model are respectively assigned to strategies A, B, and C, that is, a function or class is defined in each strategy to call the corresponding model to generate pseudo labels. Then, a plurality of different self-supervised learning strategies are combined to obtain different self-supervised learning strategies.

[0088] The self-supervised learning strategies A, B, and C are combined to obtain a plurality of different self-supervised learning strategies A, B, C, AB, AC, BC, and ABC. Among them, AB refers to a strategy of adding pseudo labels by calling the predictor neural network and the extreme gradient boosting model. The pseudo label is only assigned to the sample when the pseudo labels output by the predictor neural network + XGBoost are consistent.

[0089] In this embodiment, each single strategy is assigned an independent classification prediction algorithm, which can enable each strategy to focus on performing the most suitable self-supervised task, and the result agreed by multiple strategies in the pseudo label adding stage is more reliable, reducing the adverse effects of single strategy misjudgment.

[0090] As shown in FIG. 6, in an exemplary embodiment, S206 includes: Figure 5

[0091] S246, using the classification prediction algorithm corresponding to each different self-supervised learning strategy, respectively predicting the first multi-modal sample set and the second multi-modal sample set of the target domain to obtain a prediction result.

[0092] ​S256, determine the uncertainty and label accuracy of different self-supervised learning strategies based on the prediction results.

[0093] S266, determine the confidence of the pseudo-labels added by different self-supervised learning strategies for the first multi-modal sample set based on the uncertainty and accuracy.

[0094] S276, based on the confidence of the pseudo-labels, filter out the optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies.

[0095] The uncertainty is used to represent the credibility of the model prediction results, which can be quantified by confidence scores, entropy, etc. Confidence is a comprehensive index combining accuracy and uncertainty, which is a quantitative evaluation of the quality of pseudo-labels, used to guide the selection and use of pseudo-labels.

[0096] In specific implementation, for each self-supervised learning strategy, the corresponding classification prediction algorithm is used to predict the first multi-modal sample set and the second multi-modal sample set of the target domain, and the prediction results under each strategy are recorded. Subsequently, for the first multi-modal sample set with labels, based on the true labels of the samples in the first multi-modal sample set, the consistency proportion of the pseudo-labels and the true labels predicted by each strategy is calculated, and the label accuracy of each strategy is obtained. Subsequently, based on the pseudo-labels of the first multi-modal sample and the second multi-modal sample, the uncertainty of the strategy is determined by the maximum class probability or the entropy of the probability distribution, the greater the entropy, the higher the uncertainty. Subsequently, the label accuracy and uncertainty can be weighted and summed or in a more complex way to calculate the confidence of each strategy. Subsequently, compare the confidence scores of each strategy, select the strategy with the highest confidence as the target self-supervised learning strategy in the current round, and use the target self-learning strategy to generate the final pseudo-labels for the unlabeled samples of the target domain.

[0097] In this embodiment, by comprehensively considering the uncertainty and label accuracy of the strategy, the most suitable strategy for the current data distribution can be selected more objectively and accurately, and the effect of domain adaptation is improved.

[0098] In some embodiments, the multi-modal sample set of the source domain and the multi-modal sample set of the target domain are respectively mapped to a common feature space, comprising:

[0099] The source domain features are extracted from the multi-modal sample set of the source domain by the source domain feature extractor.

[0100] The target domain features are extracted from the multi-modal sample set of the target domain by the target domain feature extractor.

[0101] The source domain features and the target domain features are mapped to the common feature space.

[0102] The source domain feature extractor is a pre-trained or trained model, such as a fully connected neural network, a convolutional neural network, etc., used to extract high-dimensional semantic features of the input sample. The source domain features are the vector representations obtained after processing by the feature extractor, which are used for subsequent classification or domain alignment tasks. The target domain feature extractor can be the same as the source domain feature extractor, or a model that has been fine-tuned or retrained. The target domain features refer to the representations of the target domain samples in the feature space, which are used for subsequent alignment with the source domain features.

[0103] In specific implementation, the source domain features in the multi-modal samples of the source domain can be extracted by the source domain feature extractor, and the target domain features in the multi-modal samples of the target domain can be extracted by the target domain feature extractor. Subsequently, the source domain features and the target domain features can be mapped into a common feature subspace by a linear transformation matrix, a nonlinear neural network layer, an adversarial training module, etc. Since the source domain and the target domain have been mapped into a common feature space, the two domains can share the same predictor neural network.

[0104] In this embodiment, the source domain features and the target domain features are mapped into a common feature space by the feature extractors of the source domain and the target domain, so that the source domain and the target domain are as close as possible in the unified space, reducing the domain gap.

[0105] It should be understood that, although each step in the flowchart involved in each of the above-described embodiments is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above-described embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiments of the present application also provide a multi-modal data domain adaptation device for implementing the above-mentioned multi-modal data domain adaptation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-modal data domain adaptation device embodiments provided below can refer to the limitations of the multi-modal data domain adaptation method described above, which will not be repeated here.

[0107] In one exemplary embodiment, as Figure 6As shown, a multi-modal data domain adaptation apparatus 600 is provided, comprising a data acquisition module 610, a data mapping module 620, a strategy screening module 630, a self-learning module 640 and a domain adaptation module 650, wherein:

[0108] The data acquisition module 610 is configured to acquire a multi-modal sample set of a source domain and a multi-modal sample set of a target domain, wherein the multi-modal sample set of the target domain comprises a first multi-modal sample set with labels and a second multi-modal sample set without labels.

[0109] The data mapping module 620 is configured to map the multi-modal sample set of the source domain and the multi-modal sample set of the target domain to a common feature space respectively in each round of the adaptation process.

[0110] The strategy screening module 630 is configured to perform self-supervised learning on the first multi-modal sample set and the second multi-modal sample set of the target domain by using different self-supervised learning strategies, and screen an optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies.

[0111] The self-learning module 640 is configured to add pseudo labels to samples in the second multi-modal sample set without labels according to the target self-supervised learning strategy, so as to update the multi-modal sample set of the target domain.

[0112] The domain adaptation module 650 is configured to perform domain adaptation processing on a to-be-trained model based on the updated multi-modal sample set of the target domain and the multi-modal sample set of the source domain.

[0113] In some exemplary embodiments, the strategy screening module 630 is further configured to add pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain by using different self-supervised learning strategies, the different self-supervised learning strategies are self-supervised learning strategies obtained by combining at least three self-supervised learning strategies, to determine label accuracy rates of the different self-supervised learning strategies for adding pseudo labels to the first multi-modal sample set based on the first multi-modal sample set with labels, and to determine a self-supervised learning strategy with the highest label accuracy rate as the target self-supervised learning strategy.

[0114] In some exemplary embodiments, the strategy screening module 630 is further configured to add pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain by using a classification prediction algorithm corresponding to the self-supervised learning strategy in the case that the self-supervised learning strategy is a single strategy, and to add pseudo labels to the first multi-modal sample set and the second multi-modal sample set of the target domain by using classification prediction algorithms corresponding to single strategies in the mixed strategy respectively in the case that the self-supervised learning strategy is a mixed strategy, and to determine the pseudo labels as final pseudo labels of the multi-modal samples if the pseudo labels added by each single strategy are consistent for each multi-modal sample.

[0115] In some example embodiments, the strategy screening module 630 is further configured to train at least three data prediction models based on the source domain multi-modal sample set respectively, the data prediction models including at least a predictor neural network, an extreme gradient boosting model and a search model, and each data prediction model is assigned to a different single self-supervised learning strategy, and the data prediction model and the self-supervised learning strategy correspond to each other, and different self-supervised learning strategies are combined to obtain different self-supervised learning strategies.

[0116] In some example embodiments, the strategy screening module 630 is further configured to use a classification prediction algorithm corresponding to each of the different self-supervised learning strategies to respectively predict the first multi-modal sample set and the second multi-modal sample set of the target domain to obtain prediction results, determine the uncertainty and the label accuracy of the different self-supervised learning strategies based on the prediction results, determine the confidence of the pseudo label added by the different self-supervised learning strategies for the first multi-modal sample set based on the uncertainty and the accuracy, and screen the optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies based on the confidence of the pseudo label.

[0117] In some example embodiments, the data mapping module 620 is further configured to extract source domain features from the source domain multi-modal sample set through a source domain feature extractor, extract target domain features from the target domain multi-modal sample set through a target domain feature extractor, and map the source domain features and the target domain features to a common feature space.

[0118] The above-mentioned various modules in the multi-modal data domain adaptation device can be realized by software, hardware and combinations thereof in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0119] In one example embodiment, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in FIG. 6. Figure 7As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the source domain multi-modal data and the target domain multi-modal data and the like. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a multi-modal data domain adaptation method.

[0120] Those skilled in the art can understand that, Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0121] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in any one of the above multi-modal data domain adaptation method embodiments.

[0122] In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, and the computer program is executed by the processor to implement the steps in any one of the above multi-modal data domain adaptation method embodiments.

[0123] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to implement the steps in any one of the above multi-modal data domain adaptation method embodiments.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0126] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0127] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A multimodal data domain adaptation method, characterized in that: The method comprises: Acquire a multimodal sample set of a source domain and a multimodal sample set of a target domain, wherein the multimodal sample set of the target domain includes a first multimodal sample set with labels and a second multimodal sample set without labels; In each round of adaptation, the multimodal sample set of the source domain and the multimodal sample set of the target domain are respectively mapped to a common feature space; Using different self-supervised learning strategies to perform self-supervised learning on the first multimodal sample set and the second multimodal sample set of the target domain, and selecting the optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies; adding pseudo labels to samples in the unlabeled second multimodal sample set according to the target self-supervised learning strategy to update the multimodal sample set of the target domain; Based on the updated multimodal sample set of the target domain and the multimodal sample set of the source domain, domain adaptation processing is performed on the model to be trained.

2. The method according to claim 1, characterized in that The adopting different self-supervised learning strategies to perform self-supervised learning on the first multimodal sample set and the second multimodal sample set of the target domain, and selecting the optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies, includes: adding pseudo labels to the first multimodal sample set and the second multimodal sample set of the target domain using different self-supervised learning strategies, wherein the different self-supervised learning strategies are self-supervised learning strategies obtained by combining at least three self-supervised learning strategies; Determining, based on the labeled first multimodal sample set, label accuracy rates of adding pseudo labels to the first multimodal sample set using different self-supervised learning strategies; The self-supervised learning strategy with the highest label accuracy is determined as the target self-supervised learning strategy.

3. The method according to claim 2, characterized in that The adding pseudo labels to the first multimodal sample set and the second multimodal sample set of the target domain by adopting different self-supervised learning strategies includes: In a case where the self-supervised learning strategy is a single strategy, adding pseudo labels to the first multimodal sample set and the second multimodal sample set of the target domain by using a classification prediction algorithm corresponding to the self-supervised learning strategy; When the self-supervised learning strategy is a hybrid strategy, pseudo labels are added to the first multimodal sample set and the second multimodal sample set of the target domain through the classification prediction algorithm corresponding to a single strategy in the hybrid strategy. If the pseudo labels added by each single strategy are consistent for each multimodal sample, the pseudo labels are determined as the final pseudo labels of the multimodal samples.

4. The method according to claim 1, wherein The adopting different self-supervised learning strategies to perform self-supervised learning on the first multimodal sample set and the second multimodal sample set of the target domain, and selecting the optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies, includes: Using classification prediction algorithms corresponding to different self-supervised learning strategies, respectively predicting the first multimodal sample set and the second multimodal sample set of the target domain to obtain prediction results; Based on the prediction results, determining the uncertainty and label accuracy of different self-supervised learning strategies; Determining, based on the uncertainty and the accuracy, confidence levels of pseudo labels added to the first multimodal sample set by different self-supervised learning strategies; Based on the confidence of the pseudo-label, the optimal target self-supervised learning strategy in the current round is selected from different self-supervised learning strategies.

5. The method according to any one of claims 1 to 4, characterized in that After mapping the multimodal sample set of the source domain and the multimodal sample set of the target domain to a common feature space respectively, the method further includes: Training at least three data prediction models based on the multimodal sample set of the source domain, the data prediction models comprising at least a predictor neural network, an extreme gradient boosting model, and a search model; Assigning each data prediction model to a different single self-supervised learning strategy, wherein the data prediction model corresponds to the self-supervised learning strategy in a one-to-one manner; By combining multiple different self-supervised learning strategies, different self-supervised learning strategies are obtained.

6. The method according to any one of claims 1 to 4, characterized in that Mapping the multimodal sample set of the source domain and the multimodal sample set of the target domain to a common feature space respectively includes: Extracting source domain features from the multimodal sample set of the source domain by a source domain feature extractor; Extracting target domain features from the multimodal sample set of the target domain by a target domain feature extractor; The source domain features and the target domain features are mapped to a common feature space.

7. A multimodal data domain adaptation device, characterized in that: The device comprises: a data acquisition module, configured to acquire a multimodal sample set of a source domain and a multimodal sample set of a target domain, wherein the multimodal sample set of the target domain includes a labeled first multimodal sample set and an unlabeled second multimodal sample set; a data mapping module, configured to map the multimodal sample set of the source domain and the multimodal sample set of the target domain to a common feature space in each round of adaptation; a strategy screening module, configured to perform self-supervised learning on the first multimodal sample set and the second multimodal sample set of the target domain using different self-supervised learning strategies, and screen out the optimal target self-supervised learning strategy in the current round from the different self-supervised learning strategies; a self-learning module, configured to add pseudo labels to samples in the unlabeled second multimodal sample set according to the target self-supervised learning strategy, so as to update the multimodal sample set of the target domain; The domain adaptation module is used to perform domain adaptation processing on the to-be-trained model based on the updated multimodal sample set of the target domain and the multimodal sample set of the source domain.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images