A robust multimodal active learning method based on pre-trained language model

Abstract

The present invention discloses a robust multimodal active learning method based on a pre-trained language model, which adopts a two-stage hybrid strategy. First, in terms of uncertainty measurement, a masked language loss based on momentum distillation is used as an uncertainty proxy. It inherits the advantage of cold-start active learning strategies such as ALPS that use masked language loss as a metric, and considers the uncertainty of samples from the input quality itself. At the same time, it fully considers the impact of specific downstream tasks on samples, and proposes a momentum distillation method to measure the changes in pre-training loss on specific tasks. In addition, in order to address the problem that the active selection strategy is easily affected by collective outliers, the distance between the pseudo-label probability predicted during model training and other label probabilities is used as an outlier indicator, and the value of this outlier indicator is used as a basis for cluster initialization selection, thereby reducing the selection of outliers during diversity clustering and improving the robustness of the active selection algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of semi-supervised active learning, and in particular to a robust multimodal active learning method based on a pre-trained language model. Background Art

[0002] Active learning (AL) techniques seek to find and label the most valuable examples to train high-performance classifiers with the least labeling effort. It includes a discussion of scenarios where queries can be formulated, and an overview of the query strategy framework. Traditional machine learning requires relatively few labeled samples, so early AL was rarely given the value it deserves. With the development of deep learning (DL), breakthroughs have been made in various fields. At the same time, deep learning (DL) requires a lot of data, and the performance of DL models will scale monotonically with the addition of additional training data. However, obtaining a large number of high-quality annotated datasets requires a large number of people, which is impractical in disciplines that require deep expertise. Therefore, deep active learning (DAL) has gained widespread popularity as a viable method to improve model performance while reducing expensive labeling costs. The combination of DL and AL has been widely used in various fields, including image recognition, text classification, visual question answering, etc.

[0003] There are three main deep active learning approaches: uncertainty-based, diversity-based, and a hybrid of uncertainty and diversity.

[0004] Uncertainty-based DAL selects new data samples to minimize the uncertainty of the base learner. Some popular uncertainty-based methods include entropy-based methods (the entropy of the label probability distribution is used as a measure), minimum confidence (it selects the data with the lowest posterior probability for the most likely label), etc. Most of these methods are based on the uncertainty of the classifier and select the unlabeled data with the highest uncertainty score. However, for deep learning, although the neural networks used for classification tasks produce probability distributions on the labels, their softmax output scores are poor confidence estimates. A more general approach is to use Bayesian methods to estimate uncertainty. For example, multiple forward passes can be used to obtain uncertainty estimates. However, this method is limited by the randomness of random dropout, and multiple feedforward runs are required to obtain results during the test phase, so it is not a stable and efficient method. Recently, some other uncertainty-based methods use results such as KL divergence, masked language loss, and gradient embedding of the last layer as uncertainty measures.

[0005] Diversity-based methods select data points that represent the distribution of the unlabeled data pool. A typical approach is clustering methods, such as KMeans or KMeans++, which mainly select centroids by iteratively sampling points proportional to their squared distance to the nearest selected centroid.

[0006] Uncertainty-based active learning methods mainly consider the uncertainty score of samples, while diversity-based methods mainly start from the distribution of samples. The two methods have their own advantages and disadvantages and complement each other. Therefore, a hybrid-based method is proposed to fully integrate the advantages of the two methods. The hybrid method weighs uncertainty and representativeness in query selection.

[0007] The hybrid method is a fusion of the diversity method and the uncertainty method. It usually adopts a two-stage method. In the first stage, the uncertainty score is used as a metric, and then a clustering algorithm is used in the second stage to select the most representative points as the selected samples. For example, active learning based on pre-training loss (ALPS) and batch active learning based on multivariate gradient embedding (BADGE) use masked language loss and classification gradient vector as metrics, respectively, and then use KMeans or KMeans++ for diversity clustering. However, this method only extracts problematic samples close to the cluster center and is difficult to distinguish outliers.

[0008] In some multimodal tasks, such as visual question answering (VQA) and multimodal hate speech detection (MHSD), active learning selection algorithms tend to select collective outliers that harm model performance. Especially in the cold start phase, due to data scarcity and model instability, active learning strategies can easily learn these outliers, which often makes the performance of active selection algorithms lower than random selection strategies, reducing the query efficiency of active learning algorithms. Summary of the invention

[0009] In view of the problem that existing active learning algorithms are not robust enough to outliers in multimodal tasks, the present invention proposes a robust multimodal active learning method based on a pre-trained language model to improve the robustness and uncertainty of the model, thereby improving the query efficiency of the active learning algorithm.

[0010] In order to achieve the above object, the present invention provides the following technical solutions:

[0011] The present invention provides a robust multimodal active learning method based on a pre-trained language model, which adopts a two-stage active learning method. In the first stage, an uncertainty embedding vector is learned by an uncertainty metric learner, and in the second stage, robust clustering is performed by a robust clusterer. For each cluster center, the sample set closest to it is selected and sent to annotators for annotation; wherein:

[0012] The uncertainty metric learner includes a masked language model loss based on momentum distillation and an outlier indicator. The masked language model loss based on momentum distillation calculates the relative entropy (KL divergence) between the pseudo target mask generated by the momentum pre-trained model without fine-tuning and the target mask generated by the pre-trained model with fine-tuning through the momentum distillation method, and dynamically weights it with the masked language loss; the outlier indicator evaluates whether a sample is an outlier by calculating the margin between the probability of the pseudo classification label and the probability of other classification labels;

[0013] The robust clusterer utilizes the z-score method to identify normal and outlier samples, and clusters and selects data samples through a robust clustering method with outlier removal.

[0014] Furthermore, in the momentum distillation-based masked language model loss, the input tokens are randomly masked with a probability of 0.15, and the pre-training task calculates the loss between these masked tokens and the true value.

[0015] Furthermore, the loss calculation formula of the masked language model based on momentum distillation is:

[0016]

[0017] Among them, θ represents the model parameters from the pre-trained model and is updated during training, β is a hyperparameter, is the masked language model loss, y mask It is represented as a one-hot vocabulary distribution with a true value token probability of 1, t = {t0, ..., t m-1} is the input subtag, q mask (t) represents the pseudo-prediction probability of the momentum model for the mask token, p mask (t) represents the predicted probability of the multimodal pre-trained model for the mask tag, X U is a pool of unlabeled datasets containing every (t, y) pair.

[0018] Furthermore, the calculation formula of the pseudo label probability margin PPM is:

[0019]

[0020]

[0021] in, is the pseudo label y predicted by the classifier model * The logits probability, is the maximum probability of other pseudo labels, and E represents the number of iterations.

[0022] Furthermore, the pseudo-label probability margin PPM is obtained when using the active learning strategy for data selection and is used for robust clustering in the second stage.

[0023] Furthermore, the z-score value of each sample in the unlabeled dataset is expressed as:

[0024] zscore i =(PPM i -μ) / σ

[0025] PPM i is the pseudo-predicted log-probability marginal value of the i-th instance in the unlabeled dataset, and μ is the PPM in the entire sample space i The average value of the value, σ is PPM i The standard deviation over the entire sample space.

[0026] Furthermore, the outlier sample is represented as:

[0027] |zscore i |>z thr

[0028] where z thr Set to 2.5, 3, or 3.5, greater than z thr The value of is an outlier, |zscore i The smaller the value of |, the more normal it is.

[0029] Furthermore, the method of obtaining the most representative point and judging whether the point is an outlier is as follows: As the input of KMeans clustering algorithm, |zscore i |It is an auxiliary indicator for outlier detection.

[0030] Furthermore, the robust clustering algorithm with outlier removal is as follows:

[0031] First, calculate the zscore of all unlabeled dataset pools i | value, and sort in ascending order, select |zscore i The minimum value of | is taken as the initial center set C k ;

[0032] Secondly, calculate other unlabeled samples X based on the Euclidean distance u With cluster center C k distance, each point receives its nearest centroid c ki To form K clusters, recalculate the centroid of each cluster until the centroid does not change;

[0033] Finally, for each cluster center, the set of samples closest to it is selected, and the selected samples are sent to the annotators for annotation.

[0034] Further, other unlabeled samples X are calculated based on the Euclidean distance u With cluster center C k The formula for the distance is:

[0035]

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention provides a robust multimodal active learning method based on a pre-trained language model, which adopts a two-stage hybrid strategy. First, in terms of uncertainty measurement, a masked language loss based on dynamic quantity distillation is used as an uncertainty proxy. It inherits the cold-start active learning strategy of ALPS, which uses masked language loss as a metric and considers the uncertainty of samples from the input quality itself. At the same time, the influence of specific downstream tasks on samples is fully considered, and a momentum distillation method is proposed to measure the changes in pre-training loss on specific tasks. In addition, in order to address the problem that the active selection strategy is easily affected by collective outliers, the distance between the pseudo-label probability predicted during model training and other label probabilities is used as an outlier indicator, and the value of this outlier indicator is used as a basis for cluster initialization selection, thereby reducing the selection of outliers during diversity clustering and improving the robustness of the active selection algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0039] Figure 1 A flowchart of the architecture of a robust multimodal active learning method based on a pre-trained language model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] While datasets are an important driver of progress in deep learning, they can contain biases that lead to erroneous conclusions. Our work is closely related to the work on eliminating data selection bias. In the prior art, BADGE utilizes gradient embedding as the uncertainty measure for each unlabeled data sample in the first stage, and then clusters them through KMeans++ in the second stage. ALPS adopts a similar process, but replaces the uncertainty measure with Masked Language Loss (MLM) and uses KMeans as the diversity measure. In addition, while ALPS or BADGE uses KMeans or KMeans++ clustering to obtain the diversity of batch data, it has difficulty distinguishing outliers because it only extracts problematic samples close to the cluster center.

[0041] In order to solve the outlier problem in certain multimodal tasks, we proposed a robust multimodal active learning method based on pre-trained language models (RALPLM), which is a new method that uses the characteristics of pre-trained language models to assist active query strategies. Inspired by the masked language model (MLM) loss as a proxy for uncertainty, and considering the impact of training iterations on the MLM loss under specific downstream tasks, we proposed a momentum distillation method to measure the change between the original pre-trained model loss and the task-specific pre-training loss. At the same time, to ensure the quality of the data we selected, we introduced the robust clustering with outlier removal (RCOR) algorithm, which uses clustering methods and outlier indicators to select the outlier removal instances with the most information. The entire architecture is as follows Figure 1 As shown in Figure 2, our active learning algorithm framework belongs to a two-stage algorithm, which is generally divided into an uncertain vector learner and a robust clustering module.

[0042] In order to better understand the technical solution, the method of the present invention is described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the present invention provides a robust multimodal active learning method based on a pre-trained language model, which belongs to a pool-based active learning method. Each time, a batch of data is selected from the unlabeled data set through an active selection algorithm for labeling. We represent the labeled data and the annotation pool as (X L , Y L ), the unlabeled data pool is denoted as X U The goal of active learning is to select appropriate labeled candidate data from the data pool according to query strategy A under a limited label budget. Then the sample candidates are labeled and added to (X L , Y L ) to obtain the best performance of the learner for a given task. (X L , Y L) grows at each stage.

[0044] The method of the present invention is mainly applied to multimodal tasks using pre-trained visual language models. Multimodal pre-training models usually use Transformers that currently perform well in various fields to build cross-modal models. In general, multimodal pre-training models that can be used include VisualBERT, LXMERT, etc. It takes the visual area of ​​the image and the text tag of the sentence as input x. Given the weight W, the multimodal pre-training encoder h(x; W) maps x to a d-dimensional hidden representation. In order to obtain better multimodal representations, pre-training tasks play an important role. Generally, image-text matching (ITM) tasks and masked language modeling (MLM) tasks are common to most multimodal pre-training tasks. The image-text matching (ITM) task is a binary classification task that predicts whether an image or text matches. The joint embedding of the input image-text pair is calculated as the final hidden state of the [CLS] tag. Then a fully connected layer and a softmax function are followed to predict whether the image and text pair match, while the masked language modeling (MLM) task randomly masks the input image token or text token with a probability of 15%, which is a multi-classification task of predicting masked tokens. The masked language modeling task that generates the loss may come from text tags, image tags, or both. The masked language modeling (MLM) loss can be applied to AL as an uncertainty embedding. The present invention adds a distillation task as an extension between the pseudo target prediction probability generated by the original pre-trained model and the target prediction probability generated by the fine-tuned pre-trained model.

[0045] In each iteration, a small batch of instances of size b is selected from the unlabeled dataset to query their labels. The query formula is:

[0046] Q=xqstqb=1

[0047] Where x represents the instances of the unlabeled dataset, q represents the proportion of each query in the entire dataset, and b is the batch size of each query. We use the pre-trained model h(x;W) and the first token head [CLS] to fine-tune the multimodal downstream task classifier f. Assume that f is given by Parameterization, is the weight of the last layer, It is composed of the weights of all previous layers. Where F is a nonlinear function that maps input x to a vector of confidence scores for each label, and σ is a softmax function. The confidence scores are also called logits vectors, which represent the predicted probability of each label. The present invention uses logits vectors to help identify collected outliers.

[0048] In order to effectively L, Y L ) to sample so that the downstream task classifier f can perform better. In each iteration, we use the query strategy A to extract U Get the appropriate (X L , Y L ) and query their labels. Query strategy A depends on obtaining model S t In the first sampling, it is equal to the parameters of the pre-trained model. After that, S t The parameters of come from the fine-tuning of the previous iteration.

[0049] The query strategy is the key to active learning; its principle is to select the most valuable instances that make the best contribution to model learning. The criteria of value can be based on uncertainty, diversity and robustness. The present invention adopts a two-stage active learning method, which ensures uncertainty and diversity at the same time. In the first stage, the uncertainty embedding vector is learned by the uncertainty metric learner, and in the second stage, robust clustering is performed by the robust clusterer. For each cluster center, the sample set closest to it is selected and sent to the annotator for annotation; among them:

[0050] The uncertainty metric learner includes a masked language model loss based on momentum distillation and an outlier indicator. The masked language model loss based on momentum distillation calculates the relative entropy (KL divergence) between the pseudo target mask generated by the momentum pre-trained model without fine-tuning and the target mask generated by the pre-trained model with fine-tuning through the momentum distillation method, and dynamically weights it with the masked language loss; the outlier indicator evaluates whether a sample is an outlier by calculating the margin between the probability of the pseudo classification label and the probability of other classification labels;

[0051] The robust clusterer utilizes the z-score method to identify normal and outlier samples, and clusters and selects data samples without outliers through a robust clustering method with outlier removal.

[0052] The whole framework ensures uncertainty and diversity and is robust to outliers. The specific description is as follows.

[0053] 1. Uncertainty Vector Learner

[0054] Pre-trained language models can leverage external language knowledge as an auxiliary knowledge base to demonstrate better performance without fine-tuning downstream tasks. This principle facilitates zero-shot or few-shot learning using pre-trained language models. Downstream tasks that leverage the reconstruction of pre-trained models generally transform the input into a cloze form, which is most similar to the masked language model pre-training task. From this perspective, the pre-trained masked language model (MLM) task can evaluate the quality of a dataset in an unsupervised setting. Another piece of evidence is that the pre-trained masked language model (MLM) can be viewed as a pseudo-log likelihood score to evaluate the acceptability of the associated language. The masked language model (MLM) objective can be used as a random maximum pseudo-likelihood estimate (MPLE), which can evaluate the unexpected embedding of the input. Based on the above discussion, some active learning selection strategies such as ALPS also gradually use the loss of the masked language model (MLM) pre-training to guide the AL process to sample text examples. Another benefit of using the MLM loss is that it helps with active learning cold start processing, which is different from most active learning methods that rely on random training initialization.

[0055] 1.1 Masked Language Model Loss Based on Momentum Distillation

[0056] The proposed method uses the pre-training task for a deep multimodal active learning scenario and computes the masked multimodal language model loss. It is the same as the masked language model loss (MLM), the input tokens are randomly masked with a probability of 0.15, and the pre-training task predicts these masked tokens. As for the multimodal setting, the masked words can also be predicted from the visual modality, thus resolving ambiguity. All image and text tokens can be represented as m input sub-tokens t = {t0, ..., t m-1}. Randomly select 15% of the text tags and mask It is represented as a one-hot vocabulary distribution with a true value token probability of 1, p mask (t) represents the predicted probability of the multimodal model for the masked token, and the masked language model (MLM) loss can be viewed as a cross entropy loss:

[0057]

[0058] where θ represents the model parameters from the pre-trained model and is updated during training. U is a pool of unlabeled datasets containing every (t, y) pair.

[0059] can be viewed as an uncertainty proxy for difficult samples in the query language, but it ignores the impact of training iterations on downstream tasks on pre-training loss, causing this heterogeneous approach to gain gains only in the first few iterations but quickly degrade in performance. To address this problem, the present invention introduces a momentum distillation method that learns from pseudo targets generated by a momentum pre-trained model (a pre-trained model without fine-tuning), which can eliminate performance degradation over training iterations. Let q mask (t) represents the pseudo-prediction probability of the momentum model for the mask token, we can define the final loss of the transformation as:

[0060]

[0061] Here, β is a hyperparameter which we define as 0.3 for simplicity.

[0062] 1.2 Outlier Indicator

[0063] Selecting outliers from unlabeled datasets can harm the trained model. While our uncertainty embedding is mainly based on pre-training loss and is therefore somewhat resistant to noise, it is still largely affected by outliers since the pre-trained parameters are also updated in each training iteration. A better way to address outliers in a dataset is to identify and remove these outliers. However, most outlier identification methods in datasets need to be built on labeled datasets. The Area Under Margin (AUM) measures whether a sample is mislabeled (outlier) using the margin between the probability value of the true value label and the probability value of the other highest unassigned labels. The present invention extends this work and expands it to adapt to the application scenarios of active learning, and then proposes a learnable metric with unlabeled cases. The present invention converts the pseudo label y * Denote the most frequent label predicted by the classifier, and then calculate the pseudo-label probability margin (PPM) as:

[0064]

[0065] in is the logits probability of the pseudo-label predicted using the classifier model, is the maximum probability of other pseudo labels. E represents the number of iterations. It should be noted that PPM does not depend on the true value label. It is obtained when using the active learning strategy for data selection and is used for the subsequent robust clustering.

[0066] 2. Robust Clusterer

[0067] loss Active learning strategies can be guided to study the uncertainty of unlabeled data sets. However, the most uncertain examples are not necessarily the best representatives, and selecting the most uncertain examples will reduce the generalization performance of the model. Another problem is that collective outliers will damage the performance of the model and guide the active learning strategy to perform worse than random selection. Then, we proposed a robust clustering method for collective outliers to filter out the most representative samples. AUM can illustrate the confidence (logits) range of simple, difficult, and mislabeled examples. For samples that are easy to learn, the spacing value is positive and large. For outlier examples, the spacing value is negative and large. Based on these findings, we believe that the PPM (see the above formula) values ​​between outlier samples and normal samples are different. Then, the present invention uses the z-score method in statistics to identify normal and outlier samples. The z-score value of each sample of the unlabeled data set is expressed as:

[0068] zscore i =(PPM i -μ) / σ

[0069] PPM i is the pseudo-predicted log-probability marginal value of the i-th instance in the unlabeled dataset, and μ is the PPM in the entire sample space i The average value of σ is the standard deviation of PPMi in the entire sample space. The outlier sample can be expressed as:

[0070] |zscore i |>z thr

[0071] where z thr Can be set to 2.5, 3 or 3.5, greater than z thr The value of is an outlier. |zscore i The smaller the value of |, the more normal it is.

[0072] In order to obtain the most representative point and determine whether the point is an outlier, the present invention allows As the input of KMeans clustering algorithm, |zscore i | is an auxiliary indicator for detecting outliers. Then we proposed the RCOR algorithm as follows:

[0073] (1) First, calculate the zscore of all unlabeled dataset pools i | value, and sort in ascending order. Select | zscore i The minimum value of | is taken as the initial center set C k .

[0074] (2) Secondly, calculate other unlabeled samples X based on the Euclidean distance u With cluster center Ck The distance is:

[0075]

[0076] (3) Each point receives its nearest centroid To form K clusters, recalculate the centroid of each cluster until the centroid does not change.

[0077] Finally, for each cluster center, the present invention selects the sample set closest to it, and the selected samples are sent to the annotator for annotation.

[0078] The analysis on the datasets of multimodal hate speech detection and multimodal visual question answering shows that compared with other active learning strategies, the proposed method (RALPLM) achieves a better balance between diversity and uncertainty, while being significantly robust to abnormal data that is detrimental to model performance, achieving higher performance.

[0079] In summary, the robust multimodal active learning method (RALPLM) based on pre-trained language model of the present invention effectively combines multimodal pre-trained language model and deep active learning by adopting a two-stage hybrid method, and uses momentum distillation to mask the language model loss as a representative of uncertainty, while considering the data quality and specific tasks of the samples. A robust clustering with outlier removal (RCOR) method is proposed, which clusters and selects data samples without outliers, and uses the marginal value of the prediction probability of downstream tasks as the outlier indicator. Then we apply these two indicators to robust clustering to obtain the most valuable annotated samples, which improves the robustness and uncertainty of the model, thereby improving the query efficiency of the active learning algorithm.

[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust multimodal active learning method based on a pre-trained language model, characterized in that: The method is applied to multimodal tasks using pre-trained visual language models, which use Transformers to build cross-modal models, including VisualBERT and LXMERT; the pre-trained visual language model takes the visual area of ​​the image and the text tag of the sentence as input, and adopts a two-stage active learning method. The first stage learns the uncertainty embedding vector through the uncertainty metric learner, and the second stage performs robust clustering through the robust clusterer. For each cluster center, the sample set closest to it is selected and sent to the annotator for annotation; wherein: The uncertainty metric learner includes a mask language model loss based on momentum distillation and an outlier indicator. The mask language model loss based on momentum distillation calculates the relative entropy between the pseudo target mask generated by the momentum pre-trained model without fine-tuning and the target mask generated by the pre-trained model with fine-tuning through the momentum distillation method, and dynamically weights it with the mask language loss; the outlier indicator evaluates whether a sample is an outlier by calculating the margin between the probability of the pseudo classification label and the probability of other classification labels; The robust clusterer utilizes the z-normalization method to identify normal and outlier samples, and clusters and selects data samples through a robust clustering method with outlier removal.

2. The robust multimodal active learning method based on a pre-trained language model according to claim 1, characterized in that: In the masked language model loss based on momentum distillation, the input tokens are randomly masked with a probability of 0.15, and the pre-training task calculates the loss between these masked tokens and the true value.

3. The robust multimodal active learning method based on a pre-trained language model according to claim 1, characterized in that: The calculation formula of the masked language model loss based on momentum distillation is: Among them, θ represents the model parameters from the pre-trained model and is updated during training, β is a hyperparameter, is the masked language model loss, g mask It is represented as a one-hot vocabulary distribution with a true value token probability of 1, t = {t0, ..., t m-1 } is the input subtag, q mask (t) represents the pseudo-prediction probability of the momentum model for the mask token, p mask (t) represents the predicted probability of the multimodal pre-trained model for the mask tag, X U is a pool of unlabeled datasets containing every (t, y) pair.

4. The robust multimodal active learning method based on a pre-trained language model according to claim 1, characterized in that: The calculation formula of pseudo label probability margin PPM is: in, is the pseudo label y predicted by the classifier model * The logits probability, is the maximum probability of other pseudo labels, and E represents the number of iterations.

5. The robust multimodal active learning method based on a pre-trained language model according to claim 4, characterized in that: The pseudo-label probability margin PPM is obtained when using the active learning strategy for data selection and is used for robust clustering in the second stage.

6. The robust multimodal active learning method based on a pre-trained language model according to claim 5, characterized in that: The z-normalized value of each sample of the unlabeled dataset is expressed as: zscore i =(PPM i -m) / s PPM i is the pseudo-predicted log-probability marginal value of the i-th instance in the unlabeled dataset, and μ is the PPM in the entire sample space i The average value of the value, σ is PPM i The standard deviation over the entire sample space.

7. The robust multimodal active learning method based on a pre-trained language model according to claim 6, characterized in that: The outlier samples are represented as: |zscore i |>from thr where z thr Set to 2.5, 3, or 3.5, greater than z thr The value of is an outlier.

8. The robust multimodal active learning method based on a pre-trained language model according to claim 7, characterized in that: The method to obtain the most representative point and determine whether it is an outlier is: As the input of KMeans clustering algorithm, |zscore i |It is an auxiliary indicator for outlier detection.

9. The robust multimodal active learning method based on a pre-trained language model according to claim 8, characterized in that: The robust clustering algorithm with outlier removal is as follows: First, calculate the zscore of all unlabeled dataset pools i | value, and sort in ascending order, select |zscore i The minimum value of | is taken as the initial center set C k ; Secondly, calculate other unlabeled samples X based on the Euclidean distance u With cluster center C k distance, each point receives its nearest centroid c ki To form K clusters, recalculate the centroid of each cluster until the centroid does not change; Finally, for each cluster center, the set of samples closest to it is selected, and the selected samples are sent to the annotators for annotation.

10. The robust multimodal active learning method based on a pre-trained language model according to claim 9, characterized in that: Calculate other unlabeled samples X based on the Euclidean distance u With cluster center C k The formula for the distance is: dist(X u ,C k )=||x ui -c ki ||(i=1,2,...,n)。

Images