A confidence learning based biased data detection method

Abstract

The application discloses a biased data detection method based on confidence learning, comprising the following steps: 1. constructing and representing a data set containing biased data; 2. constructing a multi-modal integrated prediction module; 3. training the multi-modal integrated prediction module; 4. constructing a confidence joint counting matrix and a joint probability distribution; and 5. filtering biased data according to a bias rate. Based on the confidence learning framework, the application comprehensively considers the feature expression of text and pictures, can estimate the joint probability distribution of the biased label and the correct label, thereby can screen out correct data, and effectively reduces the influence of the biased label on related tasks.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of identification and filtering of biased data, and particularly relates to a biased data detection method based on confidence learning. BACKGROUND

[0002] In the era of rapid development of information technology, we will receive and process various information through electronic devices every day, the most typical of which are text and picture content, and meanwhile, tag information is also more and more interwoven in multi-modal content. The tag information can not only highly summarize the key information of an object, but also can realize interconnection between objects by constructing internal links to form a relationship network. However, a large amount of biased data is brought by the various data information, which has a negative impact on many fields, especially on the recommendation system. Taking the tag information as an example, due to the social labeling form, the biased label problem widely exists in the real data set. For example, the authoritative public data sets such as ImageNet and MNIST also have biased labels, which usually manifest as labeling errors or label missing, and have a very serious impact on the model performance.

[0003] In recent years, there are various methods such as collaborative filtering, generative model, deep neural network to complete the implementation of the hashtag recommendation task, but most of the researches only use text information [Hashtag recommendation methods for twitter and sina weibo: A review. in Future Internet. 2021] or single picture plus text information. According to statistics, more than 42% of Twitter tweets contain multi-modal data, and the information displayed by multiple pictures and text may be complementary, and the two together can more comprehensively reflect the characteristics of the object. At the same time, the multi-modal data type increases the difficulty of identifying biased labels. In multi-modal content, a specific label may only be related to a certain modality. In this case, if only a single modality is considered, the label may be identified as a biased label, so the multi-modal characteristics of the data need to be fully considered. At present, the biased data filtering algorithm can be divided into two categories according to its processing method, which are biased sample removal [Ensemble-based noise detection: Noise ranking and visual performance evaluation. in Data Mining and Knowledge Discovery. 2014] and biased sample relabeling. Considering the operation efficiency, the sample removal method is more common, but the research on the combination of this method with the label recommendation task and the multi-modal data field still needs further development. In view of the above background and technology, an effective biased data detection scheme is needed to effectively extract and fuse multi-modal features and reduce the interference of biased data labels. SUMMARY

[0004] The present application is to solve the above-mentioned deficiencies in the prior art, and proposes a biased data detection method based on confidence learning, so as to estimate the joint probability distribution of the correct label and the biased label based on the confidence learning framework, and to consider the feature expression of text and picture, so as to screen out correct data and effectively reduce the influence of biased labels on related tasks.

[0005] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is as follows:

[0006] The biased data detection method based on confidence learning has the characteristics of the following steps:

[0007] Step 1, the sample data set containing biased data is denoted as X = {X1, X2, …, X k ,…,X K}, wherein K represents the number of samples in the sample data set X, X k represents the kth sample, and Let X represent the k-th sample. k The text, and Let X represent the k-th sample. k The nth word in the Chinese text, where N is the number of words in the text; Let X represent the k-th sample. k The image collection, and Let X represent the k-th sample. k The m-th image in the image set; M is the number of images in the image set; Let X represent the k-th sample. k It contains a set of biased labels, and Let X represent the k-th sample. k The g-th biased label in the biased label set, where G is the number of biased labels in the biased label set;

[0008] Let the set of unique, biased labels for all samples in the sample dataset X be denoted as... This indicates the c-th biased label, where C is the number of unique biased labels in the sample dataset X.

[0009] Step 2: Construct a multimodal ensemble prediction module, including: a text classifier, a visual classifier, and a fusion layer;

[0010] Step 2.1: Construct a text classifier based on the Bi-LSTM model;

[0011] Step 2.1.1: The text classifier uses an embedding layer to analyze the nth word. Processing yields the nth word. Embedded vector Thus obtain Embedded vector set

[0012] Step 2.1.2: The text classifier will embed the vector set E. k Inputting the text into a Bi-LSTM network and obtaining the word-level representation H of the text description n :

[0013] The forward network in the Bi-LSTM network according to arrive The embedding vectors are read sequentially, and the nth word is obtained using equation (1). Forward hidden state

[0014]

[0015] In equation (1), Represents the (n-1)th word The forward-hidden state;

[0016] The backward network in the Bi-LSTM network according to arrive The embedded vectors are read sequentially, and the nth word is obtained using equation (2). Backward hidden state

[0017]

[0018] In equation (2), Represents the (n+1)th word The backward hidden state;

[0019] The Bi-LSTM network then obtains the nth word in conjunction with the context through equation (3). The representation of Thus, the k-th sample X is obtained. k text Word representation

[0020]

[0021] Step 2.1.3: The text classifier uses average pooling to process the text. To obtain text global features Then, the global features are processed through a softmax classification layer. The process is performed to generate the k-th sample X using equation (4) under textual conditions only. k For the c-th biased label Predicted probability

[0022]

[0023] In equation (4), and They represent The c-th label The parameter matrix and bias vector are given below;

[0024] Step 2.2: Construct a visual classifier based on the pre-trained VGG-16 network;

[0025] Step 2.2.1: The visual classifier will use the image set of the k-th sample. The pre-trained VGG-16 model is inputted for processing, and the output of the second-to-last fully connected layer in the VGG-16 model is taken as the visual feature representation of the image set as a feature representation set of the image set; wherein, represents the feature representation of the mth image in the image set

[0026] Step 2.2.2, the visual classifier aggregates the feature representation set of the image set using the method of average pooling, to obtain the visual feature of the kth sample X k Step 2.2.3, the visual feature is processed using an MLP layer, and the hidden representation of the visual feature is obtained using formula (5) Step 2.2.4, the hidden representation is processed using another MLP layer, and the prediction probability of the kth sample X k under the cth biased label is generated using formula (6)

[0027]

[0028]

[0029] In formula (5) and formula (6), tanh represents the hyperbolic tangent function and serves as the activation function of the hidden layer, and Wc and bc respectively represent the parameter matrix and the bias vector corresponding to the hidden layer; and respectively represent the parameter matrix and the bias vector under the cth biased label

[0030] Step 2.3, the fusion layer integrates the prediction probabilities using the weighted average method as shown in formula (10) to obtain the final integrated prediction probability of the kth sample X k under the cth biased label

[0031]

[0032] In formula (7), α and β are the weight sizes of the text classifier and the visual classifier, respectively, satisfying α, β ∈ [0, 1] and α + β = 1; ​​​​​​​​​​

[0033] Step 3, training of the multi-modal integrated prediction module:

[0034] Step 3.1, constructing a cross-entropy loss function J of the multi-modal integrated prediction module using formula (8):

[0035]

[0036] In formula (8), S represents part of the samples in the multi-modal sample data set X as a training set, |S| represents the number of samples in the training set, represents the biased label set of the f-th sample X f in the training set;

[0037] Step 3.2, training the multi-modal integrated prediction module using the error back propagation algorithm, and minimizing the loss function J for updating the module parameters until the loss function J converges, thereby obtaining an optimal multi-modal integrated prediction model;

[0038] Step 4, calculating the confidence threshold t of the j-th biased label in the biased label set j using formula (9):

[0039]

[0040] In formula (9), represents a sample subset in the sample data set X with the j-th biased label , represents the prediction probability of any one sample x in the sample subset to the j-th biased label under the parameters θ of the optimal multi-modal integrated prediction model, and || represents the counting operation;

[0041] Step 4, constructing a confidence joint counting matrix and a joint probability distribution;

[0042] Step 4.1, according to the confidence threshold, estimating the correct label set y * in the sample data set X, thereby calculating the confidence joint counting matrix of the biased label set * and the correct label set y using formula (10) and formula (11):

[0043]

[0044]

[0045] In formula (10) and formula (11), denotes the value in the i-th row and j-th column of the matrix , i.e. the number of samples that have both the i-th biased label and the j-th correct label denotes the estimated data set that has both the i-th biased label and the j-th correct label ;

[0046] Step 4.2, estimate the joint probability distribution of the biased label set and the correct label set y * according to the confidence joint count matrix

[0047] Step 4.2.1, obtain the value in the i-th row and j-th column of the revised count matrix

[0048]

[0049] Step 4.2.2, obtain the joint probability distribution estimate of the biased label set and the correct label set using formula (13)

[0050]

[0051] In formula (13), is the joint probability distribution estimate in the i-th row and j-th column, which denotes the probability that a sample in the data has both the i-th biased label and the j-th correct label , and satisfies

[0052] Step 5, filter the biased data according to the bias rate:

[0053] Step 5.1, select a number of samples from in that make formula (14) minimum and form a biased data candidate set:

[0054]

[0055] In formula (14), denotes the prediction probability of any sample x in the sample subset for the i-th biased label under the parameter θ;

[0056] ​Step 5.2, calculate the marginal of each sample in the biased data candidate set by using formula (15) and use it to sort the samples in ascending order, to obtain the sorted biased data candidate set:

[0057]

[0058] In formula (15), mean represents the average pooling operation; is the i-th biased label in the sample data set X; represents the biased label set of the k-th sample X k ; represents the prediction probability of the i-th biased label k under the parameter theta for the k-th sample X ;

[0059] Step 5.3, select a certain proportion of data from the sorted biased data candidate set for filtering, to obtain a relatively clean data set.

[0060] The electronic device comprises a memory and a processor, and is characterized in that the memory is used to store a program supporting the processor to execute the biased data detection method, and the processor is configured to execute the program stored in the memory.

[0061] The computer readable storage medium comprises a computer program stored thereon, and is characterized in that the computer program is executed by the processor to execute the steps of the biased data detection method.

[0062] Compared with the prior art, the beneficial effects of the present application are as follows:

[0063] 1. The present application proposes a biased data detection method based on confidence learning, which uses a neural network integrated prediction module based on multi-modal data to output the average prediction probability of each label, and combines a confidence learning framework to calculate the joint probability distribution of the correct label and the biased label, and further filtering effectively reduces the biased rate of the data set.

[0064] 2. The present application is different from the way of constructing a robust model structure to reduce the interference of biased data. The method proposed by the present application starts from the data level and aims to identify and filter biased labels. Compared with model structure adjustment, it greatly improves the universality and universality, and has the characteristics of simplicity, effectiveness and easy implementation.

[0065] 3. The present application does not require any prior knowledge, such as part of the high-quality label data, which is obviously more in line with the actual situation. At the same time, it has strong expansibility and can be easily extended to almost all single-modal and multi-modal data sets.

[0066] 4. The present application introduces confidence learning, adopts a statistical method based on uncertainty estimation, does not need to make a random uniform biased label distribution assumption, and can process unbalanced data. At the same time, due to the confidence threshold, the prediction probability of the multi-modal integrated prediction module does not require absolute accuracy.

[0067] 5. The present application helps to improve the quality of recommended labels through the study of multi-modal data-oriented biased label detection, has certain practical significance for improving the quality of information retrieval service and user experience, and can help researchers design more effective related recommendation systems. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 A biased data detection framework based on confidence learning of the present application;

[0069] Figure 2 A text classifier structure of the present application;

[0070] Figure 3 A visual classifier structure of the present application. DETAILED DESCRIPTION

[0071] In the present example, a biased data detection method based on confidence learning is combined with a confidence learning framework, the average prediction probability of the multi-modal integrated prediction module under each label is taken as a confidence threshold to calculate the joint probability distribution of correct labels and biased labels, so that data filtering can be performed according to this standard to obtain a relatively clean data set. As shown in Figure 1 The confidence learning framework mainly includes two modules of multi-modal integrated prediction and confidence learning filtering. Among them, the multi-modal integrated prediction module is mainly responsible for predicting the label bias probability. As shown in Figure 2 , Figure 3 The text classifier uses a bidirectional LSTM network to extract text semantic information, and the visual classifier uses a pre-trained VGG network to extract visual information. Finally, the results of the two classifiers are combined through an integrated strategy to generate the final prediction probability. The module cross-validates with biased labels as the training standard to obtain the probability distribution of different samples under different categories. The confidence learning filtering module uses the results of the multi-modal integrated prediction module to calculate a count matrix similar to a confusion matrix, and then estimates the joint probability distribution of biased labels and correct labels. Then a certain number of samples are selected as biased data candidate set according to the biased label distribution probability, and the samples are sorted according to the label bias probability. Finally, a certain proportion of data is selected from the biased data candidate set for filtering, thereby constructing a relatively clean data set and reducing the influence of biased labels on model accuracy. Specifically, it is performed according to the following steps:

[0072] Step 1, a sample data set containing biased data is denoted as X = {X1, X2, …, XK}, wherein K represents the number of samples in the sample data set X, Xk represents the kth sample, and k Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and Xk represents the text of the kth sample Xk, and k Xk represents the text of the kth sample Xk, and

[0073] Xk represents the text of the kth sample Xk, and Xk represents the text of the kth sample Xk, and

[0074] Step 2, a multi-modal integrated prediction module is constructed, including a text classifier, a visual classifier and a fusion layer;

[0075] Step 2.1, a text classifier is constructed based on a Bi-LSTM model;

[0076] Step 2.1.1, the text classifier processes the nth word Xn through an embedding layer to obtain the embedding vector of the nth word Xn, thereby obtaining the embedding vector set of Xn, and in order to ensure that the network input size is consistent, 0 vectors are filled for samples with a length less than N, and the filled values are shielded in the later link;

[0077] ​​​​​Step 2.1.2, Although LSTM has been able to deal with text data well, it ignores the influence of the context on the semantic expression of words. Therefore, in order to make full use of the auxiliary role of the context on the semantic understanding of the word, the text classifier uses Bi-LSTM to extract text features. The text classifier inputs the embedding vector set E k into the Bi-LSTM network and obtains the word-level representation H n of the text description

[0078] Forward network in Bi-LSTM network According to the order of to , the embedding vector is read, and the forward hidden state of the nth word is obtained by using formula (1)

[0079]

[0080] In formula (1), represents the forward hidden state of the (n-1)th word;

[0081] Backward network in Bi-LSTM network According to the order of to , the embedding vector is read, and the backward hidden state of the nth word is obtained by using formula (2)

[0082]

[0083] In formula (2), represents the backward hidden state of the (n+1)th word;

[0084] The Bi-LSTM network obtains the representation of the nth word combined with the context by formula (3) so as to obtain the word representation of the text k of the kth sample X

[0085]

[0086] Step 2.1.3, The text classifier uses the method of average pooling to process to obtain the global feature of the text The global feature is then classified by a softmax classification layer ​​​The process is performed to generate the k-th sample X using equation (4) under textual conditions only. k For the c-th biased label Predicted probability

[0087]

[0088] In equation (4), and They represent The c-th label The parameter matrix and bias vector are given below;

[0089] Step 2.2: Construct a visual classifier based on the pre-trained VGG-16 network;

[0090] Step 2.2.1: Since each data point in a multimodal post typically contains multiple images, using only a single image to generate visual features is inaccurate. Taking this widespread fact into consideration, corresponding adjustments were made to the structural design of the visual classifier, specifically as follows... Figure 3 As shown. The visual classifier takes the image set of the k-th sample. The input is processed into the pre-trained VGG-16 model, and the output of the penultimate fully connected layer in the VGG-16 model is used. As a photo collection The feature representation set; where, express The mth image The feature representation is as follows. For samples with fewer than M images, they are also padded with zero vectors and masked in later stages.

[0091] Step 2.2.2: The visual classifier uses average pooling to process the image set. Feature representation set Perform an aggregation operation to obtain the k-th sample X. k Image collection visual features Then use another MLP layer to process the visual features The visual features are then processed using equation (5). Hidden display Finally, another MLP layer is used to hide and show the content. The process is performed to generate the k-th sample X under visual conditions only using equation (6). k For the c-th biased label Predicted probability

[0092]

[0093]

[0094] In formula (5) and formula (6), tanh represents the hyperbolic tangent function, and serves as an activation function of a hidden layer, respectively represent a parameter matrix and a bias vector corresponding to the hidden layer; and respectively represent a parameter matrix and a bias vector corresponding to the cth biased label ;

[0095] Step 2.3, the fusion layer uses a weighted average method as shown in formula (10) to integrate the prediction probability, and obtains the kth sample X k a parameter matrix and a bias vector corresponding to the cth biased label ;

[0096]

[0097] In formula (7), α and β are weight sizes of a text classifier and a visual classifier respectively, and satisfy α, β ∈ [0, 1] and α + β = 1. The application adopts a grid search algorithm, takes averageprecision as an evaluation index, and traverses search for values of α and β in a feasible region through a step size of 0.1, so as to obtain appropriate weight values;

[0098] Step 3, training of the multi-modal integrated prediction module:

[0099] Step 3.1, the confidence learning filtering module mainly performs a series of statistical calculations on the probability obtained by the multi-modal integrated prediction module, and therefore does not have a training problem. Therefore, the training strategy of the model is mainly concentrated in the first stage;

[0100] The cross-entropy loss function J of the multi-modal integrated prediction module is constructed by using formula (8):

[0101]

[0102] In formula (8), S represents part of samples in the multi-modal sample data set X as a training set, |S| represents a sample number of the training set, represents a biased label set of the fth sample X f in the training set;

[0103] Step 3.2, the multi-modal integrated prediction module is trained by using the error back propagation algorithm, and a loss function J is minimized for updating the module parameters until the loss function J converges, so as to obtain an optimal multi-modal integrated prediction model; by minimizing the above loss function, the probability of the real label being recommended can be maximized, so as to optimize the model parameters and achieve the training goal;

[0104] Before continuing the steps, the following assumption is made: for all samples, there is a potential correct label y * ∈ [λ]; there is a class-based biased processing process when labeling y * is mapped to Therefore, each class label j∈[λ] has a certain probability of being labeled as label i∈[λ]. Obviously, this assumption is very reasonable, any label will be mislabeled as any other label with a certain probability, and the probability of different labels being mislabeled as different types of mislabeled labels is different. For example, in handwritten digit recognition, the probability of the number “4” being mislabeled as “9” or “3” is intuitively much higher than the latter. Under such an assumption, the confidence learning method estimates the joint probability distribution of the biased label and the correct label, and performs pruning based on the bias rate.

[0105] Step 4, in the multi-modal integrated prediction module, the prediction probability of each data for the biased label has been obtained. These probabilities can be called confidence, which represents the likelihood of the sample having the corresponding label. By setting a confidence threshold, the correct label of the sample can be estimated. Due to the difference between different label probability distributions and the widespread imbalance problem, it is not ideal to use a consistent confidence threshold for all labels. Therefore, different confidence thresholds are set for each type of label.

[0106] The confidence threshold t of the jth biased label in the biased label set j is calculated by using formula (9):

[0107]

[0108] In formula (9), represents a subset of sample data set X with the jth biased label , represents any one sample x in the sample subset under the parameters θ of the optimal multi-modal integrated prediction model for the jth biased label , || represents the counting operation; the expected confidence of each label is selected as the threshold, effectively considering the difference between different labels, and improving the robustness of the method;

[0109] Step 4, construct the confidence joint counting matrix and joint probability distribution;

[0110] Step 4.1, according to the confidence threshold, estimate the correct label set y * , so as to calculate the biased label set and the correct label set y * using formula (10) and formula (11)

[0111]

[0112]

[0113] In formula (10) and formula (11), represents The value of the i-th row and the j-th column in is the number of samples that have the i-th biased label and the j-th correct label represents the estimated data set that has the i-th biased label and the j-th correct label ; The specific idea is to estimate the correct label of the sample by judging the size of the confidence and the threshold. This is also intuitive, the higher the confidence of a certain label, the more likely it is the correct label of the sample;

[0114] Step 4.2, according to the confidence joint counting matrix estimate the joint probability distribution of the biased label set and the correct label set y * The purpose of modifying the counting matrix is to make the sum of the counts of each row consistent with the total number of observed samples, so that The sum of all elements in is equal to the total number of labels. Because the threshold limiting condition is used when calculating the counting matrix, there may be some sample data whose confidence on all labels is lower than the threshold, resulting in an estimated result that the sample does not have any correct label. In addition, in the case of multi-label, The sum of the counts of each row in is much larger than the total number of samples. Because of these factors, it is necessary to modify the counting matrix;

[0115] Step 4.2.1, use formula (12) to get the value of the i-th row and the j-th column of the modified counting matrix​

[0116]

[0117] Step 4.2.2: Use equation (13) to obtain the set containing partial labels. With the correct tag set y * Joint probability distribution estimation

[0118]

[0119] In equation (13), For joint probability distribution estimation The value in the i-th row and j-th column indicates that the data simultaneously contains the i-th biased label. With the j-th correct label The probability of and satisfying

[0120] Step 5: Filter the biased data based on the bias ratio:

[0121] Step 5.1, from All non-diagonal elements corresponding to Select several samples that minimize equation (14) and form a biased data candidate set:

[0122]

[0123] In equation (14), Represents a subset of samples For any sample x in the dataset, the i-th biased label is determined by parameter θ. The predicted probability;

[0124] Step 5.2: On the one hand, since this invention is based on uncertainty estimation, it cannot guarantee that all data in the candidate set are biased; on the other hand, excessive cleaning may lead to insufficient data and underfitting. Therefore, the data in the candidate set is sorted according to the sample margins.

[0125] Equation (15) is used to calculate the margin of each sample in the biased data candidate set and then used to sort the samples in ascending order to obtain the sorted biased data candidate set:

[0126]

[0127] In equation (15), mean represents the average pooling operation; Let i be the i-th biased label in the sample dataset X; Indicates sample X k A set of partial labels; representing the sample X k the i-th biased-labeled data under the parameter θ the predicted probability of the i-th biased-labeled data under the parameter θ

[0128] Step 5.3, for the sorted biased data candidate set, a certain proportion of data is selected for filtering to obtain a relatively clean data set; in order to select a suitable proportion size, the invention learns through a grid search algorithm. Specifically, first, the data processed according to different removal proportions is input into the multi-modal integrated prediction module to obtain the prediction probability of the sample to the label Then, according to the corresponding evaluation index, for example, the average precision value, the prediction effect is judged; finally, the best removal rate is selected according to the prediction effect.

[0129] In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above biased data detection method, and a processor configured to execute the program stored in the memory.

[0130] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above biased data detection method.

Claims

1. A biased data detection method based on confidence learning, characterized in that, The procedure is as follows: Step 1: Denote the sample dataset containing biased data as X = {X1, X2, ..., X}. k ,…,X K }, where K represents the number of samples in the sample dataset X, X k Let k be the k-th sample, and Let X represent the k-th sample. k The text, and Let X represent the k-th sample. k The nth word in the Chinese text, where N is the number of words in the text; Let X represent the k-th sample. k The image collection, and Let X represent the k-th sample. k The m-th image in the image set; M is the number of images in the image set; Let X represent the k-th sample. k It contains a biased label set, and Let X represent the k-th sample. k The g-th biased label in the biased label set, where G is the number of biased labels in the biased label set; Let the set of unique, biased labels for all samples in the sample dataset X be denoted as... This indicates the c-th biased label, where C is the number of unique biased labels in the sample dataset X. Step 2: Construct a multimodal ensemble prediction module, including: a text classifier, a visual classifier, and a fusion layer; Step 2.1: Construct a text classifier based on the Bi-LSTM model; Step 2.1.1: The text classifier uses an embedding layer to analyze the nth word. Processing yields the nth word. Embedded vector Thus obtain Embedded vector set Step 2.1.2: The text classifier will embed the vector set E. k Inputting the text into a Bi-LSTM network and obtaining the word-level representation H of the text description n : The forward network in the Bi-LSTM network according to arrive The embedding vectors are read sequentially, and the nth word is obtained using equation (1). Forward hidden state In equation (1), Represents the (n-1)th word The forward-hidden state; The backward network in the Bi-LSTM network according to arrive The embedded vectors are read sequentially, and the nth word is obtained using equation (2). Backward hidden state In equation (2), Represents the (n+1)th word The backward hidden state; The Bi-LSTM network then obtains the nth word in conjunction with the context through equation (3). The representation of Thus, the k-th sample X is obtained. k text Word representation Step 2.1.3: The text classifier uses average pooling to process the text. To obtain text global features Then, the global features are processed through a softmax classification layer. The process is performed to generate the k-th sample X using equation (4) under textual conditions only. k For the c-th biased label Predicted probability In equation (4), and They represent The c-th label The parameter matrix and bias vector are given below; Step 2.2: Construct a visual classifier based on the pre-trained VGG-16 network; Step 2.2.1: The visual classifier will use the image set of the k-th sample. The input is processed into the pre-trained VGG-16 model, and the output of the penultimate fully connected layer in the VGG-16 model is used. As a photo collection The feature representation set; where, express The mth image Feature representation; Step 2.2.2: The visual classifier uses average pooling to process the image set. Feature representation set Perform an aggregation operation to obtain the k-th sample X. k Image collection visual features Then use another MLP layer to process the visual features The data is processed, and the visual features are obtained using equation (5). Hidden display Finally, another MLP layer is used to hide and reveal the content. The data is processed, and Equation (6) is used to generate the k-th sample X under visual conditions only. k For the c-th biased label Predicted probability In equations (5) and (6), tanh represents the hyperbolic tangent function and is used as the activation function of the hidden layer. These represent the parameter matrix and bias vector corresponding to the hidden layer, respectively. and They represent The c-th biased label The parameter matrix and bias vector are given below; Step 2.3: The fusion layer integrates the predicted probabilities using the weighted average method shown in Equation (10) to obtain the k-th sample X. k For the c-th biased label Final ensemble prediction probability In equation (7), α and β are the weights of the text classifier and the visual classifier, respectively, satisfying that α, β ∈ [0, 1] and α + β = 1; Step 3: Training the multimodal ensemble prediction module: Step 3.1: Construct the cross-entropy loss function J of the multimodal integrated prediction module using equation (8): In equation (8), S represents a subset of samples in the multimodal sample dataset X used as the training set, and |S| represents the number of samples in the training set. Let X represent the f-th sample in the training set. f A set of partial labels; Step 3.2: Train the multimodal ensemble prediction module using the backpropagation algorithm and minimize the loss function J to update the module parameters until the loss function J converges, thereby obtaining the optimal multimodal ensemble prediction model. Step 4: Calculate the biased label set using equation (9). The j-th biased label Confidence threshold t j : In equation (9), This indicates that the j-th biased label is present in the sample dataset X. a subset of samples Represents a subset of samples For any sample x in the optimal multimodal ensemble prediction model, the j-th biased label is determined by the parameters θ. The predicted probability, || represents the counting operation; Step 4: Construct the joint confidence count matrix and joint probability distribution; Step 4.1: Estimate the correct labels in the sample dataset X based on the confidence threshold to obtain the correct label set y. * Thus, equations (10) and (11) are used to calculate the set containing partial labels. With the correct tag set y * Confidence joint counting matrix In equations (10) and (11), express The value in the i-th row and j-th column is the value that simultaneously has the i-th biased label. With the j-th correct label Number of samples This indicates that it simultaneously has the i-th biased label. With the j-th correct label The estimated dataset; Step 4.2: Based on the confidence joint counting matrix Estimate the set with biased labels With the correct tag set y * joint probability distribution Step 4.2.1: Use equation (12) to obtain the value of the i-th row and j-th column of the corrected counting matrix. Step 4.2.2: Use equation (13) to obtain the set containing partial labels. With the correct tag set y * Joint probability distribution estimation In equation (13), For joint probability distribution estimation The value in the i-th row and j-th column indicates that the data simultaneously contains the i-th biased label. With the j-th correct label The probability of and satisfying Step 5: Filter the biased data based on the bias ratio: Step 5.1, from All non-diagonal elements corresponding to Select several samples that minimize equation (14) and form a biased data candidate set: In equation (14), Represents a subset of samples For any sample x in the dataset, the i-th biased label is determined by parameter θ. The predicted probability; Step 5.2: Calculate the margin of each sample in the biased data candidate set using equation (15) and use it to sort the samples in ascending order to obtain the sorted biased data candidate set: In equation (15), mean represents the average pooling operation; Let i be the i-th biased label in the sample dataset X; Let X represent the k-th sample. k A set of partial labels; Let X represent the k-th sample. k For the i-th biased label under parameter θ The predicted probability; Step 5.3: Select a certain percentage of the top-ranked biased candidate data from the sorted dataset and filter it to obtain a relatively clean dataset.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the biased data detection method of claim 1, wherein the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the biased data detection method of claim 1.

Images