A polysemous data annotation method based on active query and group annotation

Abstract

The present invention discloses a method for labeling polysemous data based on active query and group labeling, which specifically includes the following steps: (1) the user initially selects a small portion of polysemous data for precise labeling; (2) a multi-label classification network is trained on a small portion of the labeled data using a deep neural network; (3) the label vectors of the unlabeled data are predicted using the multi-label classification network; (4) the label space of the queried examples is grouped, and a small portion of the groups is sent to domain experts for precise labeling; (5) the samples are added to the labeled set, the multi-label classification network parameters are updated, and the effect is tested on the test set; (6) if the user is satisfied with the current model performance, the method ends; otherwise, the method goes to step (3). The present invention can achieve a certain balance between the quality and cost of polysemous data labeling, and improve the labeling quality while reducing the labeling cost.

Claims

1. A method for labeling ambiguous data based on active query and group labeling, characterized by: The following steps are involved: Step 1: Obtain a certain amount of polysemous datasets represents the d-dimensional feature space, Represents the label space of C categories; initially, a small number of polysemous examples are selected for precise labeling, which is called the labeled set where Y i ∈{0,1} C is x i The label vector, y ij =1 means the jth mark is x i The positive label (correlation label), y ij =0 means the jth mark is x i The remaining u=nl unlabeled examples are called the unlabeled set. Step 2: Use deep neural network to analyze the labeled data Train the multi-label classification network N and establish the input space To output space Mapping; Step 3: Use the classification network N to predict the unlabeled set All examples in x u The probability vector and marker vector Using the uncertainty criterion Actively query an unlabeled example x that is most valuable to the performance of the current classification network N q ; The step 3 is specifically: Step 3.1: Unlabeled set All examples in are input into the classification network N for prediction. For each example x u For example, the classification network N can output the probability that each tag in its tag vector is a positive tag, forming a probability vector of length C p uj ∈(0,1); Step 3.2: Divide the label space into a positive label set and a negative label set by setting a threshold thr (usually 0.5). Labels where p uj ≥ thr are considered positive labels, and labels where p uj < thr are considered negative labels, thus obtaining the label vector predicted by N for each unlabeled example The formal expression is as follows; where returns 1 if π is true and 0 otherwise; Step 3.3: For the unlabeled set For all examples in The minimum probability value and The difference Δ between the maximum probability values u , query Δ u The unlabeled example with the smallest value is used as the query example x q , this example is considered to be the most "uncertain" example of the current classification network N, and its predicted probability vector is Labeling and training this example is most likely to improve the performance of the classification network; Step 4: For the query example x q The label vector is The small group is sent to domain experts for precise labeling, and the majority of the group is sent to multiple crowdsourced annotators for imprecise crowdsourced labeling; the returned label vector is the same as the example x q Together they form a sample (x q ,Y q ); Step 5: Convert (x q ,Y q ) Add to the labeled set Update the parameters of the classification network N and test the performance on the test set; Step 6: If the user is satisfied with the current model performance, then end; otherwise, go to step (3).

2. The method for labeling ambiguous data based on active query and group labeling according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.1: Let the classification network be the query example x q The predicted probability vector is The C marks on the mark space are divided into Sort the values ​​in descending order to get the set of tags to be distributed Among them, the ones ranked higher are the ones with a higher probability of being predicted as positive by the classification network, which are called high-probability tags; conversely, the ones ranked lower are the ones with a lower probability of being predicted as positive by the classification network, which are called low-probability tags; Step 4.2: The first h tags are sent to domain experts for accurate labeling. This part of the tags is more important for model performance, and it is necessary to obtain as accurate labeling information as possible; Step 4.3: The remaining Ch markers are divided into n "marker groups" of even length l, and the marker groups containing high-probability markers are paired with the marker groups containing low-probability markers to form a "marker group pair", that is, the high-probability marker group group a and low probability marker group b , where a=i, b=n-i+1, i=1:1:n / 2; Step 4.4: Send the "labeled group pairs" to the crowdsourced annotator for labeling and limit the cost or time upper limit τ; the annotator repeatedly scans the "labeled group pairs" and tries to find the positive label; if no positive label is found under the cost upper limit τ, the group is judged to be a and group are all negative signs; once in group a If a positive mark is found in the group a Send to domain experts for precise labeling and direct group determination b The whole is negative; once in group b If a positive mark is found in the group b Send it to domain experts for accurate labeling, and directly determine that all the labels in the group are negative; Step 4.5: Collect the annotation results of each label group and merge them to get x q The marker vector Y q .

Images