Sample selection method, apparatus, device, and medium
By using pre-trained models and multi-dimensional evaluation, high-quality data augmentation samples are selected, which solves the problem of low sample quality in natural language processing and improves the generalization ability and performance of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG UNIVERSITY OF FOREIGN STUDIES
- Filing Date
- 2022-12-14
- Publication Date
- 2026-05-12
AI Technical Summary
In the field of natural language processing, data augmentation strategies generate samples of low quality, resulting in insufficient model generalization ability. Existing evaluation methods are limited and cannot effectively screen high-quality samples.
By predicting the class probability distribution of augmented samples through pre-trained models and obtaining pseudo-labels, and combining Monte Carlo sampling with lexical similarity and semantic fluency evaluation, high-confidence and high-quality samples are selected to supplement low-confidence samples and increase data diversity.
The model's generalization ability and performance were improved. By selecting high-quality samples and recalling low-confidence samples, the effectiveness of the data augmentation strategy was enhanced, thereby improving the model's generalization ability.
Smart Images
Figure CN115840884B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a sample selection method, apparatus, device, and medium. Background Technology
[0002] Text classification is fundamental to many Natural Language Processing (NLP) tasks and is widely used in fields such as sentiment analysis and intelligent question answering. Generally, training a classifier with strong generalization ability requires a large amount of labeled data, but the high cost of manual annotation and the significant time and effort required to build a large corpus are often prohibitive. To address this issue, data augmentation strategies have been proposed. Data augmentation can significantly increase the amount of data, alleviating data insufficiency and improving the model's generalization ability. However, in the field of NLP, data augmentation faces significant challenges. Besides the discontinuity of text data, a major reason is the inherent weakness of language itself in resisting interference. Arbitrarily modifying language data can easily destroy its semantics, generating low-quality samples that greatly affect the classifier's judgment, thus creating a negative feedback effect on the model.
[0003] Therefore, selecting high-quality samples from data augmentation samples is particularly important. Summary of the Invention
[0004] The present invention provides a sample selection method, apparatus, device, and medium in several aspects, which can effectively screen out high-quality samples generated by data augmentation strategies.
[0005] The first aspect of this invention provides a sample selection method, comprising:
[0006] Obtain augmented samples, predict the class probability distribution of the augmented samples based on the pre-trained model, and obtain the pseudo-label of the augmented samples according to the class probability distribution of the augmented samples; wherein, the augmented samples are unlabeled samples generated after augmentation based on the original labeled sample data;
[0007] Based on the comparison results between the labels of the original labeled samples corresponding to the augmented samples and the pseudo labels of the augmented samples, the augmented samples are classified into clean samples and noisy samples.
[0008] The pre-trained model, trained using Monte Carlo sampling, predicts the class probability distribution of the clean samples under different model parameters. Based on the class probability distribution under different model parameters, the confidence level of the clean samples is obtained, and the clean samples are classified into high-confidence samples and low-confidence samples according to the confidence level.
[0009] The recall samples are confirmed based on the comparison results of the lexical similarity of the samples to be recalled with the set lexical similarity threshold and the semantic fluency of the samples to be recalled with the set semantic fluency threshold; wherein, the samples to be recalled include low confidence samples and noise samples.
[0010] The high-confidence samples and the recalled samples are used as the final selected samples.
[0011] A second aspect of the present invention provides a sample selection device, comprising:
[0012] A pseudo-label acquisition module is used to acquire augmented samples, predict the class probability distribution of the augmented samples based on the pre-trained model, and obtain the pseudo-label of the augmented samples according to the class probability distribution of the augmented samples; wherein, the augmented samples are unlabeled samples generated after augmentation based on the original labeled sample data;
[0013] The first classification module is used to classify the enhanced sample into clean samples and noisy samples based on the comparison results between the label of the original labeled sample corresponding to the enhanced sample and the pseudo label of the enhanced sample.
[0014] The second classification module is used to introduce the class probability distribution of the clean sample predicted by the pre-trained model after Monte Carlo sampling training under different model parameters, so as to obtain the confidence of the clean sample according to the class probability distribution under different model parameters, and classify the clean sample into high confidence sample and low confidence sample according to the confidence level.
[0015] The recall module is used to confirm the recall samples based on the comparison results of the lexical similarity of the samples to be recalled with a set lexical similarity threshold and the semantic fluency of the samples to be recalled with a set semantic fluency threshold; wherein, the samples to be recalled include low-confidence samples and noise samples.
[0016] A selection module is used to select the high-confidence samples and the recalled samples as the final selected samples. A third aspect of this invention provides a terminal device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the sample selection method provided in the above embodiments.
[0017] A fourth aspect of the present invention provides a storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to perform the sample selection method provided in the above embodiments.
[0018] Compared with existing technologies, the sample selection method provided in this invention classifies clean and noisy samples generated by data augmentation strategies. It then selects high-confidence samples from the clean samples as high-quality samples, and further selects from low-confidence samples (noisy samples and clean samples with low confidence) to supplement high-quality low-confidence samples from the high-confidence clean samples, thus completing the selection of high-quality samples from the augmented samples. This embodiment not only effectively selects high-quality samples generated by data augmentation strategies but also increases the diversity of data augmented samples by recalling low-confidence samples, enabling the model to learn more patterns and thus improving model performance and generalization. Accordingly, this invention also provides a sample selection device, apparatus, and medium. Attached Figure Description
[0019] Figure 1 This is a structural diagram of the sample selection framework provided in an embodiment of the present invention;
[0020] Figure 2 This is a flowchart illustrating the sample selection method provided in an embodiment of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Data augmentation strategies are very popular in research areas that address data scarcity. They refer to the process of transforming training data to generate new data, allowing limited data to produce value equivalent to more data, thereby improving the model's generalization ability and robustness. Data augmentation strategies are also widely used in natural language processing (NLP). However, data augmentation faces significant challenges in NLP. Besides the discontinuity of text data, a major reason is the inherent weakness of language itself in resisting interference. Arbitrarily modifying language data can easily destroy its semantics, generating low-quality samples that significantly affect the classifier's judgment, thus creating negative feedback for the model.
[0023] While data augmentation can alleviate the problem of insufficient data for model training to some extent, low-quality augmented samples may have a negative feedback effect on the model. Therefore, filtering noise from augmented samples, i.e., sample selection, is crucial. Currently, there are relevant studies on evaluating sample quality, such as data evaluation based on classifier discrimination and data evaluation based on text similarity.
[0024] Data evaluation methods based on classifier discrimination refer to using labeled data to train a text classification model, and then using the text classifier to classify or predict unlabeled data. However, this method, which uses only the classifier to evaluate the generated data, will cause the distribution of the selected data to fit the classifier, that is, to maintain the original data distribution. This results in low diversity of the selected samples and fails to improve the performance of the classifier.
[0025] Text similarity-based selection methods are technically straightforward, typically determining text similarity by calculating text distance. In natural language processing, measuring the similarity between two texts is a frequent challenge. In problems such as dialogue systems and information retrieval, measuring the similarity between sentences or phrases is particularly important. One approach is to filter model-generated samples based on the text coverage between the original and generated samples. However, this strategy only considers the similarity between generated and original samples at the lexical level, lacking semantic information.
[0026] The above sample evaluation method only evaluates from a single dimension and does not screen data-enhanced samples from multiple dimensions, so the overall quality of the selected samples is not high.
[0027] To address the negative gain problem caused by harmful samples in data augmentation strategies, this application proposes a sample selection method, apparatus, device, and medium. It classifies clean and noisy samples generated by data augmentation strategies, selects high-confidence samples from the clean samples as high-quality samples, and further selects from low-confidence samples (noisy samples and clean samples with low confidence) to supplement high-quality low-confidence samples from the high-confidence clean samples. This completes the selection of high-quality samples from the augmented samples, increases the diversity of data augmentation samples, allows the model to learn more patterns, improves model performance, and further enhances the model's generalization ability.
[0028] For example, see Figure 1 , Figure 1 A structural diagram of the sample selection framework used in the sample selection method, apparatus, device and medium of the embodiments of this application is provided.
[0029] like Figure 1 As shown in the diagram, the framework consists of four modules: (1) a pre-training module; (2) a semi-supervised training module; (3) a sample selection module; and (4) a sample recall module. In the semi-supervised training module, the original labeled samples D are used... l (X,Y) Train the pre-trained model (the figure shows an example of selecting RoBERTa for the pre-training module); in the sample selection module, use the trained pre-trained model to predict the augmented sample D.g The class probability distribution of (X,Y) is used to obtain the augmented sample D. g The pseudo-labels of (X,Y) will be used to enhance sample D. g By comparing the pseudo-labels of (X,Y) with their true labels, clean samples D are classified. clean and noise sample D noisy And calculate the clean sample D using MC dropout. clean The uncertainty predicted in the pre-trained model is used to select samples with model uncertainty for self-training using an entropy-based strategy, resulting in high-confidence samples D. easy and low confidence sample D hard Finally, in the sample recall module, samples with low confidence (D) are automatically retrieved based on two dimensions: lexical similarity and semantic fluency. hard and D noisy We recalled some high-quality, confident samples from the sample.
[0030] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0031] See Figure 2 , Figure 2 This is a flowchart illustrating the sample selection method provided in this embodiment of the invention. The sample selection method in this embodiment includes steps S11 to S15:
[0032] S11. Obtain augmented samples, predict the class probability distribution of the augmented samples based on the pre-trained model, and obtain the pseudo-label of the augmented samples according to the class probability distribution of the augmented samples; wherein, the augmented samples are unlabeled samples generated after augmentation based on the original labeled sample data.
[0033] For example, an original labeled sample can generate one or more augmented samples through a data augmentation strategy. Suppose the original labeled sample is {I, have, questions, want, ask, you}, then the augmented samples generated by data augmentation are {I, have, want, ask, you, questions}, {I, ask, you, of, have, want, questions}, etc.
[0034] S12. Based on the comparison results between the labels of the original labeled samples corresponding to the augmented samples and the pseudo labels of the augmented samples, the augmented samples are classified into clean samples and noisy samples.
[0035] For example, augmented samples whose corresponding original labeled samples have the same label and pseudo-label are considered clean samples, while augmented samples whose corresponding original labeled samples have different labels and pseudo-labels are considered noise samples. The clean samples are then placed into the clean sample set D. clean In the middle, the noise samples are placed into the noise sample set D. noisy middle.
[0036] S13. The pre-trained model after Monte Carlo sampling training predicts the class probability distribution of the clean sample under different model parameters, so as to obtain the confidence of the clean sample according to the class probability distribution under different model parameters, and classify the clean sample into high confidence sample and low confidence sample according to the confidence level.
[0037] Specifically, this embodiment of the invention enhances the credibility of the generated samples by quantifying data, thereby solving the problem of distinguishing between high-confidence and low-confidence samples in sample selection.
[0038] S14. Based on the comparison results of the lexical similarity of the samples to be recalled with the set lexical similarity threshold and the semantic fluency of the samples to be recalled with the set semantic fluency threshold, the samples to be recalled are confirmed; wherein, the samples to be recalled include low confidence samples and noise samples.
[0039] For example, because the data generated by data augmentation contains noisy data, the model will be affected by the noisy data and gradually deviate during the training process of the pre-trained model, thus affecting the performance of the pre-trained model. Low confidence samples also have great uncertainty in the prediction results of the model. Therefore, low confidence samples and noisy samples are classified as low confidence samples.
[0040] However, even among low-confidence samples, there may be high-quality samples. Although these samples have lower prediction confidence during the training of the pre-trained model, they actually improve the training and performance of the pre-trained model. Therefore, in this embodiment of the invention, multiple dimensions are used to recall low-confidence samples, and lexical similarity and semantic fluency are used to evaluate whether low-confidence samples are high-quality samples.
[0041] S15. The high-confidence samples and the recalled samples are selected as the final samples.
[0042] The technical solution provided in this invention obtains augmented samples, predicts the class probability distribution of the augmented samples based on a pre-trained model, and obtains pseudo-labels for the augmented samples based on the class probability distribution. Based on the comparison between the labels of the original labeled samples corresponding to the augmented samples and the pseudo-labels of the augmented samples, the augmented samples are classified into clean samples and noisy samples. The pre-trained model trained using Monte Carlo sampling predicts the class probability distribution of the clean samples under different model parameters, thereby obtaining the confidence level of the clean samples based on the class probability distribution under different model parameters. The clean samples are then classified into high-confidence samples and low-confidence samples based on the confidence level. Based on the comparison between the lexical similarity of the samples to be recalled and a set lexical similarity threshold, and the semantic fluency of the samples to be recalled and a set semantic fluency threshold, the recalled samples are confirmed. The samples to be recalled include low-confidence samples and noisy samples. Finally, the high-confidence samples and the recalled samples are selected as the final samples. The implementation not only effectively filters out high-quality samples generated by data augmentation strategies, but also increases the diversity of data augmentation samples by recalling low-confidence samples, enabling the model to learn more patterns, thereby improving the model's performance and generalization ability.
[0043] In an optional implementation, before predicting the class probability distribution of the augmented samples based on the trained pre-trained model, the method further includes:
[0044] Obtain the original labeled samples, and train the pre-trained model using a semi-supervised method based on the original labeled samples to obtain the trained pre-trained model.
[0045] In this embodiment, a semi-supervised method is used to train a pre-trained model to predict and filter data-augmented samples. The semi-supervised method augments the labeled dataset using unlabeled data. In practice, the pre-trained model is trained using the original labeled samples, and then used to predict unlabeled augmented samples. The augmented sample with the highest prediction confidence is selected and given a pseudo-label. This pseudo-labeled augmented sample is then incorporated into the current training samples for further training until the pre-trained model converges.
[0046] For example, the semi-supervised method employs self-training. In implementation, a pre-trained model is trained using a self-training method to predict and filter data-augmented samples. Assume D... l ={x l ,y l} represents the original labeled sample, y l Represents the original labeled sample x l The C class tags, xl This represents a sequence consisting of n tokens. (D) g ={x g ,y g} represents the augmented sample generated by data augmentation methods. The specific framework implementation is as follows:
[0047] S1, using the original labeled sample D l The pre-trained model is trained as the teacher network, and cross-entropy is used as its loss function L.
[0048] S2. Utilize this teacher network to predict and enhance sample D. g The probability of belonging to a certain class label.
[0049] S3. Based on the class label probabilities predicted by the teacher network, multiple screening strategies are used to filter the augmented samples, resulting in a subset S of the augmented samples. g .
[0050] S4. Select the filtered samples S g Compared with the original labeled sample D l Together they are used to train student networks.
[0051] S5. Train the teacher network and student network alternately, treating the current student network as the teacher network and returning it to step S2, until the pre-trained model converges.
[0052] In one alternative implementation, the pre-trained model is a pre-trained language model using the BERT architecture.
[0053] For example, the pre-trained model uses RoBERTa. RoBERTa is an improved version of BERT, which, by improving the training task and data generation method, can use a larger vocabulary and be trained on large datasets with large batches of longer sequences. The BERT model uses two tasks for pre-training: Next Sentence Prediction (NSP) and Masked Language Modeling (MLM). In RoBERTa, the NSP task is removed, and only the MLM task is retained for pre-training.
[0054] For classification problems, the first position in the input sentence is marked as the [CLS] start symbol, and the final hidden state h corresponding to its token is... i It is typically used as an aggregate sequence representation for classification tasks. For each token in a given sentence, its input representation is constructed by summing the corresponding token, segment, and position embeddings.
[0055] Based on this, through experimental comparison, the embodiments of the present invention use RoBERTa as a pre-trained model. This pre-trained model acts as an encoder for language features. By obtaining the first token ([CLS]) of the final hidden state, it represents the sentence encoding S of the sample, thus obtaining the sentence vector representation of the sample. For each sample, the sentence vector representation S output by the pre-trained model RoBERTa is... i for:
[0056] S i =RoBERTa(a i ,b i ,c i )
[0057] Among them, a i ,b i ,c i It involves embedding tokens, segments, and positions.
[0058] In an optional implementation, step S13, "introducing the class probability distribution of the clean sample predicted by the pre-trained model after Monte Carlo sampling training under different model parameters, so as to obtain the confidence level of the clean sample based on the class probability distribution under different model parameters," specifically includes:
[0059] The pre-trained model, trained using Monte Carlo sampling, predicts the class probability distribution of the clean sample under different model parameters. Based on the class probability distribution under different model parameters, the confidence level of the clean sample is calculated using the information entropy formula.
[0060] In this embodiment, Monte Carlo dropout is used to extract different model parameters, obtaining different output results for the same clean sample in the same pre-trained model. Specifically, in the t-th dropout, the pre-trained model predicts the probability that the clean sample belongs to class c. Represented as:
[0061]
[0062] in, Let represent the parameter matrix at the t-th dropout, x be the sentence vector representation of the clean sample, T be the total number of dropouts, and B be the bias value.
[0063] Based on the class probability distribution of each dropout, information entropy is used to quantify the confidence of clean samples, thus obtaining the confidence level H(p) of clean samples. i This refers to the ease with which the pre-trained model can distinguish the clean sample.
[0064]
[0065]
[0066]
[0067] Where C represents the number of class tags, and represents the ease of identification and the difficulty of identification of the i-th clean sample, respectively.
[0068] After determining the confidence levels of all clean samples, clean samples with confidence levels greater than a preset confidence threshold are classified into the low-confidence sample set D. hrad Clean samples with a confidence level less than or equal to the confidence threshold are classified as high-confidence samples D. easy The confidence threshold can be determined experimentally. For example, if the experiment selects samples with a confidence level of 0.1 to 0.3, and the experimental results show that a confidence level of 0.25 is optimal, then the confidence threshold is set to 0.25.
[0069] In one optional implementation, the step of predicting the class probability distribution of the augmented sample based on the trained pre-trained model, and obtaining the pseudo-label of the augmented sample according to the class probability distribution of the augmented sample, specifically includes:
[0070] Predict the class probability distribution of the augmented samples based on the trained pre-trained model. And obtain pseudo-labels Among them, class probability distribution and pseudo-tags Calculated using the following formula:
[0071]
[0072]
[0073] in, To enhance the samples, θ is the parameter of the pre-trained model, softmax() is the softmax() function in the pre-trained model, W is the parameter matrix, b is the bias value, and argmaxf() is the argmaxf() function in the pre-trained model.
[0074] As can be seen, in this optional implementation, a binary classification task is introduced by predicting the pre-trained model after training to obtain pseudo-labels for augmented samples, and the augmented samples with incorrect predictions are filtered out by comparing the pseudo-labels with the labels of the corresponding original labeled samples.
[0075] In one optional implementation, the word similarity threshold is obtained through the following steps:
[0076] Calculate the word similarity between each high-confidence sample and its corresponding original labeled sample, and obtain the word similarity threshold based on the word similarity between all high-confidence samples and their corresponding original labeled samples;
[0077] The word similarity is calculated using the following formula:
[0078]
[0079] Where J(x) represents word similarity, x g x l These represent the augmented sample and the original labeled sample corresponding to the augmented sample, respectively.
[0080] Specifically, calculate the average lexical similarity J between all high-confidence samples and their corresponding original labeled samples. avg J avg This serves as the threshold for word similarity.
[0081] As can be seen, in this optional implementation, the word similarity threshold is calculated by calculating the word similarity between all high-confidence samples and their corresponding original labeled samples. This word similarity threshold is then used as one of the indicators for evaluating the quality of low-confidence samples (including noise samples and low-confidence samples, as mentioned above). Furthermore, calculating the word similarity threshold using high-confidence samples eliminates the need for model normalization and probabilistic modeling, thus reducing the workload of standardization and probabilistic modeling.
[0082] In one optional implementation, the semantic fluency threshold is obtained as follows:
[0083] The semantic fluency of the high-confidence sample is obtained by calculating the difference between the perplexity of the original labeled sample corresponding to the high-confidence sample and the perplexity of the high-confidence sample.
[0084] The semantic fluency threshold is obtained based on the semantic fluency of all the high-confidence samples.
[0085] Specifically, semantic fluency can be obtained through Statistical Language Models (SLM), a crucial metric in the field of syntax error correction. Therefore, this embodiment proposes a metric to evaluate the semantic fluency of augmented samples. Specifically, assuming an original labeled sample... There are n words in total, and one of its augmented samples is There are m words in total. First, the perplexity of their high-confidence samples is calculated using a ResLSTM language model trained on a One-Billion corpus. Perplexity of the original labeled samples corresponding to the high-confidence samples
[0086]
[0087]
[0088] in, Let represent the j-th word of the i-th sentence in the original labeled sample. This represents the j-th word of the i-th sentence in the augmented sample.
[0089] Perplexity of the high-confidence samples Perplexity of the original labeled samples corresponding to the high-confidence samples Perform the difference operation to obtain the semantic fluency F of the high-confidence sample. ik (x):
[0090]
[0091] The semantic fluency F for all high-confidence samples ik (x) Find the average value F avg , will F avg This serves as the semantic fluency threshold.
[0092] As can be seen, in this optional implementation, the semantic fluency threshold is calculated by evaluating the semantic fluency of all high-confidence samples, thus using the semantic fluency threshold as another indicator to evaluate the quality of low-confidence samples (including noise samples and low-confidence samples, as mentioned above). Furthermore, calculating the semantic fluency threshold using high-confidence samples eliminates the need for model normalization and probabilistic modeling, reducing the workload of standardization and probabilistic modeling.
[0093] In an optional implementation, S14, "confirming the recall sample based on the comparison results of the lexical similarity of the sample to be recalled with a set lexical similarity threshold and the semantic fluency of the sample to be recalled with a set semantic fluency threshold," specifically includes:
[0094] Samples to be recalled that have a word similarity greater than the word similarity threshold and a semantic fluency greater than the semantic fluency threshold are selected as the recall samples.
[0095] In this embodiment, samples to be recalled with both word similarity greater than the word similarity threshold and semantic fluency greater than the semantic fluency threshold are selected as high-quality samples for recall. This is to supplement the high-quality samples (high-confidence samples) that have been initially screened with low-confidence high-quality samples, thereby enhancing the diversity of data augmentation samples.
[0096] Accordingly, embodiments of the present invention also provide a sample selection device, comprising:
[0097] A pseudo-label acquisition module is used to acquire augmented samples, predict the class probability distribution of the augmented samples based on the pre-trained model, and obtain the pseudo-label of the augmented samples according to the class probability distribution of the augmented samples; wherein, the augmented samples are unlabeled samples generated after augmentation based on the original labeled sample data;
[0098] The first classification module is used to classify the enhanced sample into clean samples and noisy samples based on the comparison results of the labels and pseudo-labels of the original labeled samples corresponding to the enhanced sample.
[0099] The second classification module is used to introduce the class probability distribution of the clean sample predicted by the pre-trained model after Monte Carlo sampling training under different model parameters, so as to obtain the confidence of the clean sample according to the class probability distribution under different model parameters, and classify the clean sample into high confidence sample and low confidence sample according to the confidence level.
[0100] The recall module is used to confirm the recall samples based on the comparison results of the lexical similarity of the samples to be recalled with a set lexical similarity threshold and the semantic fluency of the samples to be recalled with a set semantic fluency threshold; wherein, the samples to be recalled include low-confidence samples and noise samples.
[0101] The selection module is used to select the high-confidence samples and the recalled samples as the final selected samples.
[0102] In an optional implementation, before predicting the class probability distribution of the augmented samples based on the trained pre-trained model, the method further includes:
[0103] Obtain the original labeled samples, and train the pre-trained model using a semi-supervised method based on the original labeled samples to obtain the trained pre-trained model.
[0104] In one optional implementation, the word similarity threshold is obtained through the following steps:
[0105] Calculate the word similarity between each high-confidence sample and its corresponding original labeled sample, and obtain the word similarity threshold based on the word similarity between all high-confidence samples and their corresponding original labeled samples;
[0106] The word similarity is calculated using the following formula:
[0107]
[0108] Where J(x) represents word similarity, x g x l These represent the augmented sample and the original labeled sample corresponding to the augmented sample, respectively.
[0109] In one optional implementation, the semantic fluency threshold is obtained as follows:
[0110] The semantic fluency of the high-confidence sample is obtained by calculating the difference between the perplexity of the original labeled sample corresponding to the high-confidence sample and the perplexity of the high-confidence sample.
[0111] The semantic fluency threshold is obtained based on the semantic fluency of all the high-confidence samples.
[0112] In one optional implementation, the pre-trained model, trained using Monte Carlo sampling, predicts the class probability distribution of the clean samples under different model parameters, and the confidence level of the clean samples is obtained based on the class probability distribution under different model parameters. Specifically, this includes:
[0113] The pre-trained model, trained using Monte Carlo sampling, predicts the class probability distribution of the clean sample under different model parameters. Based on the class probability distribution under different model parameters, the confidence level of the clean sample is calculated using the information entropy formula.
[0114] In one optional implementation, the step of confirming the recall sample based on the comparison results of the lexical similarity of the sample to be recalled with a set lexical similarity threshold and the semantic fluency of the sample to be recalled with a set semantic fluency threshold specifically involves:
[0115] Samples to be recalled that have a word similarity greater than the word similarity threshold and a semantic fluency greater than the semantic fluency threshold are selected as the recall samples.
[0116] In one alternative implementation, the pre-trained model is a pre-trained language model using the BERT architecture.
[0117] It should be noted that the sample selection device provided in the embodiments of the present invention is used to execute all the steps and processes of the sample selection method provided in the above embodiments. The working principles and beneficial effects of the two are one-to-one, and will not be elaborated further here.
[0118] Accordingly, embodiments of the present invention also provide a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the sample selection method provided in the above embodiments, for example... Figure 2 S11 to S15.
[0119] Accordingly, embodiments of the present invention also provide a storage medium, the storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to execute the sample selection method provided in the above embodiments, for example... Figure 2 S11 to S15.
[0120] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A sample selection method, characterized in that, include: Obtain augmented samples, predict the class probability distribution of the augmented samples based on the pre-trained model, and obtain the pseudo-label of the augmented samples according to the class probability distribution of the augmented samples; wherein, the augmented samples are unlabeled samples generated after augmentation based on the original labeled sample data; Based on the comparison results between the labels of the original labeled samples corresponding to the augmented samples and the pseudo labels of the augmented samples, the augmented samples are classified into clean samples and noisy samples. The pre-trained model, trained using Monte Carlo sampling, predicts the class probability distribution of the clean samples under different model parameters. Based on the class probability distribution under different model parameters, the confidence level of the clean samples is obtained, and the clean samples are classified into high-confidence samples and low-confidence samples according to the confidence level. The recall samples are confirmed based on the comparison results of the lexical similarity of the samples to be recalled with the set lexical similarity threshold and the semantic fluency of the samples to be recalled with the set semantic fluency threshold; wherein, the samples to be recalled include low confidence samples and noise samples. The high-confidence samples and the recalled samples are used as the final selected samples.
2. The sample selection method as described in claim 1, characterized in that, Before predicting the class probability distribution of the augmented samples based on the pre-trained model, the method further includes: Obtain the original labeled samples, and train the pre-trained model using a semi-supervised method based on the original labeled samples to obtain the trained pre-trained model.
3. The sample selection method as described in claim 1, characterized in that, The word similarity threshold is obtained through the following steps: Calculate the word similarity between each high-confidence sample and its corresponding original labeled sample, and obtain the word similarity threshold based on the word similarity between all high-confidence samples and their corresponding original labeled samples; The word similarity is calculated using the following formula: Where J(x) represents word similarity, x g x l These represent the augmented sample and the original labeled sample corresponding to the augmented sample, respectively.
4. The sample selection method as described in claim 1, characterized in that, The semantic fluency threshold is obtained in the following way: The semantic fluency of the high-confidence sample is obtained by calculating the difference between the perplexity of the original labeled sample corresponding to the high-confidence sample and the perplexity of the high-confidence sample. The semantic fluency threshold is obtained based on the semantic fluency of all the high-confidence samples.
5. The sample selection method as described in claim 1, characterized in that, The pre-trained model, trained using Monte Carlo sampling, predicts the class probability distribution of the clean samples under different model parameters, and the confidence level of the clean samples is obtained based on the class probability distribution under different model parameters. Specifically, this includes: The pre-trained model, trained using Monte Carlo sampling, predicts the class probability distribution of the clean sample under different model parameters. Based on the class probability distribution under different model parameters, the confidence level of the clean sample is calculated using the information entropy formula.
6. The sample selection method as described in claim 1, characterized in that, The process of confirming the recall samples based on the comparison results of the lexical similarity of the samples to be recalled with the set lexical similarity threshold and the semantic fluency of the samples to be recalled with the set semantic fluency threshold is as follows: Samples to be recalled that have a word similarity greater than the word similarity threshold and a semantic fluency greater than the semantic fluency threshold are selected as the recall samples.
7. The sample selection method as described in claim 1, characterized in that, The pre-trained model is a pre-trained language model using the BERT structure.
8. A sample selection device, characterized in that, include: A pseudo-label acquisition module is used to acquire augmented samples, predict the class probability distribution of the augmented samples based on the pre-trained model, and obtain the pseudo-label of the augmented samples according to the class probability distribution of the augmented samples; wherein, the augmented samples are unlabeled samples generated after augmentation based on the original labeled sample data; The first classification module is used to classify the enhanced sample into clean samples and noisy samples based on the comparison results between the label of the original labeled sample corresponding to the enhanced sample and the pseudo label of the enhanced sample. The second classification module is used to introduce the class probability distribution of the clean sample predicted by the pre-trained model after Monte Carlo sampling training under different model parameters, so as to obtain the confidence of the clean sample according to the class probability distribution under different model parameters, and classify the clean sample into high confidence sample and low confidence sample according to the confidence level. The recall module is used to confirm the recall samples based on the comparison results of the lexical similarity of the samples to be recalled with a set lexical similarity threshold and the semantic fluency of the samples to be recalled with a set semantic fluency threshold; wherein, the samples to be recalled include low-confidence samples and noise samples. The selection module is used to select the high-confidence samples and the recalled samples as the final selected samples.
9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the sample selection method as described in any one of claims 1 to 7.
10. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to perform the sample selection method as described in any one of claims 1 to 7.