Text sampling method and device for improving annotation efficiency

Abstract

The invention discloses a text sampling method and device for improving annotation efficiency. The sampling method comprises the steps: S1, acquiring an original text set; s2, extracting a vector of each original text through a vectorization model according to the original text; s3, judging the degree of the original text piece according to the vector of each original text, and removing the original text with high similarity from the original text set to obtain a final text set; and S4, in the final text of the final text set, performing random sampling to obtain a final annotation text. Therefore, the information amount of text sampling is effectively improved, the text annotation efficiency is improved, and the final model performance is improved.

Description

technical field [0001] The present invention relates to a text sampling method and device, in particular to a text sampling method and device for improving labeling efficiency. Background technique [0002] Natural language processing (Natural Language Processing, NLP) technology can efficiently systematically analyze, understand and extract text data, so that computers can understand natural language and generate natural language, and then realize effective communication between humans and computers using natural language. Interaction (e.g. use of applications such as message auto-reply, voice assistant, etc.). [0003] In order to realize the industrial application of natural language processing, it is necessary to manually label a part of the text data in some scenarios for model training. For example, for Weibo data, mark "XX is so handsome" as a female user's Weibo, and "Girlfriend's birthday, what gift would you like to send" as a male user's Weibo, and then use natur...

AI Technical Summary

Problems solved by technology

[0006] 1. As a result, the final extracted data for labeling is relatively similar, and the amount of information carried between texts is low, resulting in a reduction in the labeling effect;

[0007] 2. It is necessary to determine the concept set, which has poor versatility and cannot effectively deal with scenarios without concept keywords / regularity;

[0008] 3. It is necessary to divide attribute tags and emotions, which has poor versatility and cannot effectively handle texts with atypical relationships between attribute tags and emotions, and it is more complicated to deal with the relationship between attribute tags and emotions, and the cost is significantly higher than that commonly used in this field label cost

Images