Method and device for labeling corpus

Abstract

The invention discloses a corpus labeling method and device, which obtains the text vector of the corpus by vectorizing the corpus to be processed; clusters the corpus by using the DBSCAN clustering algorithm according to the text vector of the corpus class processing to obtain long-tail corpus and to-be-labeled corpus; for the long-tail corpus, return to clustering again; for the to-be-labeled corpus, set labels to obtain the labeled corpus; The corpus is merged to obtain the final labeled corpus, without the need to adjust the number of clusters multiple times, the algorithm is simpler, the labeling efficiency is higher, and the reliability is better.

Description

Technical field[0001]The present invention relates to the field of natural language processing, in particular, a tanguition method and apparatus for applying the method.Background technique[0002]The corpus is the basic resources of the corpus linguistic research, but also the main resources of empirical language research methods. The traditional corpus is mainly used in dictionary compilation, language teaching, traditional language research, natural language processing based on statistics or examples. With the development of Internet big data and artificial intelligence technology, corpus is also widely used.[0003]The corpus is stored in the actual use of the language in the actual use of the language. For example, the user message directly from the web page, customer service dialogue, etc .; the corpus is the basic resource that carries the language knowledge, but does not mean language knowledge; true corpus It is necessary to be processed to become useful resources, and the proc...

AI Technical Summary

Problems solved by technology

[0006] The second is to use an unsupervised algorithm (clustering algorithm) to cluster the labeled data, and then label each category; this method can directly label the corpus without relying on too much prior information, but some manual intervention is required later

[0007] For the second labeling method, the most common one is to pre-label the corpus with the k-means algorithm as the core algorithm, but the disadvantage is that because k-means needs to achieve a given number of clusters, the possible result is that it depends on experience first Specify a number of clusters, and then adjust the value of the number of clusters continuously according to the clustering effect until the appropriate value is adjusted. The whole process is too redundant

Images