A screening method of vocabulary list

Abstract

The present invention provides a vocabulary screening method, comprising: determining the frequency of preset vocabulary in a pre-stored corpus; determining the position information of the preset vocabulary in a preset segment in the pre-stored corpus, and according to The location information obtains the location entropy corresponding to the preset vocabulary; according to the frequency of the determined preset vocabulary appearing in the pre-stored corpus and the obtained location entropy corresponding to the preset vocabulary, calculate the preset Vocabulary weights: According to the calculated weight results of the preset vocabulary, relevant preset vocabulary is screened out from a pre-stored corpus to form a screening vocabulary list. To improve the accuracy and reliability of screening vocabulary.

Description

technical field [0001] The invention relates to the technical field of vocabulary screening, in particular to a vocabulary screening method. Background technique [0002] As we all know, the choice of vocabulary has a huge impact on the performance of deep learning. If the vocabulary is too large, it will affect the performance of the online system on the one hand; on the other hand, it is not realistic, because there will be endless new words in various languages. , Compound words appear, that is, there is no upper limit to the size of the vocabulary; if the vocabulary is too small, it will also cause many unregistered words to appear frequently in practical applications, so it is very important to choose an appropriate vocabulary. [0003] Moreover, according to Zipf's Law, most of the vocabulary is in the long tail of word frequency. And according to the prior art of screening vocabulary, but generally to the screening of vocabulary, just give the weight of word accordin...

AI Technical Summary

Problems solved by technology

And according to the existing technology of screening vocabulary, but generally to the screening of vocabulary, just give the weight of word according to general statistical information such as word frequency in the corpus, take the word of TopN after sorting and put into vocabulary list

There are the following problems in the process of using its method to screen vocabulary: First, the selection of TopN is meaningless under the statistics of general word frequency: usually the size of N is determined by engineering needs

Through the above statement, it is not enough to use word frequency as the basis for screening vocabulary

Images