Technology and system for automatically recognizing Chinese new words in single-word-string mode and affix mode

Abstract

The invention provides a technology and a system for automatically recognizing Chinese new words in a single-word-string mode and an affix mode, namely a method for recognizing the Chinese new words in the single-word-string mode and the affix mode on the basis of a large-scale short text language database. Statistics and rules are combined in the method, formation modes of the new words and statistics of word frequencies of the new words are combined, the new words are divided into new words in the single-word-string mode and new words in the affix mode, different extraction methods and different new word filtering methods are respectively adopted, and the Chinese new words in the two modes are extracted by combining the methods with word frequency information.

Description

technical field [0001] The invention relates to the field of natural language processing, in particular to a control method for automatic recognition and extraction of new Chinese words and a corresponding control system. Background technique [0002] Chinese automatic word segmentation is the basis for processing Chinese natural language, but due to the rapid development of information, Chinese language has undergone tremendous changes in a wide range of fields, and more and more new words continue to appear on the Internet. It has brought great challenges to the dictionary creation of Chinese word segmentation tools, and will inevitably lead to a decrease in the accuracy of word segmentation. Therefore, new word recognition has become a bottleneck in the field of Chinese information processing. The new word automatic recognition technology is of great help in improving the accuracy of Chinese word segmentation technology. In addition, the new word automatic discovery can ...

AI Technical Summary

Problems solved by technology

[0004] The advantages of the rule-based method are high accuracy and strong pertinence, but there will be great difficulties in establishing and maintaining rules

Moreover, the rules are generally related to certain fields, so the portability and adaptability are relatively poor

The advantages of statistical-based methods are flexibility, strong adaptability, and good portability, but they require a large-scale corpus for training.

And because there are relatively few features that can be counted, there are generally disadvantages of sparse data and low extraction accuracy.

Images