Method for realizing Chinese named entity identification by utilizing uncertain word segmentation information

Abstract

The invention relates to a method for realizing Chinese named entity identification by utilizing uncertain word segmentation information. The invention aims to prevent word segmentation errors from being introduced into an identification system while enabling word segmentation information to play a role in the identification system. According to the method, a Chinese named entity recognition modelis realized by utilizing uncertain word segmentation information, the uncertain word segmentation information comprises all word segmentation conditions rather than one single condition, character candidate word segmentation position information is coded, and uncertain information of word segmentation is integrated by adopting a dynamic attention mechanism. In the recognition process, the model dynamically selects beneficial word segmentation information and automatically ignores error information, and finally an optimal word segmentation result is obtained. Compared with the prior art, the method has the advantages of effectively relieving error cascading, enhancing character vector semantic expression, being low in word segmentation error rate and the like.

Description

technical field [0001] The invention relates to the technical field of natural language processing (NLP), relates to a method for Chinese named entity recognition (NER), in particular to a method for realizing Chinese named entity recognition (UIcwsNN) by using uncertain word segmentation information. Background technique [0002] Named entity recognition is a fundamental task in the field of NLP and has rich underlying applications. However, compared with English, Chinese sentences have no delimiters, that is, Chinese text is a sequence of words, and words cannot be directly distinguished from each other, and word-level information is very important for named entity recognition. Existing word segmentation tools will output a large number of wrong word segmentation results, which makes named entity recognition difficult to achieve and the recognition effect is not ideal. [0003] The existing Chinese named entity recognition methods usually regard it as a character sequence...

AI Technical Summary

Problems solved by technology

The character sequence is tagged at the character level, and word segmentation information is added to the tagging system in the form of feature vectors. However, word segmentation error information will also be introduced. Even if the named entity and word segmentation model are trained at the same time, the word segmentation error information will still flow into the named entity. system, generating noise or errors, this multi-task joint learning will undoubtedly increase the overhead of model training

In summary, these methods have a common negligence when using word segmentation information, that is, they all introduce the correct word segmentation information into the entity recognition system or module. No matter whether the information is completely correct or not, the wrong word segmentation information will inevitably give Entity Recognition Brings Negative Disruption

Images