Network text named entity recognition method based on neural network probability disambiguation

Abstract

The invention discloses a network text named entity recognition method based on neural network probability disambiguation. The method includes: performing word segmentation on an unlabeled corpus, utilizing Word2Vec to extract a word vector, converting a sample corpus into a word feature matrix, windowing, building a deep neural network for training, adding a softmax function into an output layer of the neural network, and performing normalization to acquire a probability matrix of named entity type corresponding to each word; re-windowing the probability matrix, and utilizing a condition random field model for disambiguation to acquire final named entity annotation. A word vector increment learning method without changing structure of the neural network is provided according to the characteristic that network words and new words exist, and a probability disambiguation method is adopted to deal with the problem that network texts are nonstandard in grammatical structure and contain a lot of wrongly written or mispronounced characters, so that the method has high accuracy in network text named entity recognition tasks.

Description

technical field [0001] The invention relates to the processing and analysis of network texts, in particular to a method for network text named entity recognition based on neural network probability disambiguation. Background technique [0002] The network makes the speed and scale of information collection and dissemination reach an unprecedented level, and realizes global information sharing and interaction. It has become an indispensable infrastructure for the information society. Modern communication and dissemination technologies have greatly improved the speed and breadth of information dissemination. But the problems and "side effects" that come with it are: sometimes people are at a loss due to the surging information, and it becomes very difficult to quickly and accurately obtain the information they need most from the vast sea of ​​information. How to analyze named entities such as people, places, and institutions that Internet users are concerned about from massiv...

AI Technical Summary

Problems solved by technology

The reason why artificial neural networks are difficult to be practical is that artificial neural networks often need to convert words into vectors in the word vector space in the field of named entity recognition, so the corresponding vectors cannot be obtained for new vocabulary, so large-scale practical applications cannot be obtained

[0005] Based on the above status quo, the named entity recognition for network text mainly has the following problems: First, due to the large number of network vocabulary, new vocabulary, and typos in network text, it is impossible to train a word vector space containing all words to train the neural network.

Second, the arbitrary language forms, irregular grammatical structures, and many typos in network texts lead to a decline in the accuracy of named entity recognition.

Images