Unregistered word identification method and system using five-stroke character root deep learning

Abstract

The invention belongs to the technical field of natural language data processing, and discloses an unregistered word identification method and system using five-stroke character root deep learning. The method comprises the steps of converting a Chinese character into four English letters according to a five-stroke character root table; then inputting an embedded vector serving as an embedded vector of the model into an embedded vector corresponding to the words in a corpus to train a neural network model; and finally, enabling the model to output a most similar vocabulary vector in a previous corpus, and using the vocabulary vector as an important basis for identifying the unlogged vocabularies to better identify the unlogged vocabularies. According to the present invention, the Chinese character words with close radicals mostly have the same part-of-speech, and the five-stroke codes of the Chinese character words are similar, so that the neural network entity identification method based on the five-stroke roots is provided and can improve the performance of identifying the unlogged words through the neural network model. According to the present invention, the word vectors are used for representing the words based on deep learning, so that the sparse problem of the high-latitude vector space is solved, and the method is simpler and more effective.

Description

technical field [0001] The invention belongs to the technical field of processing natural language data, and in particular relates to an unregistered word recognition method and system using deep learning of Wubi radicals. Background technique [0002] At present, the commonly used existing technologies in the industry are as follows: "Named Entities", which are widely used in the field of natural language processing, were originally proposed at the Sixth Information Understanding Conference in 1996, and most of the research on MUC-6 is based on rules Methods, such as lexical rules for word forms or parts of speech. Formulate character matching rules based on prompt words before and after named entities, context, etc., mainly focusing on information extraction tasks. Named entities are objects of interest that can be used to solve specific problems. Sekine believes that the general seven subcategories of named entities cannot meet the application requirements of automatic q...

AI Technical Summary

Problems solved by technology

[0006] (1) The rule-based method needs to manually formulate several rules, the feasibility is too low, when the application fields are very different, the portability is poor, and the rules need to be re-formulated

[0007] (2) The method based on machine learning and the recognition method based on the neural network model cannot recognize unregistered words

[0011] At present, the professional terms in various fields are complex in categories, general in content, large in information volume, and complex in composition

As a result, people cannot accurately and completely describe or express, but use some aliases, abbreviations, words, etc. to describe. Then, problems arise, and there are often typos, ambiguous words, similar meanings, etc.

This would have a severe impact on the domain's name recognition

Images