Unregistered word identification method and system using five-stroke character root deep learning

Abstract

The invention belongs to the technical field of natural language data processing, and discloses an unregistered word identification method and system using five-stroke character root deep learning. The method comprises the steps of converting a Chinese character into four English letters according to a five-stroke character root table; then inputting an embedded vector serving as an embedded vector of the model into an embedded vector corresponding to the words in a corpus to train a neural network model; and finally, enabling the model to output a most similar vocabulary vector in a previous corpus, and using the vocabulary vector as an important basis for identifying the unlogged vocabularies to better identify the unlogged vocabularies. According to the present invention, the Chinese character words with close radicals mostly have the same part-of-speech, and the five-stroke codes of the Chinese character words are similar, so that the neural network entity identification method based on the five-stroke roots is provided and can improve the performance of identifying the unlogged words through the neural network model. According to the present invention, the word vectors are used for representing the words based on deep learning, so that the sparse problem of the high-latitude vector space is solved, and the method is simpler and more effective.

Description

technical field

[0001] The invention belongs to the technical field of processing natural language data, and in particular relates to an unregistered word recognition method and system using deep learning of Wubi radicals. Background technique

[0002] At present, the commonly used existing technologies in the industry are as follows: "Named Entities", which are widely used in the field of natural language processing, were originally proposed at the Sixth Information Understanding Conference in 1996, and most of the research on MUC-6 is based on rules Methods, such as lexical rules for word forms or parts of speech. Formulate character matching rules based on prompt words before and after named entities, context, etc., mainly focusing on information extraction tasks. Named entities are objects of interest that can be used to solve specific problems. Sekine believes that the general seven subcategories of named entities cannot meet the application requirements of automatic q...

Examples

Embodiment 1

[0054] The present invention is a model combining LSTM and Wubi radicals for the recognition of Chinese named entities. The invention encodes the input character sequence and all potential words matching the Wubi radical dictionary. In contrast to character-based approaches, the present invention explicitly utilizes word and word order information. Gated recurrent units enable the model to select the most relevant characters and words from a sentence to generate better named entity recognition results.

[0055] In terms of input word embedding, the embodiments of the present invention use Wubi radicals to represent Chinese characters, and these representations are combined as character embeddings, which can enhance the exploration of morphological and semantic information of characters, and automatically extract n-gram features with neural networks. Divide each character into strokes to propose an n-gram model, each character is represented by 4 English letters. For differen...

technical field

[0001] The invention belongs to the technical field of processing natural language data, and in particular relates to an unregistered word recognition method and system using deep learning of Wubi radicals. Background technique

[0002] At present, the commonly used existing technologies in the industry are as follows: "Named Entities", which are widely used in the field of natural language processing, were originally proposed at the Sixth Information Understanding Conference in 1996, and most of the research on MUC-6 is based on rules Methods, such as lexical rules for word forms or parts of speech. Formulate character matching rules based on prompt words before and after named entities, context, etc., mainly focusing on information extraction tasks. Named entities are objects of interest that can be used to solve specific problems. Sekine believes that the general seven subcategories of named entities cannot meet the application requirements of automatic q...

Examples

Embodiment 1

[0054] The present invention is a model combining LSTM and Wubi radicals for the recognition of Chinese named entities. The invention encodes the input character sequence and all potential words matching the Wubi radical dictionary. In contrast to character-based approaches, the present invention explicitly utilizes word and word order information. Gated recurrent units enable the model to select the most relevant characters and words from a sentence to generate better named entity recognition results.

[0055] In terms of input word embedding, the embodiments of the present invention use Wubi radicals to represent Chinese characters, and these representations are combined as character embeddings, which can enhance the exploration of morphological and semantic information of characters, and automatically extract n-gram features with neural networks. Divide each character into strokes to propose an n-gram model, each character is represented by 4 English letters. For differen...

Images