Grain condition named entity recognition method based on new word discovery and Flat-lattice

Abstract

The invention provides a grain condition named entity recognition method based on new word discovery and Flat-lattice. The method comprises the following steps: firstly, crawling grain condition related data from a known network and a grain dictionary by utilizing a python crawler technology to form a text corpus, and preprocessing corpora; then, obtaining new words from the grain condition text corpus by using an N-grams algorithm, assisting a word segmentation algorithm to perform word segmentation, and constructing a grain condition dictionary by using Word2vec according to a result after word segmentation; then, dividing the dictionary into 15 entity category labels, and performing BIOES labeling on the grain situation corpus according to the labels; thirdly, coding the input characters and all words which can be matched in the dictionary by adopting a Flat-lattice model, and inputting the coded input characters and all words into the model for training; and finally, performing prediction by using the trained deep learning model. According to the grain condition named entity recognition method based on new word discovery and Flat-lattice, grain condition entities can be effectively extracted from multi-source heterogeneous data, and a basis is provided for downstream tasks such as grain condition knowledge graph construction.

Description

Technical field [0001] The present invention is a field of natural language processing, in particular to a new word based on discovery of Flat-lattice and grain condition named entity recognition method. Background technique [0002] With the rapid development of information technology in the food industry, "information explosion" and "lack of knowledge" increasingly serious contradiction. The mass of grain condition record text data, papers and patents grain condition accumulation, efficiently and accurately excavated grain condition data from these entities, for follow-up studies, such as grain condition decision system constructed grain condition and mapping knowledge with for greater convenience. In extracting data from multiple sources in a heterogeneous specific physical process, named entity recognition (Named Entity Recognition, referred NER) is an indispensable technique. By named entity recognition technique, may be extracted from grain condition data showing informatio...

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a grain situation named entity recognition method based on new word discovery and Flat-lattice, which is used to solve the low recognition rate of proper nouns in the grain situation field, wrong word segmentation affects the effect of entity recognition, and the current lack of follow-up research The structured grain data set and other issues

Images