Internet text entity recognition method and system, electronic equipment and storage medium

Abstract

The invention discloses an internet text entity recognition method and system, electronic equipment and a storage medium. The method comprises the following steps: inputting a historical internet text into an entity recognition AI model to obtain an initialized full quantizer table; constructing a full quantizer dictionary tree according to the initialized full quantizer table; according to an entity recognition AI model and the full quantizer dictionary tree, performing recognition processing on the real-time sampled internet text to obtain a selected word list; constructing a selected word dictionary tree according to the selected word list; splitting the real-time internet text to be recognized according to preset Chinese sentence segmentation symbols to obtain split sub sentences; matching the split sub-sentences with the selected word dictionary tree to obtain matched sub-sentences; and splicing the matched clauses according to a preset sequence, inputting the spliced matched clauses into an entity recognition AI model to obtain an entity recognition result, and performing category output according to entity categories. The to-be-recognized real-time internet text is screened sentence by sentence according to the selected word list, and only sentences possibly containing entities are left, so that the calculated text quantity is greatly reduced, and the operation cost is reduced.

Description

technical field [0001] The invention relates to the field of text data processing, in particular to an Internet text entity recognition method, system, electronic equipment and storage medium. Background technique [0002] Entity recognition is a very important part of text sequence labeling tasks. Its full name in Chinese is "Named Entity Recognition", and its English name is "NER". Identification and extraction of name and other information. [0003] With the development of the Internet, the text data carried by the Internet is also increasing rapidly, and the demand for entity recognition is becoming increasingly urgent in more and more scenarios. Better and faster requirements are put forward for the recognition effect and computing performance of the entity recognition model. [0004] At present, the common practice in the industry for entity recognition is to manually mark a large amount of text, and then rely on the marked corpus to use Bert / BiLSTM / TextCNN and other...

AI Technical Summary

Problems solved by technology

[0006] In view of this, the embodiment of the present invention provides an Internet text entity recognition method, system, electronic equipment and storage medium to solve the problem of high operating cost of entity recognition in the prior art

Images