Method and device for batch structuring of large-scale text information

Abstract

The invention discloses a large-scale text information batch structuring method and apparatus, which can process a large amount of text information in batches in a short time. The method comprises thesteps of establishing different segmentation and extraction rules according to target information of different text information, and providing a rule input interface in a configuration file form; according to corresponding segmentation and extraction rules, finishing automatic information extraction operation on single text information in sequence in a pipeline processing way; and according to data type format and length of a target information item, establishing a database relationship table, converting the text information subjected to automatic information extraction into structured records, storing the structured records in the database relationship table, inferring candidate information items from successfully extracted structured record data by using a statistic machine learning method for the text information from which key information is not successfully extracted, and correcting existing rules according to the candidate information items and the corresponding text informationand rule contents.

Description

technical field [0001] The invention relates to the field of computers, in particular to a method and device for batch structuring of large-scale text information. Background technique [0002] In recent years, with the advent of the era of big data, the rapid growth of data has become a common opportunity and challenge for many industries. The part of "opportunity" is that by analyzing a large amount of data, data owners can dig out many frequent patterns and obtain a lot of potential information, and can predict the future trend and development of related industries based on this information, so as to make corresponding decisions , to obtain a large amount of income; and the "challenge" part is that although through the Internet, anyone can easily obtain a large amount of data, and even some professionals can crawl the Internet more quickly and efficiently through crawlers. However, how to efficiently manage and utilize such massive data has become a difficult problem. ...

AI Technical Summary

Problems solved by technology

However, this method is not only very inefficient, but also wastes a lot of human resource costs and increases overhead.

In addition, this method also has great limitations. It is necessary to hire a large number of corresponding personnel to analyze and extract each type of different text information, and it is not reusable. , is not a very good method and cannot meet the needs of today's big data era

Images