Automatic text collection method for specific field

Abstract

The invention provides an automatic text collection method oriented to a specific field. The method comprises the following steps: establishing a url scheduling link pool and a task manager according to a preset target site; the method comprises the following steps: mapping input keywords and words in a Chinese synonym library to a high-dimensional vector space through word2vec, and calculating to generate a subject word group; the task manager cleans the html unstructured data in the accessed url page to obtain a long text; extracting Chinese feature sentences of the long text, extracting feature words in the Chinese feature sentences of the long text, and generating a long text feature word group; mapping the subject word group and the long text feature word group to a high-dimensional vector space, and calculating a semantic similarity value of the subject word group and the long text feature word group; and if the semantic similarity value reaches a set threshold value, writing the structured data in the url page corresponding to the long text feature word group into a database. According to the method, domain focusing is realized by converting rule matching of Chinese words and long texts into semantic distance calculation.

Description

technical field [0001] The invention belongs to the technical field of computer information and natural language processing, and in particular relates to an automatic text collection method oriented to a specific field. Background technique [0002] With the development of the Internet, rich and diverse text data are spread on the Internet. As a mainstream information, Internet text information has greater research value than other information sources. It is very necessary to focus and collect Internet news accurately and efficiently. It is of great significance in the fields of information retrieval and data mining. [0003] Text capture is a program that automatically crawls web pages and extracts web content, the purpose of which is to obtain information resources from the Internet. The general collection method does not distinguish which data the user wants in the collection process, so it will cause a large amount of invalid data to be stored, which not only brings cha...

Images