Content extraction method based on keyword matching

Abstract

The invention discloses a content extraction method based on keyword matching. Keywords in a webpage source code Keywords label are counted, a standard library is built according to the keywords, and a corresponding DOM tree is built; hierarchical traversal is conducted on the DOM tree, the number of keywords contained in all nodes of the DOM trees is counted, the keyword weights of the nodes are calculated according to the ratio relation between the keyword number of the nodes and the keyword number of father nodes of the nodes, the content node containing a content text is effectively screened out and positioned by judging the maximum keyword weight of node children, and content extraction is completed; and a similarity matching method is proposed in order to solve the problem that a short text cannot be effectively extracted through a keyword matching method, and according to the similarity matching method, paragraph texts and page titles are converted into 8-bit binary data, the similarity is judged according to the Hamming distance, and content extraction of the short text is achieved. According to the content extraction method, matching is conducted according to the keywords set in a webpage, neither data training nor sample learning is needed, and the method is out of limitation of a website structure and has the good universality.

Description

technical field [0001] The invention relates to the technical field of text mining, in particular to a text extraction method based on keyword matching. Background technique [0002] With the rapid development of Web technology, web pages have become the main carrier of information release and information consumption. Therefore, in the monitoring of public opinion on the Internet, it is very important to strengthen the information filtering of web pages; and in the information filtering of web pages, the information extraction or text extraction of web pages becomes the key. However, there are many types of existing webpages, different webpages have different structures, and the website will be revised from time to time. At the same time, a large number of advertisements and other noises are inserted in the webpages. These problems make it difficult to extract the text of the webpage. Existing text extraction methods mainly include: (1) Realize text extraction by analyzing ...

AI Technical Summary

Problems solved by technology

However, there are many types of existing webpages, different webpage structures, and the website will be revised from time to time. At the same time, a large number of advertisements and other noises are inserted in the webpage. These problems make it difficult to extract the text of the webpage.

Existing text extraction methods mainly include: (1) Realize text extraction by analyzing the WLR and node hierarchical relationship of DOM tree nodes. This kind of method has high time complexity and low efficiency; (2) Design label path The feature system realizes the distinction between text and noise from different perspectives. On the basis of feature similarity analysis, the feature fusion strategy based on combined feature selection can quickly and efficiently extract text. However, this type of method is highly dependent on the structure of the website. ; (3) Automatic information extraction, which only extracts web pages based on their own relevant characteristics. This method has a relatively high error rate in the text extraction of short text web pages

Images