Chinese similar web page de-emphasis method based on microcosmic characteristic

Abstract

The invention discloses a Chinese similar web page de-emphasis method based on microcosmic characteristics in order to solve the problem of automatic detection of content similar to Chinese web pages. The Chinese similar web page de-emphasis method considering syntactic information and semantic information of web pages both comprises the following steps: firstly, establishing a text term co-occurrence picture according to extracted web page effective information; secondly, extracting document characteristic vectors, wherein the document characteristic vectors comprise keyword position information and keyword terms ; finally, establishing a document keyword inverted index file by sufficiently using a retrieval system and classified information; completing document characteristic vector retrieval match according to the inverted index file, and thereby, detecting and investigating similar web pages. The Chinese similar web page de-emphasis method can effectively reduce the harmful effect of arithmetic accuracy by noise information, considers the content and structure information of the web page text, sufficiently uses the advantages of a retrieval and classification system simultaneously, obtains good effect of de-emphasis accuracy rate larger than 90 percent and average recalling rate larger than 80 percent and is especially suitable for large-scale web page de-emphasis.

Description

technical field [0001] The invention relates to a method for deduplicating Chinese approximate web pages, which belongs to the technical field of computer network intelligent information retrieval. technical background [0002] With the unprecedented development of Internet technology and scale, the Internet has become one of the main channels for obtaining information. According to the survey as of July 2007, there are more than 125 million websites in total. Because of its convenient and quick retrieval function, search engine has become the main tool for information retrieval by network users today. Among them, the quality of information retrieval and its work efficiency will directly affect the overall performance of search engines. According to the statistical report released by China Internet Network Information Center in July 2005, when users answered the question of "the biggest problem encountered when searching for information", 44.6% of users chose the option of ...

AI Technical Summary

Problems solved by technology

In the worst case (all documents are approximate documents), the time complexity of the I-Match algorithm is O(nlogn)

[0010] These existing detection methods have the following defects and deficiencies: Shingle-based methods need to perform exact matching when detecting completely duplicate documents, which will cause documents with similar content to be missed

Term-based methods are not enough to only use keyword entries. Sometimes, web documents with different content may have the same keywords, which may cause misjudgment and is not enough for the detection of document similarity.

Images