A web data similarity detection method based on two-level filtering of structure and content

Abstract

The invention discloses a Web data similarity detection method based on two-stage filtration of a structure and content. On the basis of a traditional universal similarity detection method, the distribution characteristics of the structure and the content of Web data are dug out, and detected document sets are subjected to two-stage filtration; the first-stage filtration of the two-stage filtration is structure similarity filtration, wherein each Web document is modeled into a Tag tree structure so as to remove the document sets dissimilar in structure, the remaining documents are subjected to key content extraction, key content is expressed in the form of tuple vectors, and key messages are connected to generate character string sets; the second-stage filtration of the two-stage filtration is to conduct Trie tree structure modeling on the character string sets generated after the first-stage filtration, and similar character strings are connected to obtain a final result. Multiple experiments prove that by the adoption of the method, the efficiency of data similarity detection in the web field can be improved remarkably.

Description

technical field [0001] The invention belongs to the field of similarity detection of Web data structure and content, and in particular relates to a method for similarity detection of Web data based on secondary filtering of structure and content. Background technique [0002] Web data is usually stored and exchanged in the form of XML or HTML, and the data is organized through various tags. Due to the fast speed of web data transmission and the flexible format of html webpages, the chances of reprinting or changing the widely spread webpages are relatively high, and the data organization forms belonging to the same topic may be different from each other, but the similarity is high. Therefore, the detection and exclusion of similarity data is very meaningful to the fields of data classification and clustering, data mining, focused search engines, pattern recognition and article detection. [0003] Traditional Web data similarity detection methods mainly focus on the two fiel...

AI Technical Summary

Problems solved by technology

For example, for documents and data in the web space, if the similarity mining is carried out according to the general existing similarity detection method, it is difficult to efficiently extract the data because the structural characteristics of the web data and the distribution area characteristics of the string content are not fully utilized. Find similar content blocks

Images