Web data similarity detection method based on two-stage filtration of structure and content

Abstract

The invention discloses a Web data similarity detection method based on two-stage filtration of a structure and content. On the basis of a traditional universal similarity detection method, the distribution characteristics of the structure and the content of Web data are dug out, and detected document sets are subjected to two-stage filtration; the first-stage filtration of the two-stage filtration is structure similarity filtration, wherein each Web document is modeled into a Tag tree structure so as to remove the document sets dissimilar in structure, the remaining documents are subjected to key content extraction, key content is expressed in the form of tuple vectors, and key messages are connected to generate character string sets; the second-stage filtration of the two-stage filtration is to conduct Trie tree structure modeling on the character string sets generated after the first-stage filtration, and similar character strings are connected to obtain a final result. Multiple experiments prove that by the adoption of the method, the efficiency of data similarity detection in the web field can be improved remarkably.

Description

Technical field [0001] The invention belongs to the field of similarity detection of Web data structure and content, and in particular relates to a Web data similarity detection method based on secondary filtering of structure and content. Background technique [0002] Web data is usually stored and exchanged in the form of XML or HTML, and the data is organized through various tags. Due to the fast dissemination of web data and the flexible format of html webpages, the more widely disseminated webpages are more likely to be reprinted or modified, and the data organization forms of the same subject may be different from each other, but the similarity is higher. Therefore, it is very meaningful to detect and exclude similarity data, to classify and cluster data, data mining, focused search engines, pattern recognition and article detection. [0003] Traditional Web data similarity detection methods mainly focus on the two fields of structural similarity detection of Web data and si...

AI Technical Summary

Problems solved by technology

For example, for documents and data in the web space, if the similarity mining is carried out according to the general existing similarity detection method, it is difficult to efficiently extract the data because the structural characteristics of the web data and the distribution area characteristics of the string content are not fully utilized. Find similar content blocks

Images