Method and device for deduplication of web pages

Abstract

The present application discloses a webpage deduplication method and device. The method comprises: extracting a feature code of a to-be-processed webpage; converting the feature code into a key value, and searching in a storage space whether the key value exists; if the key value exists, determining whether a preset requirement is met between the number of characters of the to-be-processed webpage and the number of characters of a webpage corresponding to the key value in the storage space; and if a determination result is that the preset requirement is met, determining that the to-be-processed webpage duplicates. The webpage deduplication method and device disclosed by the present application are capable of implementing effective deduplication for webpages with same contents and moreover are capable of saving storage resources and improving retrieval experience of users.

Description

technical field [0001] The present application relates to the field of the Internet, in particular to a method and device for deduplication of webpages. Background technique [0002] In the current web search results, users often get redundant pages with the same content, which not only waste storage resources, but also bring a lot of inconvenience to users' search [0003] However, there are not many methods for deduplication of Chinese webpages at present, and the methods are not complete. Among them, the main methods for deduplication of Chinese webpages are: method based on clustering, method based on the same URL, method based on keyword position sequence, method based on feature The method of sentence extraction, etc.; the following is a brief analysis of this: [0004] 1. Clustering method [0005] Clustering is to divide a collection of objects into several classes, and the objects in each class are similar to each other, but not similar to objects of other classes...

AI Technical Summary

Problems solved by technology

But the disadvantage is that the text content structure information of the webpage is not used, and the duplicate webpage caused by reprinting cannot be deduplicated.

[0023] This algorithm converts the full-text comparison of two texts into the comparison of several words and sentences, which reduces the time complexity and space complexity of the algorithm to a certain extent, but it is not ideal for large-scale webpage deduplication, because its Finding long common subsequences can take a lot of time

[0024] The above analysis shows that the webpage deduplication algorithms in the prior art have their own advantages, but they also have shortcomings; among them, for example, the method of clustering is inefficient, and the method of excluding the same URL cannot deduplicate webpages with the same text content; There are also various defects in other methods, so the technical problem of deduplication of web pages is not well solved in the prior art

Images