Method and device for deduplication of web pages

A webpage and equipment technology, applied in the field of webpage deduplication methods and equipment, can solve the problems of low efficiency, inability to duplicate webpage deduplication, and inability to deduplicate webpages with the same text content, so as to save storage resources, improve retrieval experience, and improve retrieval efficiency. The effect of the experience

CN105574004BActive Publication Date: 2019-06-21ADVANCED NEW TECH CO LTD
8 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2019-06-21

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The present application discloses a webpage deduplication method and device. The method comprises: extracting a feature code of a to-be-processed webpage; converting the feature code into a key value, and searching in a storage space whether the key value exists; if the key value exists, determining whether a preset requirement is met between the number of characters of the to-be-processed webpage and the number of characters of a webpage corresponding to the key value in the storage space; and if a determination result is that the preset requirement is met, determining that the to-be-processed webpage duplicates. The webpage deduplication method and device disclosed by the present application are capable of implementing effective deduplication for webpages with same contents and moreover are capable of saving storage resources and improving retrieval experience of users.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present application relates to the field of the Internet, in particular to a method and device for deduplication of webpages. Background technique

[0002] In the current web search results, users often get redundant pages with the same content, which not only waste storage resources, but also bring a lot of inconvenience to users' search

[0003] However, there are not many methods for deduplication of Chinese webpages at present, and the methods are not complete. Among them, the main methods for deduplication of Chinese webpages are: method based on clustering, method based on the same URL, method based on keyword position sequence, method based on feature The method of sentence extraction, etc.; the following is a brief analysis of this:

[0004] 1. Clustering method

[0005] Clustering is to divide a collection of objects into several classes, and the objects in each class are similar to each other, but not similar to objects of other classes...

Examples

Embodiment Construction

[0068] The purpose of this application is to provide a method and device for deduplication of web pages, which are used to effectively deduplicate web pages with the same content, save storage resources, and improve user search experience.

[0069] In order to achieve the above technical objectives, such as figure 1 As shown, Embodiment 1 of the present application provides a webpage deduplication method, which specifically includes the following steps:

[0070] Step 101, extracting the feature code of the webpage to be processed;

[0071] Specifically, before step 101, it also includes: determining the type of webpage obtained; if the webpage obtained is a themed webpage (that is, a webpage containing text content), the text content of the webpage obtained is uniformly edited , and use the edited webpage as the webpage to be processed.

[0072] Specifically, after it is determined that the obtained webpage is a themed webpage, since the editing formats of the text content o...