Big data URL deduplication method based on convolutional features and multiple hash mapping

A hash mapping and big data technology, applied in network data retrieval, other database retrieval, and other database queries, etc., can solve the problems of slow log URL deduplication processing speed and poor effect, saving resources and reducing hash collisions The probability of , the effect of fast recognition speed

CN108874941BActive Publication Date: 2021-09-21成都知道创宇信息技术有限公司
10 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2021-09-21

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a large data URL deduplication method based on convolution features and multiple hash mappings. By using a fast feature mapping algorithm based on convolution features and multiple hash function mappings, a BitSet is established, and each URL is After a convolution operation and multiple hash functions are mapped to multiple bits, the probability of collision is greatly reduced, thereby identifying URLs in independent web logs. Compared with the traditional deduplication algorithm, the method of the invention saves more resources, and at the same time greatly reduces the probability of hash collision, and the recognition speed is very fast.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the technical field of URL deduplication, in particular to a large data URL deduplication method based on convolution features and multiple hash maps. Background technique

[0002] One of the existing big data URL processing technologies is to use HashSet to store the visited URLs, and it only takes a cost close to O(1) to find out whether a URL has been visited. This method has the following disadvantages: It consumes too much memory. With the increase of URLs, it will occupy more and more memory. Even if there are only 100 million URLs, each URL is counted as 50 characters, and 5GB of memory is required. The actual amount of large data processing will be much larger.

[0003] The second of the existing big data URL processing technology is that the URL is saved to HashSet or database after one-way hashing such as MD5 or SHA-1. Since the length of the information summary after the string is processed by MD5 is only 128Bit, af...

Examples

Embodiment Construction

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0032] The present invention establishes a BitSet by using a fast feature mapping algorithm based on convolution features and multiple hash function mapping, and maps each URL to multiple bits through a convolution operation and multiple hash functions, greatly reducing conflicts probability, thereby identifying URLs in individual web logs. The details are as follows:

[0033] 1. Extract the WEB access log from the WEB server or the traditional WAF device, and then filter out the required HOST (domain name) and URL (uniform resource locator). Specific screening method:

[0034] 1. Filter out redundant fields in WAF logs

[0035] You can use the filter method in scala language to filter. First use the split method in scala to split each log by a space character:

[0036] val fields=line. split("")

[0037] Then customize a filter...