Bloom filter and data processing method

Abstract

The invention provides a Bloom filter and a data processing method, each bit of a bit array of the Bloom filter is expanded into a counter, and the Bloom filter comprises: an initialization module used for initializing configuration parameters and generating a hash object used for storing the configuration parameters during initialization; the duplicate judgment verification module that is used for judging whether the received key exists or not; the storage module that is used for storing the key when the judgment and verification module judges that the received key does not exist, and addingone to the counter value corresponding to each bit; and the deleting module that is used for deleting the key when the judgment and verification module judges that the received key exists, and subtracting one from the counter value corresponding to each bit. According to the invention, the Bloom filter has a deletion function, the data accuracy can be ensured, and the high availability of the Bloom filter service can be improved.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a Bloom filter and a data processing method. Background technique [0002] In the context of a large amount of data, it is used to deal with repetitive filters for filtering data. Commonly used scenarios are like and unlike functions, follow and unfollow functions. When the number of users is particularly large, if you still use the key and value methods in related technologies to store records, the required memory will be very large, and non-hot data is not conducive to query data speed. In addition, there are application scenarios including file upload, verification of repeated image upload, etc. In these application scenarios, if there is no function that can be deleted when Bloom filter is selected as the storage and judgment basis, these application scenarios cannot be applied, because click There are inverse operations for likes, concerns, files, etc. Therefore, t...

AI Technical Summary

Problems solved by technology

When the number of users is particularly large, if you still use the key and value methods in related technologies to store records, the required memory will be very large, and non-hot data is not conducive to query data speed

In addition, there are application scenarios including file upload, verification of repeated image upload, etc. In these application scenarios, if there is no function that can be deleted when Bloom filter is selected as the storage and judgment basis, these application scenarios cannot be applied, because click There are inverse operations for likes, concerns, files, etc.

Images