Data processing method and apparatus, and storage medium

Abstract

The application provides a data processing method and device and a storage medium, relates to the technical field of data deduplication, and comprises the following steps: obtaining to-be-inspected data, the to-be-inspected data is obtained according to monitoring data, performing hash processing on the to-be-inspected data to obtain a first hash code, performing hash processing on the first hash code based on multiple different hash functions to obtain multiple second hash codes corresponding to the to-be-inspected data, assigning values of corresponding dimensions in a data set to preset values according to the multiple second hash codes, and determining that the monitoring data is duplicate data if the values of the corresponding dimensions before the assignment are all preset values. The to-be-inspected data is processed twice in the application, the first hash processing effectively shortens the data length, the second hash processing includes multiple different hash functions and data sets of different dimensions, effectively reduces the probability of hash conflict, and effectively improves the speed and accuracy of data deduplication.

Images