Data deduplication method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a data deduplication method and device, computer equipment and a storage medium. The data deduplication method comprises the steps: obtaining a data access request, and extracting a feature field in the data access request; cleaning the feature fields, and normalizing the cleaned feature fields; splicing the feature fields to generate a feature field combination, and compressing the feature field combination by using a hash algorithm; identifying the compressed feature field based on a preset database cluster, and judging whether the feature field is arepeated field or not according to an identification result; and if the feature field is a repeated field, storing the feature field into a preset exception handling queue, otherwise, outputting a prompt message which is used for prompting that the feature field is a normal field. The invention provides a data deduplication method based on data processing, capable of reducing the consumption of alarge amount of repeated data for database resources, saving the memory space of a database, and improving the reputation of an enterprise.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data deduplication method, device, computer equipment and storage medium. Background technique [0002] At present, when interacting with the EMS (Express Mail Service, express postal service) system, a large amount of repeated express logistics dynamic information is often obtained from the EMS system. Because the obtained large amount of express logistics dynamic information needs to be stored, it needs to occupy the background The large amount of storage space in the database creates a very large storage pressure on the background database. In severe cases, it may cause the background database to fail to work normally. In addition, due to the large amount of express logistics dynamic information in the background database, there are duplicate information. Certain screening or de-duplication processing will send repeated logistics text messages or other logistics ne...

AI Technical Summary

Problems solved by technology

[0002] At present, when interacting with the EMS (Express Mail Service, express postal service) system, a large amount of repeated express logistics dynamic information is often obtained from the EMS system. Because the obtained large amount of express logistics dynamic information needs to be stored, it needs to occupy the background The large amount of storage space in the database creates a very large storage pressure on the background database. In severe cases, it may cause the background database to fail to work normally. In addition, due to the large amount of express logistics dynamic information in the background database, there are duplicate information. Certain screening or de-duplication processing will send repeated logistics text messages or other logistics news pushes to corresponding users, which will easily cause user complaints and have a serious impact on corporate reputation

Images