Picture duplicate removal method and device and storage medium

Abstract

The invention relates to a picture deduplication method and device and a storage medium, and the method comprises the following steps: a, collecting a to-be-deduplicated picture, and carrying out the preprocessing of the picture; b, training an encoder by using the preprocessed picture, and extracting a feature vector of the picture by using the trained encoder; c, clustering the pictures according to the feature vectors, and sequentially calculating the similarity between the pictures in the same kind of pictures; and d, removing other pictures similar to the same picture in the same kind of pictures. Before similarity comparison between the pictures is carried out, the pictures are clustered firstly, so that comparison of every two pictures in massive pictures is avoided, the calculation amount is reduced, and the picture duplicate removal efficiency is improved.

Description

technical field [0001] The invention relates to a picture deduplication method, device and storage medium. Background technique [0002] Image deduplication is an important part of image data set cleaning. Image deduplication technology can filter images with high similarity in content in the image data set. Therefore, there is no need to waste time on similar pictures, so as to save computer storage space, reduce computing resources of redundant pictures, and provide convenience for downstream picture usage tasks. [0003] In the existing picture deduplication technology, the features such as color, texture and shape of each picture in the picture data set are generally extracted first. Then use the similarity of the features to measure the degree of coincidence between pictures, and clear the pictures with a high degree of coincidence, so as to achieve the purpose of deduplication. Such methods usually use methods such as manual detection or MD5 matching to complete the ...

AI Technical Summary

Problems solved by technology

However, manual detection will generate large labor costs, and the detection speed is slow, and there are differences in standards

However, although the MD5 value of the image is used to search, although the speed is faster, it can only filter out the exact same image, and cannot complete the cleaning of similar images.

[0004] It can be seen that with the increase in the number of pictures in the picture library, the current technology needs to traverse the picture features in the database, which is time-consuming for the deduplication of massive pictures.

In addition, the similarity between pictures calculated by the existing method will increase exponentially with the number of comparisons, which will make the deduplication work more time-consuming, so the existing method is not suitable for deduplication operations of massive pictures

Images