Image deduplication method, device, apparatus and storage medium

Abstract

The application relates to an image deduplication method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: obtaining a plurality of target images. Each target image is processed by a preset compression algorithm to generate feature information of each target image, so as to obtain a plurality of feature information. The plurality of target images and the plurality of feature information are input into a trained clustering model to obtain at least one first clustering cluster. The first clustering cluster comprises at least one target image, and the similarity between the feature information of the target image in the first clustering cluster and the clustering center information corresponding to the first clustering cluster is greater than a first preset similarity threshold. The target images in each first clustering cluster are screened to obtain at least one second clustering cluster. The similarity between any two target images in the second clustering cluster is less than a second preset similarity threshold. Therefore, the time for deduplicating the plurality of images can be reduced.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image deduplication method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, with the development of information technology, management devices (such as terminals and servers) have increasingly higher demands for information management. For example, management devices need to perform deduplication processing on multiple images.

[0003] Currently, in the process of deduplicating multiple images, the management device first needs to acquire multiple images and determine the similarity between each pair of images. Then, based on the similarity between each pair of images, the management device filters out images with lower similarity to obtain the deduplicated images. However, in this technical solution, when the number of images acquired by the management device is large, it increases the computational workload of determining the similarity between pairs of images, and increases the time required for the management device to perform deduplication on multiple images. Summary of the Invention

[0004] This application provides an image deduplication method, apparatus, device, and storage medium to at least solve the technical problem of long processing times for deduplication of multiple images in related technologies. The technical solution of this application is as follows:

[0005] According to a first aspect of this application, an image deduplication method is provided. The image deduplication method includes: an image deduplication device (hereinafter referred to as the "deduplication device") acquiring multiple target images. The deduplication device processes each target image using a preset compression algorithm to generate feature information for each target image, thereby obtaining multiple feature information. The deduplication device inputs the multiple target images and the multiple feature information into a trained clustering model to obtain at least one first cluster. The first cluster includes at least one target image, and each first cluster corresponds to a cluster center. The similarity between the feature information of the target images in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. The deduplication device filters the target images in each first cluster to obtain at least one second cluster. The similarity between any two target images in the second cluster is less than a second preset similarity threshold.

[0006] Based on the aforementioned technical means, the deduplication device can acquire multiple target images and process each target image using a preset compression algorithm to generate feature information for each target image, thereby obtaining multiple feature information. In other words, the deduplication device can compress the image information of each image using a preset compression algorithm, generating unique, concise information that indicates the image's features. This reduces the computational load of subsequent clustering processing of multiple images by the deduplication device, improving the speed of clustering. Then, the deduplication device can input the multiple target images and multiple feature information into a trained clustering model to obtain at least one first cluster. Each first cluster includes at least one target image, and each first cluster corresponds to a cluster center. The similarity between the feature information of the target images in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. That is, the deduplication device can perform unsupervised clustering operations using the trained clustering model. This improves the efficiency of clustering multiple images by the deduplication device and reduces the time spent on subsequent deduplication processing of multiple images. Subsequently, the deduplication device can filter the target images in each first cluster to obtain at least one second cluster, where the similarity between any two target images in the second cluster is less than a second preset similarity threshold. In other words, the deduplication device can perform deduplication on multiple images by determining the similarity between pairs of images within each cluster, avoiding the need to determine the similarity between images from different clusters. This reduces the number of times the deduplication device needs to determine the similarity between pairs of images, reduces the time spent deduplicating multiple images, and improves the efficiency of deduplication on multiple images.

[0007] In one possible implementation, the second preset similarity threshold is the first threshold, and the similarity between target images is the feature similarity. The process of "the deduplication device filters the target images in each first cluster to obtain at least one second cluster" includes: the deduplication device determines a first similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster. The first similarity set includes the feature similarity between the feature information of any two target images in the corresponding first cluster. The deduplication device filters the target images in each first cluster based on the first threshold and the first similarity set corresponding to each first cluster to obtain at least one second cluster, where the feature similarity between the feature information of any two target images in the second cluster is less than the first threshold.

[0008] According to the aforementioned technical means, during the process of the deduplication device filtering target images in each first cluster to obtain at least one second cluster, the deduplication device can determine the feature similarity between any two target images in each first cluster based on the feature information of the target images in each first cluster. Then, the deduplication device can filter target images in each first cluster based on a first threshold and the feature similarity between the feature information of all target images in each first cluster to obtain at least one second cluster. In other words, the deduplication device can determine the similarity between pairs of images based on the feature information of each target image, and perform deduplication processing on multiple images based on the similarity between pairs of images. This reduces the computational load of the deduplication device in determining the similarity between pairs of images, and reduces the time required for the deduplication device to process multiple images.

[0009] In one possible implementation, the second preset similarity threshold is a second threshold, the similarity between target images is image similarity, and the similarity between feature information and cluster center information is feature similarity. The process of "the deduplication device filters the target images in each first cluster to obtain at least one second cluster" includes: the deduplication device determines a second similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster and the cluster center information corresponding to each first cluster. The second similarity set includes the feature similarity between the feature information of the target images in the corresponding first cluster and the cluster center information corresponding to the corresponding first cluster. The deduplication device filters the target images in each first cluster based on a third preset similarity threshold and the second similarity set corresponding to each first cluster to obtain at least one third cluster. The feature similarity between the feature information of the target images in the third cluster and the cluster center information corresponding to the third cluster is less than the third preset similarity threshold. The deduplication device determines a third similarity set corresponding to each third cluster based on the target images in each third cluster. The third similarity set includes the image similarity between any two target images in the corresponding third cluster. The deduplication device then filters the target images in each third cluster based on a second threshold and the third similarity set corresponding to each third cluster, obtaining at least one second cluster. The feature similarity between the feature information of the target images in the second cluster and the cluster center information corresponding to the second cluster is less than a third preset similarity threshold, and the image similarity between any two target images in the second cluster is less than the second threshold.

[0010] According to the above technical means, during the process of the deduplication device filtering target images in each first cluster to obtain at least one second cluster, the deduplication device can determine the feature similarity between the feature information of any target image in each first cluster and the cluster center information corresponding to the first cluster based on the feature information of the target images in each first cluster. Next, the deduplication device can filter target images in each first cluster based on a third preset similarity threshold and the feature similarity between the feature information of any target image in each first cluster and the cluster center information corresponding to the first cluster, obtaining at least one third cluster. Then, the deduplication device can determine the image similarity between any two target images in each third cluster based on the target images in each third cluster. Next, the deduplication device can filter target images in each third cluster based on a second threshold and the image similarity between all target images in each third cluster, obtaining at least one second cluster. Similarly, the deduplication device can determine the feature similarity between the feature information of any two target images in each third cluster based on the feature information of the target images in each third cluster. Next, the deduplication device can filter the target images in each third cluster based on the first threshold and the feature similarity between the feature information of all target images in each third cluster, thus obtaining at least one second cluster. In other words, the deduplication device can not only deduplicate multiple images by determining the similarity between pairs of images within a cluster, but also by determining the feature similarity between the feature information of an image within a cluster and the cluster center information corresponding to that cluster. This improves the accuracy and completeness of deduplication for multiple images.

[0011] In one possible implementation, the preset compression algorithm is any of the following algorithms: mean hash algorithm, perceptual hash algorithm, and difference hash algorithm.

[0012] Based on the aforementioned technical methods, the deduplication device can select different preset compression algorithms to generate image feature information according to requirements. This improves the operability of the image feature information generated by the deduplication device.

[0013] According to a second aspect of this application, an image deduplication device is provided, comprising: an acquisition module and a processing module. The acquisition module is used to acquire multiple target images. The processing module is used to process each target image using a preset compression algorithm to generate feature information for each target image, thereby acquiring multiple feature information. The processing module is further used to input the multiple target images and multiple feature information into a trained clustering model to obtain at least one first cluster, each first cluster including at least one target image, each first cluster corresponding to a cluster center, and the similarity between the feature information of the target images in the first cluster and the cluster center corresponding to the first cluster being greater than a first preset similarity threshold. The processing module is further used to filter the target images in each first cluster to obtain at least one second cluster, wherein the similarity between any two target images in the second cluster is less than a second preset similarity threshold.

[0014] In one possible implementation, the second preset similarity threshold is the first threshold, and the similarity between target images is the feature similarity. Specifically, the processing module is used to determine a first similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster. The first similarity set includes the feature similarity between the feature information of any two target images in the corresponding first cluster. The processing module is further used to filter the target images in each first cluster based on the first threshold and the first similarity set corresponding to each first cluster, obtaining at least one second cluster, where the feature similarity between the feature information of any two target images in the second cluster is less than the first threshold.

[0015] In one possible implementation, the second preset similarity threshold is a second threshold, the similarity between target images is image similarity, and the similarity between feature information and cluster center information is feature similarity. Specifically, the processing module is used to determine a second similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster and the cluster center information corresponding to each first cluster. The second similarity set includes the feature similarity between the feature information of the target images in the corresponding first cluster and the cluster center information corresponding to the corresponding first cluster. The processing module is further used to filter the target images in each first cluster based on a third preset similarity threshold and the second similarity set corresponding to each first cluster to obtain at least one third cluster. The feature similarity between the feature information of the target images in the third cluster and the cluster center information corresponding to the third cluster is less than the third preset similarity threshold. The processing module is further used to determine a third similarity set corresponding to each third cluster based on the target images in each third cluster. The third similarity set includes the image similarity between any two target images in the corresponding third cluster. The aforementioned processing module is further configured to filter the target images in each third cluster based on the second threshold and the third similarity set corresponding to each third cluster, thereby obtaining at least one second cluster. The feature similarity between the feature information of the target images in the second cluster and the cluster center information corresponding to the second cluster is less than the third preset similarity threshold, and the image similarity between any two target images in the second cluster is less than the second threshold.

[0016] In one possible implementation, the preset compression algorithm is any of the following algorithms: mean hash algorithm, perceptual hash algorithm, and difference hash algorithm.

[0017] According to a third aspect provided in this application, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions. The processor is configured to execute instructions to implement the methods described in the first aspect and any possible embodiments thereof.

[0018] According to a fourth aspect provided in this application, a computer-readable storage medium is provided that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0019] According to the fifth aspect provided in this application, a computer program product is provided, the computer program product including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0020] Therefore, the above-mentioned technical features of this application have the following beneficial effects:

[0021] (1) The deduplication device can acquire multiple target images and process each target image using a preset compression algorithm to generate feature information for each target image, thereby obtaining multiple feature information. In other words, the deduplication device can compress the image information of each image using a preset compression algorithm to generate unique, concise information that indicates the image's features. This reduces the computational load of subsequent clustering processing of multiple images by the deduplication device, improving the speed of clustering processing. Afterward, the deduplication device can input multiple target images and multiple feature information into a trained clustering model to obtain at least one first cluster. The first cluster includes at least one target image, and each first cluster corresponds to a cluster center. The similarity between the feature information of the target images in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. In other words, the deduplication device can complete unsupervised clustering operations using the trained clustering model. This improves the efficiency of clustering multiple images by the deduplication device and reduces the time required for subsequent deduplication processing of multiple images. Subsequently, the deduplication device can filter the target images in each first cluster to obtain at least one second cluster, where the similarity between any two target images in the second cluster is less than a second preset similarity threshold. In other words, the deduplication device can perform deduplication on multiple images by determining the similarity between pairs of images within each cluster, avoiding the need to determine the similarity between images from different clusters. This reduces the number of times the deduplication device needs to determine the similarity between pairs of images, reduces the time spent deduplicating multiple images, and improves the efficiency of deduplication on multiple images.

[0022] (2) During the process of the deduplication device filtering target images in each first cluster to obtain at least one second cluster, the deduplication device can determine the feature similarity between any two target images in each first cluster based on the feature information of the target images in each first cluster. Then, the deduplication device can filter target images in each first cluster based on a first threshold and the feature similarity between the feature information of all target images in each first cluster to obtain at least one second cluster. In other words, the deduplication device can determine the similarity between pairs of images based on the feature information of each target image, and perform deduplication processing on multiple images based on the similarity between pairs of images. This reduces the computational load of the deduplication device in determining the similarity between pairs of images and reduces the time required for the deduplication device to perform deduplication processing on multiple images.

[0023] (3) During the process of the deduplication device filtering target images in each first cluster to obtain at least one second cluster, the deduplication device can determine the feature similarity between the feature information of any target image in each first cluster and the cluster center information corresponding to the first cluster based on the feature information of the target images in each first cluster. Next, the deduplication device can filter target images in each first cluster based on a third preset similarity threshold and the feature similarity between the feature information of any target image in each first cluster and the cluster center information corresponding to the first cluster, to obtain at least one third cluster. Afterwards, the deduplication device can determine the image similarity between any two target images in each third cluster based on the target images in each third cluster. Next, the deduplication device can filter target images in each third cluster based on a second threshold and the image similarity between all target images in each third cluster, to obtain at least one second cluster. Similarly, the deduplication device can determine the feature similarity between the feature information of any two target images in each third cluster based on the feature information of the target images in each third cluster. Next, the deduplication device can filter the target images in each third cluster based on the first threshold and the feature similarity between the feature information of all target images in each third cluster, thus obtaining at least one second cluster. In other words, the deduplication device can not only deduplicate multiple images by determining the similarity between pairs of images within a cluster, but also by determining the feature similarity between the feature information of an image within a cluster and the cluster center information corresponding to that cluster. This improves the accuracy and completeness of deduplication for multiple images.

[0024] (4) The deduplication device can select different preset compression algorithms to generate image feature information according to requirements. In this way, the operability of the image feature information generated by the deduplication device can be improved.

[0025] It should be noted that the technical effects of any of the implementation methods in aspects two through five can be found in the technical effects of the corresponding implementation methods in aspect one, and will not be repeated here.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0028] Figure 1This is a schematic diagram of a communication system according to an exemplary embodiment;

[0029] Figure 2 This is a flowchart illustrating an image deduplication method according to an exemplary embodiment;

[0030] Figure 3 This is a schematic diagram illustrating an example of a cluster according to an exemplary embodiment;

[0031] Figure 4 This is an example diagram illustrating a training clustering model according to an exemplary embodiment;

[0032] Figure 5 This is a flowchart illustrating yet another image deduplication method according to an exemplary embodiment;

[0033] Figure 6 This is a block diagram illustrating an image deduplication device according to an exemplary embodiment;

[0034] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0035] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0036] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0037] Before providing a detailed description of the image deduplication method provided in the embodiments of this application, the implementation environment and application scenarios of the embodiments of this application will be introduced first.

[0038] First, the application scenarios of the embodiments of this application will be introduced.

[0039] In the development of autonomous driving, the iteration of visual algorithms requires a large amount of labeled data to drive the rapid iteration of various algorithms. This data is usually collected from vehicles and undergoes a series of data cleaning operations, including desensitization, parsing, frame extraction, scene labeling, and manual cleaning, to obtain labeled data. During the data cleaning process, parsing and scene labeling devices can clean up about 30% of the valid data. Manual intervention is also used to clean similar images, especially data recorded in traffic light and traffic congestion scenarios, to ensure the validity and diversity of the labeled data. With the exponential increase in the amount of data collected from vehicles, data cleaning requires a significant investment of manpower to ensure a stable data source for labeling (i.e., data to be labeled). Therefore, how to quickly deduplicate similar data in massive amounts of images, accelerate data flow, and reduce labor costs has become an urgent technical problem to be solved.

[0040] In related technologies, during the deduplication process of multiple images by a management device, the device first acquires multiple images and determines the similarity between pairs of images using offset and hash algorithms. Then, based on the similarity between pairs of images, the management device filters out images with lower similarity to obtain the deduplicated images.

[0041] In summary, current technical solutions, such as offset algorithms and hash algorithms, improve the comparison time for single or duplicate images. However, for massive image scenarios, the time required for image comparison still increases exponentially. Furthermore, hash bucket mapping reduces the overhead of feature comparison but increases the overhead of mapping; the larger the image volume, the greater the mapping computation overhead. It is also more suitable for scenarios with duplicate images, but has significant errors when searching for similar images. Therefore, there is an urgent need for an efficient method for deduplicating similar images to achieve rapid cleaning of massive amounts of autonomous driving data, accelerate data flow, and shorten the algorithm iteration cycle.

[0042] To address the aforementioned problems, this application provides an image deduplication method applicable to scenarios involving deduplication of multiple images. The deduplication device processes the multiple images to be deduplicated using a preset compression algorithm, generating feature information for each image. In other words, the deduplication device compresses the image information of each image using the preset compression algorithm, generating unique, concise information that indicates the image's features. This reduces the computational load of subsequent clustering processing of multiple images by the deduplication device, improving the speed of clustering. Then, the deduplication device inputs the multiple images and the feature information of each image into a trained clustering model, obtaining at least one first cluster. The similarity between the feature information of the images in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. That is, the deduplication device performs unsupervised clustering operations using the trained clustering model. This improves the efficiency of clustering multiple images by the deduplication device and reduces the time required for subsequent deduplication processing of multiple images. Next, the deduplication device filters the images in each first cluster to obtain at least one second cluster, where the similarity between any two images in the second cluster is less than a second preset similarity threshold. In other words, the deduplication device can perform deduplication on multiple images by determining the similarity between pairs of images within each cluster, avoiding the need to determine the similarity between images from different clusters. This reduces the number of times the deduplication device needs to determine the similarity between pairs of images, reduces the time spent deduplicating multiple images, and improves the efficiency of deduplication on multiple images.

[0043] The implementation environment of the embodiments of this application is described below.

[0044] like Figure 1 The diagram illustrates a communication system provided in an embodiment of this application. The communication system includes: an acquisition device (such as terminal 101), an image deduplication device (such as server 102), and a management device (such as terminal 103). Server 102 can communicate with both terminal 101 and terminal 103 via wired / wireless communication.

[0045] Specifically, terminal 101 can acquire multiple images and send them to server 102. Next, server 102 can receive the multiple images from terminal 101 and generate feature information for each image. Then, server 102 clusters the multiple images based on the images and their feature information, obtaining at least one cluster. Next, server 102 can deduplicate the images within each cluster based on the similarity between images in each cluster, obtaining at least one deduplicated cluster, and sends this deduplicated cluster to terminal 103. Finally, terminal 103 can present the received clusters from server 102 to the staff.

[0046] It should be noted that, in the embodiments of this application, the server (such as server 102) can be a single physical server, or it can be a server cluster consisting of multiple servers. Alternatively, the server cluster can be a distributed cluster. Or, the server can be a cloud server. The embodiments of this application do not limit the specific implementation method of the server.

[0047] The terminal (such as terminal 101, terminal 103) can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, or other device with sending and receiving functions. This application does not impose any special restrictions on the specific form of the terminal. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting device.

[0048] In this embodiment, server 102 may be deployed with clustering model 104. The trained clustering model 104 includes the following model parameters: at least one cluster center, where each cluster center corresponds to an image category. The trained clustering model 104 can perform operations such as convolution, rectified linear units (ReLU) function, and pooling on the feature information of each of the received multiple images, thereby clustering the images whose feature information has a high similarity to the cluster center.

[0049] In one possible design, server 102 may include an auto-encoder, and clustering model 104 may be deployed on the auto-encoder in server 102.

[0050] It should be noted that, in the embodiments of this application, the clustering model 104 is a model constructed by the K-means algorithm.

[0051] Specifically, server 102 can input multiple images and the feature information of each image into the trained clustering model 104 to obtain at least one cluster.

[0052] For ease of understanding, the image deduplication method provided in this application will be described in detail below, taking the server as the execution subject and in conjunction with the accompanying drawings.

[0053] Figure 2 This is a flowchart illustrating an image deduplication method according to an exemplary embodiment, such as... Figure 2 As shown, the image deduplication method includes the following steps: S201-S204.

[0054] S201, The server acquires multiple target images.

[0055] Among them, multiple target images are multiple images to be deduplicated.

[0056] In other words, multiple target images may contain duplicate images or images with high similarity.

[0057] It should be noted that the embodiments of this application do not limit the method by which the server acquires multiple target images.

[0058] In one possible implementation, the server can receive multiple target images from the acquisition device.

[0059] In another possible implementation, the server can retrieve multiple target images from the stored images.

[0060] S202. The server processes each target image using a preset compression algorithm to generate feature information for each target image, thereby obtaining multiple feature information.

[0061] In this context, each target image corresponds to a feature information, which is unique, concise, and indicates the characteristics of the corresponding target image.

[0062] In other words, without distortion, the amount of information in the feature information is less than the amount of information in the corresponding target image.

[0063] For example, if the target image is 255175273491068, then the feature information of the target image can be 1011011.

[0064] In one possible implementation, the server can use concurrent processing technology in the cloud to process each target image through a preset compression algorithm, generating feature information for each target image to obtain multiple feature information.

[0065] It should be noted that, with the same server resource configuration (e.g., 8 central processing units (CPUs) and 32 gigabytes (GB)), the time to extract feature information from 100,000 images was 54 minutes and 50 seconds under non-concurrent conditions, and 22 minutes and 22 seconds under concurrent conditions. Therefore, by increasing resource configuration based on concurrent processing techniques, the speed of feature information extraction can be linearly improved.

[0066] In this embodiment, the preset compression algorithm can be any of the following algorithms: mean hash algorithm, perceptual hash algorithm, and difference hash algorithm.

[0067] In other words, the server can select different preset compression algorithms to generate image feature information according to requirements. This improves the operability of the image feature information generated by the server.

[0068] It should be noted that hash algorithms, as a compression function, possess irreversibility, unique output values, and unpredictability, effectively avoiding collisions. Specifically, the difference hash algorithm has a faster computation speed than the perceptual hash algorithm but slower than the mean hash algorithm. Furthermore, the difference hash algorithm has higher accuracy than the mean hash algorithm but lower accuracy than the perceptual hash algorithm. Therefore, in the embodiments of this application, the preset compression algorithm used by the server is typically the difference hash algorithm.

[0069] It should be noted that the process by which the server processes each target image using a preset compression algorithm to generate feature information for each target image in order to obtain multiple feature information can be referred to in the introduction of conventional techniques for scaling and mapping image information using hash algorithms (such as mean hash algorithm, perceptual hash algorithm and difference hash algorithm) to obtain hash values, which will not be elaborated here.

[0070] S203. The server inputs multiple target images and multiple feature information into the trained clustering model to obtain at least one first cluster.

[0071] The first cluster includes at least one target image, and each first cluster corresponds to a cluster center information. The similarity between the feature information of the target image in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold.

[0072] In one possible implementation, the model parameters of the trained clustering model may include at least one cluster center and a first preset similarity threshold, where one cluster center corresponds to one image type (i.e., one cluster center corresponds to one cluster). After the server inputs multiple target images and multiple feature information into the trained clustering model, the server can use the trained clustering model to determine the similarity between each cluster center and each feature, and classify the multiple target images according to the determined similarity between the multiple cluster center and feature information and the first similarity threshold, to obtain at least one first cluster.

[0073] In one possible design, the similarity between cluster center information and feature information is called feature similarity, and a first preset similarity threshold is the threshold corresponding to feature similarity. During the process of the server determining the similarity between cluster center information and feature information, the server can determine the feature similarity between cluster center information and feature information based on the ratio between the distance between cluster center information and feature information calculated using the K-means algorithm and a preset distance. The feature similarity between cluster center information and feature information can be expressed by Formula 1.

[0074]

[0075] in, D0 is used to indicate the similarity between the m-th cluster center information and the n-th feature information, and D0 is used to indicate the preset distance. m,n Used to indicate the distance between the m-th cluster center and the n-th feature information calculated by the K-means algorithm.

[0076] In other words, by calculating the distance between each image and the cluster center, and based on a set distance threshold, images within the distance threshold range are determined to be similar images.

[0077] For example, the multiple target images include: image A, image B, and image C. The model parameters of the trained clustering model include: cluster center information A, cluster center information B, and cluster center information C. Specifically, the distance between the feature information of image A and cluster center information A is 1.6, the distance between the feature information of image A and cluster center information B is 10, the distance between the feature information of image A and cluster center information C is 16, the distance between the feature information of image B and cluster center information A is 2, the distance between the feature information of image B and cluster center information B is 8, the distance between the feature information of image B and cluster center information C is 20, the distance between the feature information of image C and cluster center information A is 25, the distance between the feature information of image C and cluster center information B is 1, and the distance between the feature information of image C and cluster center information C is 20.

[0078] If the preset distance is 1, then the feature similarity between the feature information of image A and cluster center information A is 0.625, the feature similarity between the feature information of image A and cluster center information B is 0.1, the feature similarity between the feature information of image A and cluster center information C is 0.0625, the feature similarity between the feature information of image B and cluster center information A is 0.5, the feature similarity between the feature information of image B and cluster center information B is 0.125, the feature similarity between the feature information of image B and cluster center information C is 0.05, the feature similarity between the feature information of image C and cluster center information A is 0.04, the feature similarity between the feature information of image C and cluster center information B is 1, and the feature similarity between the feature information of image C and cluster center information C is 0.05.

[0079] If the first preset similarity threshold in the model parameters of the trained clustering model is 0.4, then at least one first cluster includes: cluster A corresponding to cluster center information A and cluster B corresponding to cluster center information B. Cluster A includes: image A and image B, and cluster B includes: image C.

[0080] In one possible implementation, during the process of the server inputting multiple target images and multiple feature information into the trained clustering model, the server can map the multiple feature information to a low-dimensional subspace, obtaining multiple mapped feature information. The feature information before mapping is high-dimensional feature information, and the feature information after mapping is low-dimensional feature information. Then, the server can input the multiple target images and multiple mapped feature information into the trained clustering model to obtain at least one first cluster.

[0081] S204. The server filters the target images in each first cluster to obtain at least one second cluster.

[0082] In the second cluster, the similarity between any two target images is less than the second preset similarity threshold.

[0083] In one possible implementation, during the process of the server filtering target images in each first cluster, the server can determine the similarity between any two target images in each first cluster, and filter out target images with similarity less than the second preset similarity threshold according to the second preset similarity threshold, thereby obtaining at least one second cluster.

[0084] In one possible design, during the process of the server determining the similarity between any two target images in each first cluster, the server can obtain a fourth cluster from at least one first cluster. The fourth cluster is any cluster within the at least one first cluster and may include a first image, a second image, and a third image. The first, second, and third images are all images from at least one target image within the fourth cluster, and the first, second, and third images are distinct from each other. Next, the server can determine the similarity between the first image and all target images in the fourth cluster except for the first image. Similarly, the server can determine the similarity between the second image and all target images in the fourth cluster except for the first and second images, and the server can also determine the similarity between the third image and all target images in the fourth cluster except for the first, second, and third images.

[0085] In other words, I represents the index of the image being compared, and J represents the index of the image being compared. In a single-class comparison, I is compared with images with an index greater than J. If I is equal to J, it means that the two images have the same index, that is, they are the same image, and there is no need to compare it with itself when deduplicating. If I is less than J, it means that the two images have been compared before, and there is no need to compare them again.

[0086] For example, such as Figure 3 As shown, the fourth cluster includes images A, B, C, and D. In determining the similarity between images A, B, C, and D, the server can use image A as a reference and sequentially determine the similarity between A and B, A and C, and A and D. Next, using image B as a reference, the server can sequentially determine the similarity between B and C, B and D. Finally, using image C as a reference, the server can determine the similarity between C and D.

[0087] In other words, during the process of determining the similarity between pairs of images, the server can record the images whose similarity has been determined, and then only determine the similarity between pairs of images that have not been recorded. In this way, the repeated determination of the similarity between two images can be avoided, the amount of computation required for determining the similarity between pairs of images by the server can be reduced, and the efficiency of determining the similarity between pairs of images can be improved.

[0088] In another possible design, during the process of the server determining the similarity between the first image and all target images in the fourth cluster other than the first image, if the server determines that the similarity between the first image and the second image is greater than or equal to a second preset similarity threshold, then the server deletes the second image from the fourth cluster. Afterwards, the server does not need to determine the similarity between the second image and all target images in the fourth cluster other than the first and second images.

[0089] In other words, during the process of similar image removal and comparison, if the entire set of images is set as α, the similarity result set of the compared images is β, the remaining unique data is α-β, and the remaining number of comparisons is α-J-β. The more similar images there are, the fewer the remaining number of comparisons. Let subsequent similar image sets be γ, δ, etc., and the number of comparisons for image I is α-J-β-γ-δ, with the number of comparisons decreasing linearly. β+γ+δ represents the set of similar images to be removed.

[0090] For example, in combination Figure 3 In the fourth cluster shown, during the process of the server determining the similarity between image A and images B, C, and D sequentially, using image A as a reference, if the server determines that the similarity between image A and image B is greater than (or equal to) a second preset similarity threshold, and the similarity between image A and image B and the similarity between image A and image C are both less than the second preset similarity threshold, then the server deletes image B from the fourth cluster. Afterwards, the server can determine the similarity between image C and image D using only image C as a reference, without needing to determine the similarity between image B and image C or image B and image D.

[0091] In other words, during the process of determining the similarity between two images, the server can delete images with high similarity. Afterward, the server does not need to determine the similarity between the deleted images and other images. This reduces the computational load for determining the similarity between pairs of images, improving the efficiency of this process.

[0092] It should be noted that, in the embodiments of this application, the similarity between target images can be feature similarity, or the similarity between target images can be image similarity.

[0093] In one possible implementation, the second preset similarity threshold can be a first threshold, which is a threshold corresponding to feature similarity. During the process of the server filtering target images in each first cluster to obtain at least one second cluster, the server can determine a first similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster. The first similarity set includes the feature similarity between the feature information of any two target images in the corresponding first cluster. Then, the server can filter target images in each first cluster based on the first threshold and the first similarity set corresponding to each first cluster to obtain at least one second cluster, where the feature similarity between the feature information of any two target images in the second cluster is less than the first threshold. Here, one first cluster corresponds to one second cluster, and the target images in the first cluster include the target images in the corresponding second cluster.

[0094] In one possible design, the server can determine the feature similarity between two target images based on the calculated Hamming distance between their feature information and the character length of the feature information.

[0095] The feature similarity between the feature information of two target images can be represented by Formula 2.

[0096]

[0097] in, H is used to indicate the similarity between the x-th feature and the y-th feature. x,y L0 is used to indicate the Hamming distance between the x-th feature and the y-th feature, and L0 is used to indicate the character length of the feature.

[0098] In other words, by comparing the fingerprints (i.e., feature information) of two images, the Hamming distance is calculated. This distance measures how many different 64-bit hash values ​​(i.e., feature information) are present. The smaller the number of different bits, the more similar the images are. Hamming distance: the number of steps required to transform one set of binary data into another; it measures the difference between two images. The smaller the Hamming distance, the higher the similarity. A Hamming distance of 0 means the two images are completely identical.

[0099] For example, at least one first cluster includes: cluster A, cluster B, and cluster C. Cluster A includes: image A (i.e., the target image), image B, and image C; cluster B includes: image D and image E; cluster C includes: image F; the feature information of image A is 100000; the feature information of image B is 101100; the feature information of image C is 101110; the feature information of image D is 011110; the feature information of image E is 001001; and the feature information of image F is 000101. The Hamming distance between the feature information of image A and the feature information of image B is 2, the Hamming distance between the feature information of image A and the feature information of image C is 3, the Hamming distance between the feature information of image B and the feature information of image C is 1, the Hamming distance between the feature information of image D and the feature information of image E is 4, the feature similarity between the feature information of image A and the feature information of image B is 0.67, the feature similarity between the feature information of image A and the feature information of image C is 0.5, the feature similarity between the feature information of image B and the feature information of image C is 0.83, and the feature similarity between the feature information of image D and the feature information of image E is 0.33. If the first threshold is 0.6, then at least one second cluster includes: cluster D, cluster E, and cluster F, wherein cluster D is cluster A after filtering the images within the cluster, and cluster D includes: image A and image C; cluster E is cluster B after filtering the images within the cluster, and cluster E includes: image D and image E; cluster F is cluster C after filtering the images within the cluster, and cluster F includes: image F.

[0100] Understandably, during the process of the server filtering target images in each first cluster to obtain at least one second cluster, the server can determine the feature similarity between any two target images in each first cluster based on the feature information of the target images in each first cluster. Then, the server can filter target images in each first cluster based on a first threshold and the feature similarity between the feature information of all target images in each first cluster to obtain at least one second cluster. In other words, the server can determine the similarity between pairs of images based on the feature information of each target image, and perform deduplication processing on multiple images based on the pairwise similarity. This reduces the computational load of determining the similarity between pairs of images and reduces the time spent on deduplication processing of multiple images.

[0101] In another possible implementation, the second preset similarity threshold can also be a second threshold, which is a threshold corresponding to image similarity. During the process of the server filtering target images in each first cluster to obtain at least one second cluster, the server can determine a fourth similarity set corresponding to each first cluster based on the target images in each first cluster. The fourth similarity set includes the image similarity between any two target images in the corresponding first cluster. Then, the server can filter target images in each first cluster based on the second threshold and the fourth similarity set corresponding to each first cluster to obtain at least one second cluster, where the image similarity between any two target images in the second cluster is less than the second threshold.

[0102] It should be noted that the process by which the server determines the fourth similarity set corresponding to each first cluster, and filters the target images in each first cluster according to the second threshold and the fourth similarity set corresponding to each first cluster to obtain at least one second cluster, can be referred to the above description of the process by which the server determines the first similarity set corresponding to each first cluster, and filters the target images in each first cluster according to the first threshold and the first similarity set corresponding to each first cluster to obtain at least one second cluster. It will not be elaborated here.

[0103] The technical solution provided by the above embodiments brings at least the following beneficial effects: The server can acquire multiple target images and process each target image using a preset compression algorithm to generate feature information for each target image, thereby obtaining multiple feature information. In other words, the server can compress the image information of each image using a preset compression algorithm to generate unique, concise information that indicates the image's features. This reduces the computational load of subsequent clustering processing of multiple images by the server, improving the speed of clustering processing. Then, the server can input multiple target images and multiple feature information into a trained clustering model to obtain at least one first cluster. The first cluster includes at least one target image, and each first cluster corresponds to a cluster center. The similarity between the feature information of the target images in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. That is, the server can complete unsupervised clustering operations using the trained clustering model. This improves the efficiency of clustering multiple images by the server and reduces the time spent on subsequent deduplication processing of multiple images. The server can then filter the target images within each first cluster to obtain at least one second cluster, where the similarity between any two target images within the second cluster is less than a second preset similarity threshold. In other words, the server can deduplicate multiple images by determining the similarity between pairs of images within each cluster, avoiding the need to determine the similarity between images from different clusters. This reduces the number of times the server needs to determine the similarity between pairs of images, decreases the time spent deduplicating multiple images, and improves the efficiency of deduplicating multiple images.

[0104] In some embodiments, the server stores a third preset similarity threshold, which is a threshold corresponding to feature similarity. If the second preset similarity threshold is the second threshold, then during the process of the server filtering target images in each first cluster to obtain at least one second cluster, the server can determine a second similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster and the cluster center information corresponding to each first cluster. The second similarity set includes the feature similarity between the feature information of the target images in the corresponding first cluster and the cluster center information corresponding to the corresponding first cluster. Then, the server can filter target images in each first cluster based on the third preset similarity threshold and the second similarity set corresponding to each first cluster to obtain at least one third cluster, where the feature similarity between the feature information of the target images in the third cluster and the cluster center information corresponding to the third cluster is less than the third preset similarity threshold.

[0105] Next, the server can determine a third similarity set corresponding to each third cluster based on the target images in each third cluster. This third similarity set includes the image similarity between any two target images within the corresponding third cluster. Then, the server can filter the target images in each third cluster based on a second threshold and the third similarity set corresponding to each third cluster, obtaining at least one second cluster. The feature similarity between the feature information of the target images in the second cluster and the cluster center information corresponding to the second cluster is less than a third preset similarity threshold, and the image similarity between any two target images in the second cluster is less than the second threshold. Specifically, one first cluster corresponds to one second cluster and one third cluster. Target images in the first cluster include target images in the corresponding third cluster, and target images in the third cluster include target images in the corresponding second cluster.

[0106] Optionally, if the second preset similarity threshold is the same as the first threshold, then after the server obtains at least one third cluster, the server can determine a fifth similarity set corresponding to each third cluster based on the feature information of the target images in each third cluster. The fifth similarity set includes the feature similarity between the feature information of any two target images in the corresponding third cluster. Next, the server can filter the target images in each third cluster based on the second threshold and the fifth similarity set corresponding to each third cluster to obtain at least one second cluster. The feature similarity between the feature information of the target images in the second cluster and the cluster center information corresponding to the second cluster is less than the third preset similarity threshold, and the feature similarity between the feature information of any two target images in the second cluster is less than the first threshold.

[0107] Understandably, during the process of the server filtering target images in each first cluster to obtain at least one second cluster, the server can determine the feature similarity between the feature information of any target image in each first cluster and the cluster center information corresponding to the first cluster, based on the feature information of the target images in each first cluster. Next, the server can filter target images in each first cluster based on a third preset similarity threshold and the feature similarity between the feature information of any target image in each first cluster and the cluster center information corresponding to the first cluster, obtaining at least one third cluster. Then, the server can determine the image similarity between any two target images in each third cluster based on the target images in each third cluster. Next, the server can filter target images in each third cluster based on a second threshold and the image similarity between all target images in each third cluster, obtaining at least one second cluster. Similarly, the server can determine the feature similarity between the feature information of any two target images in each third cluster based on the feature information of the target images in each third cluster. Next, the server can filter the target images in each third cluster based on the first threshold and the feature similarity between the feature information of all target images in each third cluster, thus obtaining at least one second cluster. In other words, the server can deduplicate multiple images not only by determining the similarity between pairs of images within a cluster, but also by determining the feature similarity between the feature information of an image within a cluster and the cluster center information corresponding to that cluster. This improves the accuracy and completeness of deduplication for multiple images.

[0108] In the embodiments of this application, such as Figure 4 As shown, during the server-side training of the clustering model, the server acquires a training set, which includes multiple sample images and multiple sample feature information, with one sample image corresponding to one sample feature information. The server then transforms (i.e., performs dimensionality reduction) on each sample feature information, mapping the high-dimensional feature information to a low-dimensional subspace, resulting in multiple low-dimensional sample feature information. Next, the server inputs these low-dimensional sample feature information into the clustering model to obtain the clustering results (i.e., clusters), and determines multiple image categories and the cluster center information corresponding to each image category based on the clustering results. The server then adjusts the model parameters of the clustering model based on the cluster center information corresponding to each image category. Finally, the server re-inputs the multiple low-dimensional sample feature information into the clustering model until the cluster center information in the model parameters converges, resulting in the trained clustering model.

[0109] It should be noted that the server can use 500 positive and negative samples of image fingerprints (i.e., the training set) as cold start data for the algorithm model.

[0110] The following describes the flowchart of the image deduplication method provided in this application, with reference to specific embodiments. Figure 5 As shown, the image deduplication method provided in this application includes: step one, step two, step three, and step four.

[0111] Step 1: The server extracts the image fingerprint (i.e., the server executes S202).

[0112] Step 2: The server performs unsupervised clustering on the image features (i.e., the server instructs S203).

[0113] Step 3: The server performs fingerprint similarity comparison on the clustering results (i.e., the server determines the similarity between any two target images in each first cluster).

[0114] Step 4: The server removes duplicate images (i.e., the server filters out target images with a similarity less than the second preset similarity threshold).

[0115] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the image deduplication device or electronic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] This application embodiment can, based on the above method, exemplarily divide an image deduplication device or electronic device into functional modules. For example, the image deduplication device or electronic device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0117] Figure 6 This is a block diagram illustrating an image deduplication apparatus according to an exemplary embodiment. (Refer to...) Figure 6The image deduplication device 600 includes an acquisition module 601 and a processing module 602.

[0118] The acquisition module 601 is used to acquire multiple target images. The processing module 602 is used to process each target image using a preset compression algorithm to generate feature information for each target image, thereby acquiring multiple feature information. The processing module 602 is also used to input the multiple target images and multiple feature information into a trained clustering model to obtain at least one first cluster. The first cluster includes at least one target image, and each first cluster corresponds to a cluster center. The similarity between the feature information of the target images in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. The processing module 602 is also used to filter the target images in each first cluster to obtain at least one second cluster. The similarity between any two target images in the second cluster is less than a second preset similarity threshold.

[0119] In one possible implementation, the second preset similarity threshold is the first threshold, and the similarity between target images is the feature similarity. Specifically, the processing module 602 is used to determine a first similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster. The first similarity set includes the feature similarity between the feature information of any two target images in the corresponding first cluster. The processing module 602 is also used to filter the target images in each first cluster based on the first threshold and the first similarity set corresponding to each first cluster, obtaining at least one second cluster, where the feature similarity between the feature information of any two target images in the second cluster is less than the first threshold.

[0120] In one possible implementation, the second preset similarity threshold is a second threshold, the similarity between target images is image similarity, and the similarity between feature information and cluster center information is feature similarity. Specifically, the processing module 602 is used to determine a second similarity set corresponding to each first cluster based on the feature information of the target images in each first cluster and the cluster center information corresponding to each first cluster. The second similarity set includes the feature similarity between the feature information of the target images in the corresponding first cluster and the cluster center information corresponding to the corresponding first cluster. The processing module 602 is also used to filter the target images in each first cluster based on a third preset similarity threshold and the second similarity set corresponding to each first cluster to obtain at least one third cluster. The feature similarity between the feature information of the target images in the third cluster and the cluster center information corresponding to the third cluster is less than the third preset similarity threshold. The processing module 602 is also used to determine a third similarity set corresponding to each third cluster based on the target images in each third cluster. The third similarity set includes the image similarity between any two target images in the corresponding third cluster. The aforementioned processing module 602 is further configured to filter the target images in each third cluster according to the second threshold and the third similarity set corresponding to each third cluster, to obtain at least one second cluster, wherein the feature similarity between the feature information of the target images in the second cluster and the cluster center information corresponding to the second cluster is less than the third preset similarity threshold, and the image similarity between any two target images in the second cluster is less than the second threshold.

[0121] In one possible implementation, the preset compression algorithm is any of the following algorithms: mean hash algorithm, perceptual hash algorithm, and difference hash algorithm.

[0122] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0123] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 includes, but is not limited to, a processor 701 and a memory 702.

[0124] The memory 702 described above is used to store the executable instructions of the processor 701. It is understood that the processor 701 is configured to execute instructions to implement the image deduplication method in the above embodiments.

[0125] It should be noted that those skilled in the art will understand that Figure 7The electronic device structure shown does not constitute a limitation on the electronic device; the electronic device may include, but is not limited to, other electronic devices. Figure 7 This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0126] Processor 701 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in memory 702, and by calling data stored in memory 702, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Processor 701 may include one or more processing units. Optionally, processor 701 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 701.

[0127] The memory 702 can be used to store software programs and various data. The memory 702 may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, application programs (such as processing units) required by at least one functional module, etc. Furthermore, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0128] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 including instructions, which can be executed by a processor 701 of an electronic device 700 to implement the image deduplication method in the above embodiments.

[0129] In actual implementation, Figure 6 The functions of the acquisition module 601 and the processing module 602 can both be provided by Figure 7 The processor 701 calls the computer program stored in the memory 702 to implement the process. The specific execution process can be found in the description of the image deduplication method in the previous embodiment, and will not be repeated here.

[0130] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0131] In an exemplary embodiment, this application also provides a vehicle, the vehicle including Figure 7The processor 701 shown is used to perform the image deduplication method described in the above embodiments.

[0132] In an exemplary embodiment, this application also provides a computer program product including one or more instructions, which can be executed by a processor of an electronic device to complete the image deduplication method described above.

[0133] It should be noted that when one or more instructions in the computer-readable storage medium or computer program product are executed by the processor of the electronic device, they implement the various processes of the above-described image deduplication method embodiments and achieve the same technical effect as the above-described image deduplication method. To avoid repetition, they will not be described again here.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0139] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image deduplication method, characterized by, The method includes: Acquire multiple target images; Each target image is processed by a preset compression algorithm to generate feature information for each target image, thereby obtaining multiple pieces of feature information; The plurality of target images and the plurality of feature information are input into the trained clustering model to obtain at least one first cluster. The first cluster includes at least one target image. Each first cluster corresponds to a cluster center information. The similarity between the feature information of the target image in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. The target images in each of the first clusters are filtered to obtain at least one second cluster, wherein the similarity between any two target images in the second cluster is less than a second preset similarity threshold. The step of filtering the target images in each of the first clusters to obtain at least one second cluster includes: When the second preset similarity threshold is the second threshold, the similarity between the target images is image similarity, and the similarity between the feature information and the cluster center information is feature similarity, a second similarity set corresponding to each first cluster is determined based on the feature information of the target image in each first cluster and the cluster center information corresponding to each first cluster. The second similarity set includes the feature similarity between the feature information of the target image in the corresponding first cluster and the cluster center information corresponding to the corresponding first cluster. Based on a third preset similarity threshold and the second similarity set corresponding to each first cluster, the target images in each first cluster are filtered to obtain at least one third cluster. The feature similarity between the feature information of the target image in the third cluster and the cluster center information corresponding to the third cluster is less than the third preset similarity threshold. Based on the target image in each third cluster, a third similarity set corresponding to each third cluster is determined, wherein the third similarity set includes the image similarity between any two target images in the corresponding third cluster; Based on the second threshold and the third similarity set corresponding to each third cluster, the target images in each third cluster are filtered to obtain at least one second cluster. The feature similarity between the feature information of the target image in the second cluster and the cluster center information corresponding to the second cluster is less than the third preset similarity threshold, and the image similarity between any two target images in the second cluster is less than the second threshold.

2. The method of claim 1, wherein, The preset compression algorithm is any of the following algorithms: mean hash algorithm, perceptual hash algorithm, and difference hash algorithm.

3. An image deduplication apparatus, characterized by, The device includes: The acquisition module is used to acquire multiple target images; The processing module is used to process each of the target images using a preset compression algorithm to generate feature information for each of the target images, thereby obtaining multiple pieces of feature information; The processing module is further configured to input the plurality of target images and the plurality of feature information into the trained clustering model to obtain at least one first cluster, wherein the first cluster includes at least one target image, one first cluster corresponds to one cluster center information, and the similarity between the feature information of the target image in the first cluster and the cluster center information corresponding to the first cluster is greater than a first preset similarity threshold. The processing module is further configured to filter the target images in each of the first clusters to obtain at least one second cluster, wherein the similarity between any two target images in the second cluster is less than a second preset similarity threshold. Wherein, when the second preset similarity threshold is the second threshold, the similarity between the target images is image similarity, and the similarity between the feature information and the cluster center information is feature similarity, the processing module is specifically used to determine a second similarity set corresponding to each first cluster based on the feature information of the target image in each first cluster and the cluster center information corresponding to each first cluster. The second similarity set includes the feature similarity between the feature information of the target image in the corresponding first cluster and the cluster center information corresponding to the corresponding first cluster. The processing module is further configured to filter the target image in each first cluster according to a third preset similarity threshold and the second similarity set corresponding to each first cluster, to obtain at least one third cluster, wherein the feature similarity between the feature information of the target image in the third cluster and the cluster center information corresponding to the third cluster is less than the third preset similarity threshold. The processing module is further configured to determine a third similarity set corresponding to each third cluster based on the target images in each third cluster, wherein the third similarity set includes the image similarity between any two target images in the corresponding third cluster; The processing module is further configured to filter the target images in each third cluster according to the second threshold and the third similarity set corresponding to each third cluster to obtain at least one second cluster, wherein the feature similarity between the feature information of the target image in the second cluster and the cluster center information corresponding to the second cluster is less than the third preset similarity threshold, and the image similarity between any two target images in the second cluster is less than the second threshold.

4. The apparatus of claim 3, wherein, The preset compression algorithm is any of the following algorithms: mean hash algorithm, perceptual hash algorithm, and difference hash algorithm.

5. An electronic device, comprising: include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that, When the computer-executable instructions stored in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is capable of performing the method as described in any one of claims 1 to 2.

Images