An image deduplication method, apparatus, electronic device, and storage medium

Abstract

This application discloses an image deduplication method, apparatus, electronic device, and storage medium. It can extract image features from the original image; calculate pairwise similarity between image features; cluster images based on a preset similarity threshold and the similarity between image features to obtain image categories; obtain the heatmap and coefficient of variation corresponding to the original image; divide the heatmaps under the same image category into at least one coefficient of variation interval; select a representative heatmap from the heatmaps of each coefficient of variation interval to determine the deduplicated image. The image features provide good distinguishability between image categories, which is beneficial for ensuring the clustering effect. The interval division can group similar images into the same interval and images with large differences into different intervals, separating images with different data characteristics. This helps ensure that the deduplicated image retains the effective information of the original image and maximizes the data value of the original data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image deduplication method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, image deduplication often involves first clustering the images, and then selecting a small number of images from each cluster to complete the deduplication. This deduplication method has limited effectiveness; the resulting images may lose a significant amount of valuable information from the original images, resulting in a substantial difference in data value compared to the original images. Summary of the Invention

[0003] This application provides an image deduplication method, apparatus, electronic device, and storage medium, which helps to retain more effective information of the original image after deduplication, improve the deduplication effect, and reduce the loss of value of the image after deduplication.

[0004] In a first aspect, embodiments of this application provide an image deduplication method, including:

[0005] Feature extraction is performed on multiple original images to be deduplicated to obtain the image features of the original images;

[0006] The similarity between each pair of the image features is calculated to obtain the similarity between the image features;

[0007] Image clustering is performed based on a preset similarity threshold and the similarity between the image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category;

[0008] Obtain the heatmap corresponding to the original image, and the coefficient of variation corresponding to the heatmap;

[0009] The coefficient of variation of the heatmap corresponding to the original image under each image category is divided into coefficient of variation intervals to obtain at least one coefficient of variation interval under each image category.

[0010] Select a representative heatmap from the heatmaps of each coefficient of variation interval, and determine the deduplicated image based on the original image corresponding to the representative heatmap of each coefficient of variation interval.

[0011] Secondly, embodiments of this application also provide an image deduplication device, comprising:

[0012] The feature extraction unit is used to extract features from multiple original images to be deduplicated, thereby obtaining the image features of the original images.

[0013] A similarity unit is used to calculate the similarity between each pair of the image features to obtain the similarity between the image features.

[0014] A classification unit is used to perform image clustering based on a preset similarity threshold and the similarity between the image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category;

[0015] An acquisition unit is used to acquire the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap;

[0016] The interval division unit is used to divide the coefficient of variation of the heatmap corresponding to the original image under each image category into a coefficient of variation interval, so as to obtain at least one coefficient of variation interval under each image category.

[0017] The representative selection unit is used to select a representative heatmap from the heatmaps of each coefficient of variation interval, and to determine the deduplicated image based on the original image corresponding to the representative heatmap of each coefficient of variation interval.

[0018] Thirdly, embodiments of this application also provide an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps of any of the image deduplication methods provided in embodiments of this application.

[0019] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the steps of any of the image deduplication methods provided in embodiments of this application.

[0020] Fifthly, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in any of the image deduplication methods provided in embodiments of this application.

[0021] In this embodiment, feature extraction is performed on multiple original images to be deduplicated to obtain image features of the original images; similarity calculation is performed on each pair of image features to obtain the similarity between the image features; image clustering is performed based on a preset similarity threshold and the obtained similarity to obtain at least one image category; thus, image clustering is based on the similarity between image features, and the image features make the image categories have good distinguishability. This embodiment can also obtain the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap; for the image categories obtained by clustering, the coefficient of variation interval is divided based on the coefficient of variation corresponding to the heatmap of the original image to obtain at least one coefficient of variation interval under each image category; a representative heatmap is selected from the heatmaps of each coefficient of variation interval, and the deduplicated image is determined based on the original image corresponding to the representative heatmap of each coefficient of variation interval. In the steps after clustering, the interval division can group similar images together as much as possible, and divide images with large differences into different intervals, separating images with different data characteristics. This helps to ensure that the representative image selected from each interval retains the effective information of the original image. While ensuring the amount of deduplication, the deduplicated image can replace the original image to play the data value. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic flowchart of one embodiment of the image deduplication method provided in this application.

[0024] Figure 2 This is a schematic flowchart of one embodiment of the image deduplication method provided in this application.

[0025] Figure 3 This is a schematic flowchart of an embodiment of the image deduplication method based on asynchronous calls provided in this application.

[0026] Figure 4 This is a schematic flowchart of one embodiment of the image deduplication method provided in this application.

[0027] Figure 5 This is a schematic flowchart of an embodiment of image clustering provided in this application.

[0028] Figure 6 This is a schematic flowchart of an embodiment of image clustering provided in this application.

[0029] Figure 7 This is a schematic flowchart of an embodiment of selecting an interval to represent a heatmap under the same image category provided in this application.

[0030] Figure 8 This is a schematic diagram of heatmaps for different anomaly coefficients in an embodiment of this application;

[0031] Figure 9 This is a schematic flowchart of an embodiment of selecting a representative heatmap for each anomaly coefficient interval provided in this application.

[0032] Figure 10 This is a schematic diagram of the process by which the upstream task obtains the heat map in an embodiment of this application;

[0033] Figure 11 This is a schematic diagram of the image deduplication device provided in the embodiments of this application;

[0034] Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. At the same time, in the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0036] This application provides an image deduplication method, apparatus, electronic device, and computer-readable storage medium.

[0037] Specifically, this embodiment will be described from the perspective of the image deduplication device as the executing entity. This image deduplication device can be integrated into an electronic device; that is, the image deduplication method of this embodiment can be executed by an electronic device. Optionally, the electronic device can be a terminal device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, touch screen, game console, or personal computer (PC), etc. Optionally, the electronic device can be a server. The server can be a standalone server, or a server network or server cluster, including but not limited to computers, network hosts, single network servers, multiple network server sets, or cloud servers composed of multiple servers. The cloud server consists of a large number of computers or network servers based on cloud computing.

[0038] In some embodiments, the image deduplication device can also be integrated into the terminal and server cluster to achieve image deduplication; the image deduplication includes, but is not limited to, deduplication of images local to the terminal, deduplication of images in the cloud album, etc.

[0039] For example, the image deduplication method provided in this application embodiment can be applied to an image deduplication system. This image deduplication system may include a terminal and a server. The terminal can be a device that includes both receiving and transmitting hardware, i.e., a device with receiving and transmitting hardware capable of performing bidirectional communication over a bidirectional communication link. The terminal and server can communicate bidirectionally via a network, and the server can be used to run deduplication algorithms, etc.

[0040] The terminal can transmit the image (or image identifier, etc.) that needs to be deduplicated to the server, and the server will execute the steps of the image deduplication method of this embodiment based on the received information to determine the deduplicated image in the original image.

[0041] The terminal outputs the deduplication result and a deduplication query message. The query message asks the user whether to delete any images from the original image other than the deduplicated ones. For the example where the deleted images are located locally on the terminal, if the terminal receives confirmation of the deduplication query message, it deletes the non-deduplicated images from the original image. For the example in a cloud photo album, if the server receives confirmation of the deduplication query message from the terminal, it deletes the images from the cloud photo album.

[0042] The following detailed description, in conjunction with the accompanying drawings, illustrates that the executing entity in this embodiment is an electronic device capable of invoking a deduplication algorithm. It should be noted that the order of description in the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the accompanying drawings.

[0043] This application primarily aims to remove duplicate data from massive image datasets, filtering out representative images that contain all the valid information from the original data. The solution significantly reduces the number of images to be processed, while the small number of deduplicated images can completely replace the large number of original images to meet subsequent image processing needs. While maximizing the value of the data, it reduces the cost of storing redundant data and minimizes the transmission of redundant data in low-bandwidth network environments, thereby improving the efficiency of data processing personnel.

[0044] Please refer to Figure 1 The specific process of this image deduplication method can be summarized in steps 101 to 106, where:

[0045] Step 101: Extract features from the multiple original images to be deduplicated to obtain the image features of the original images.

[0046] In this embodiment, the process of removing duplicates from the original image to obtain a deduplicated image can be regarded as a deduplication task. This deduplication task can be independent or it can be a task in a task chain, that is, the deduplication task can have upstream tasks and / or downstream tasks.

[0047] The image deduplication method of this embodiment can be applied to any image deduplication scenario, such as deduplicating images locally on a terminal, deduplicating images in the cloud (e.g., deduplicating images in a cloud album), or deduplicating images stored on a server. This embodiment does not impose any restrictions on the number, type, or source of the original images. In one example, the original images can be determined based on a deduplication request sent by the terminal requiring deduplication, or they can be determined from information received from or transmitted by an upstream task; this embodiment does not impose any restrictions on this.

[0048] To provide a comprehensive understanding of the entire plan, we will first use an example. Figure 2 The process of the image deduplication method in this embodiment will be briefly described.

[0049] The implementation of the scheme in this embodiment relies on a deduplication algorithm, such as... Figure 2 The deep deduplication algorithm is responsible for the specific data deduplication calculation. In an image dataset containing original images, the deep deduplication algorithm clusters the image features of the original images to obtain multiple image categories, and then selects a representative image from each category.

[0050] Taking local image deduplication on the terminal as an example, the image deduplication algorithm can be set in the terminal. The terminal can call the local image deduplication algorithm to execute steps 101-106, thereby determining the deduplicated image of the original image, which makes it easier for the user to decide whether to delete other images in the original image besides the deduplicated image. Alternatively, the terminal can send the original image to the server, and the server can call the image deduplication algorithm to determine the deduplicated image.

[0051] Taking image deduplication in cloud photo albums as an example, image deduplication algorithms, such as deep deduplication algorithms, can be set in the server. The terminal can send a deduplication request to the server for all or some of the original images in the cloud photo album, triggering the server to filter the deduplicated images based on the image deduplication algorithm and return the filtering results, so that the user can decide whether to delete other images in the original images other than the deduplicated images.

[0052] Taking the deduplication task as an example of a task chain, the image deduplication algorithm can select a representative image based on the output of the upstream task on the original image during the representative image selection stage. For example, the output of the upstream task can be the heat map of the original image in step 104, and the coefficient of variation corresponding to the heat map.

[0053] Optionally, algorithms for deduplication of massive amounts of data may not have a response time in seconds and may require asynchronous service calls. Therefore, see [link to relevant documentation]. Figure 2 In this embodiment, the deduplication can also be achieved using asynchronous engineering. The asynchronous service is responsible for accepting requests, parsing data input, and returning deduplication results.

[0054] In one example, the scheme of this embodiment further includes: obtaining a deduplication request for multiple original images, and sending a deduplication task identifier corresponding to the deduplication request to the sending end of the deduplication request. In this example, the step "extracting features from the multiple original images to be deduplicated to obtain the image features of the original images" includes: asynchronously calling a deduplication algorithm to extract features from the multiple original images to be deduplicated to obtain the image features of the original images. In this scheme, after determining the deduplicated image in step 106, it may further include: storing the correspondence between the deduplicated image and the deduplication task identifier in a preset message queue.

[0055] In this embodiment, the deduplication request includes image acquisition information that identifies multiple original images to be deduplicated. For example, in one example, the deduplication request may be sent by a client, and the image acquisition information may include the image storage address of the multiple original images to be deduplicated, or the storage address of that image storage address, such as the Elasticsearch (ES) address. In this embodiment, the multiple original images to be deduplicated can be obtained based on the ES storage address.

[0056] Optionally, in one example, the image acquisition information may also include the user identifier corresponding to the cloud album to be deduplicated. The server can determine the cloud album based on the user identifier, thereby determining the multiple original images to be deduplicated. Optionally, in one example, the image acquisition information may also include the image encoding of the multiple images to be deduplicated in the album, facilitating the server or terminal performing the deduplication to determine the original images to be deduplicated.

[0057] In this embodiment, a deduplication algorithm can be invoked through an asynchronous service interface to extract features from the multiple original images to be deduplicated, thereby obtaining the image features of the original images. For ease of understanding, the following will combine... Figure 3 The asynchronous service call in this embodiment will be described in detail.

[0058] See Figure 3 Upon receiving a deduplication request containing an Elasticsearch address from a client, the server transmits the Elasticsearch address to the deep deduplication algorithm via an asynchronous service interface. It receives the task ID corresponding to the deduplication request and sends it to the client. The asynchronous service then uses the deep deduplication algorithm to obtain the image address of the original image based on the Elasticsearch address, retrieves the original image based on the image address, performs image deduplication, and obtains the deduplication result (e.g., the deduplicated image). The deduplication result is then stored in relation to the task ID. Optionally, a pre-defined message queue is created on the server to store the mapping between the deduplication result and the task ID. The technology of this pre-defined message queue is not limited; for example, it could be a RabbitMQ queue (RabbitMQ is an open-source message broker and queue server). After obtaining the deduplication result, the deep deduplication algorithm can publish the mapping between the deduplication result and the task ID to the RabbitMQ queue for storage. In one example, the image identifier of the deduplicated image can be obtained, and the mapping between the image identifier and the task ID can be stored in the queue.

[0059] In one example, when deduplication is completed, the status of the deduplication task can also be obtained, such as deduplication successful or deduplication failed. The correspondence between the status information, the deduplication result, and the task ID is stored in a preset message queue.

[0060] Step 102: Calculate the similarity between each pair of the image features to obtain the similarity between the image features.

[0061] Optionally, when calculating the pairwise similarity between image features, the two image features for which similarity is calculated can be image features extracted from different original images; or, they can be image features from different original images as well as image features from the same original image (with a similarity of 1).

[0062] Step 103: Perform image clustering based on a preset similarity threshold and the similarity between the image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category.

[0063] The image deduplication algorithm in this embodiment includes two main steps: 1) image clustering and 2) representative image selection. For example, the flowchart of an image deduplication method example can be found here. Figure 4 The deep deduplication algorithm performs deep feature extraction and calculates similarity to classify the data. The representative image selection algorithm selects at least one image representative from each image category based on statistical features. Optionally, the image clustering and representative image selection algorithms can be independent entities, and the two algorithms can serve as separate algorithms for specific tasks as needed.

[0064] 1) Clustering

[0065] Clustering is a typical task in unsupervised learning. It groups similar samples into the same category and dissimilar samples into different categories based on feature measurement criteria. In this embodiment, the clustering algorithm can use any feasible feature extraction scheme to extract features from the original image. In an optional example, the clustering in this embodiment can be based on deep learning. Deep learning-based clustering algorithms use deep networks to extract high-dimensional features into low-dimensional features, and then divide the data into appropriate categories based on similarity.

[0066] Compared to traditional clustering, which is directly calculated on the original data, deep clustering can: (1) extract effective image features: Deep neural networks can remove redundant feature dimensions and convert the data into a "cluster-friendly" representation, that is, the learned latent features (non-linear) are more suitable for clustering application scenarios. (2) improve the dimensionality curse caused by high-dimensional data: The dimension of the extracted representation is often much smaller than the dimension of the original data, and batch processing makes deep clustering algorithms more suitable for large-scale high-dimensional datasets. According to the optimization method, deep clustering methods can be divided into: independent optimization, step-by-step optimization and joint optimization. Independent optimization refers to the deep network acting only as a feature extractor to provide input for subsequent traditional clustering algorithms. Step-by-step optimization refers to the alternating optimization of parameters by using network backpropagation and traditional clustering closed-form solutions. Joint optimization refers to the joint optimization containing design network loss and clustering constraint loss, with all parameters updated by network backpropagation, which is consistent with the optimization method of familiar deep learning algorithms. The main difference between different deep clustering algorithms lies in the deep network architecture and clustering method. The loss function usually includes: network loss and clustering loss. Network loss refers to the loss function based on the deep network structure, such as the reconstruction loss of the autoencoder. Clustering loss is a regularization term that constrains the data to have discriminative power, such as agglomerative clustering loss and sparse subspace clustering loss.

[0067] Optionally, deep clustering can be divided into two main steps: deep network feature extraction and feature clustering. An example flow is as follows: Figure 5 As shown, deep features can be extracted from the original images in the image dataset based on neural networks to obtain image features. Then, the similarity between image features is calculated, and then image clustering is performed based on the similarity. The two main steps of deep clustering are described below: feature extraction and feature clustering.

[0068] A) Feature Extraction

[0069] In this embodiment, step 101 is the feature extraction step in the clustering scheme. In this step, the algorithm can use any neural network capable of extracting deep features to extract image features. Optionally, the network structure used to extract deep features includes, but is not limited to, autoencoders (AE), deep subspace clustering networks (DSCN), deep spectral clustering networks (SpectralNet), generative adversarial networks (GAN), graph convolutional networks (GCN), etc.

[0070] Optionally, the step "extracting features from multiple original images to be deduplicated, thereby obtaining image features of the original images" may include:

[0071] The original images to be deduplicated are preprocessed separately to obtain input features;

[0072] The input features are extracted using a pre-defined network model to obtain the image features of the original image. The number of dimensions of the image features is lower than that of the input features.

[0073] Therefore, extracting high-dimensional features of an image into low-dimensional features removes redundant features, which helps reduce the amount of data processing, improve deduplication efficiency, reduce resource consumption, and also improve the bit-curse of high-dimensional data.

[0074] Optionally, in this embodiment, the preset network model can be a pre-trained model with feature extraction capabilities. By using a pre-trained model to extract image features, high-dimensional features can be directly extracted from the original data without the need for training. Therefore, the algorithm is applicable to any image data and has good generalization performance.

[0075] Optionally, the type of pre-trained model is not limited and can be determined according to the characteristics of the image to be processed. For example, it can include VGG-19 model, ResNet series networks, BERT model which is good at processing text data, etc.

[0076] For example, this application can use the output of the 21-layer network structure of VGG-19, that is, the output of the ninth convolution layer. The original image is preprocessed and reconstructed into 3×512×256-dimensional input features. The input features are then convolved by a deep network to obtain 512×16×8-dimensional (32x downsampling) output features.

[0077] B) Feature Clustering

[0078] In this embodiment, steps 102 and 103 belong to the feature clustering steps. The clustering scheme used in feature clustering can be any feasible scheme, and this application does not limit it.

[0079] The effectiveness of clustering algorithms primarily relies on similarity metrics. Common similarity metrics include distance, kernel functions, and similarity coefficients. Clustering algorithms divide all data instances into suitable groups based on metric values, aiming to maximize homogeneity within the same group and maximize heterogeneity between different groups. Major clustering algorithms can be broadly categorized into hierarchical methods, model-based algorithms, partitioning methods, and density-based algorithms, including but not limited to k-means, NMF (non-negative matrix factorization), and spectral clustering.

[0080] Among partitioning methods based on minimizing the objective function, the concise and efficient k-means clustering is the most widely used and studied. This embodiment can also use the k-means algorithm for clustering; for example, if the preset number of image clusters is k, and the original image dataset D = {x} 1， x2, ..., x n The image features in D contain n original images and are located in p-dimensional space. The k-means algorithm assigns a cluster to each image feature in D, where the cluster is represented by the centroids of all image features within the group. The optimization objective of the algorithm is to minimize the squared error between the image features and the image features corresponding to the centroids within the group. Specifically, the step "clustering images based on a preset similarity threshold and the similarity between the image features to obtain at least one image category" may include:

[0081] Randomly select k image features from the image features as the initial centroids of k clusters;

[0082] Calculate the similarity between all image features and each centroid (corresponding image feature), where the similarity includes, but is not limited to, Euclidean distance;

[0083] The image features are assigned to the cluster corresponding to the nearest centroid;

[0084] Use the average value of all image features within the cluster as the new centroid of the group;

[0085] The objective function is calculated based on the image features in each cluster. If the difference between the objective function value and the value after the previous iteration is less than the threshold, or if the centroid of the cluster no longer moves, the algorithm converges and each cluster becomes an image category; otherwise, the algorithm returns to the step "Calculate the similarity between all image features and each centroid (corresponding image feature)".

[0086] The k-means clustering algorithm is sensitive to the initial value of the centroid, making it unsuitable for non-convex or imbalanced datasets, and it is also sensitive to outliers. In real-world scenarios, diverse image data may result in datasets that do not meet the requirements, leading to poor clustering results. Furthermore, the algorithm requires specifying the number of clusters, which limits its flexibility. To address these drawbacks, this embodiment proposes a new clustering method where the number of clusters is adaptively determined by the similarity between data features, eliminating the need for prior knowledge of the actual number of clusters.

[0087] In related technologies, the clustering scheme employed might be: two images are directly classified into the same category if their feature similarity exceeds a threshold. The problem with this method is that image similarity is not transitive. For example, image A and image B have high similarity, image B and image C have high similarity, but images A and C are significantly different. Without restrictions, this similarity can propagate infinitely; for instance, image C also has high similarity to image D, image D also has high similarity to image E, and so on. This would result in A, B, C, D, E, and so on, all being classified into the same category, leading to accumulated data differences within each category and low classification quality.

[0088] In one example, the similarity between any two data points of the same type could be limited to a threshold. However, this clustering approach can lead to overly strict category division criteria and overly detailed grouping, resulting in ineffective data deduplication. In this embodiment, a "first-come, first-served" classification scheme can be adopted. This means that each original image's image features are grouped only once. After comparing similarity to classify the images, the image features do not participate in subsequent classifications, thus blocking transitivity at the source and preventing the accumulation of differences during subsequent comparisons. For example, image feature A and image feature B have high similarity, so A and B are grouped into one category. Image feature B and image feature C have high similarity, but B has already been classified, so C is classified separately. Since A was compared with B before C, B and A are grouped into the same category.

[0089] Optionally, in one example, "clustering images based on a preset similarity threshold and the similarity between image features to obtain at least one image category" may include:

[0090] The similarity scores corresponding to the same unclassified image feature in the image features are compared with a preset similarity threshold.

[0091] If among the multiple similarities corresponding to the same unclassified image feature, there is a target similarity not lower than the preset similarity threshold, the unclassified image features in the image features corresponding to the target similarity are classified into the same image category to obtain at least one image category.

[0092] Understandably, if the similarity between the unclassified image feature and other image features (or other unclassified image features) is lower than a preset similarity threshold, then the unclassified image feature is classified into an image category.

[0093] Optionally, the preset similarity threshold can be set according to actual needs, such as any value between 0.5 and 1.

[0094] Optionally, an unclassified image feature can be selected from the image features each time, and the similarity of this unclassified image feature with other image features can be compared with a preset similarity threshold. If there is a target similarity among these similarities that is not lower than the preset similarity threshold, the unclassified image features in the image features to which the target similarity belongs are classified into the same image category. Here, "other images" refers to other images in the image features besides the unclassified image feature.

[0095] Optionally, when comparing the similarity between the unclassified image feature and other image features with a preset similarity threshold, classified image features can also be excluded. For example, the similarity between the unclassified image feature and other unclassified image features can be compared with a preset similarity threshold.

[0096] In one example, step 102 may calculate a similarity matrix containing similarity scores, where each row and column corresponds one-to-one with an original image. For instance, if there are k original images, the similarity matrix is ​​a K*K matrix, where each row corresponds to one original image and records the similarity between that original image and the K original images, and each column corresponds to one original image and records the similarity between that original image and the K original images. This embodiment uses the feature similarity matrix for clustering and directly classifies images by setting a similarity threshold. The number of categories is adaptively determined by the similarity between image features, without needing to pre-set the actual number of categories. In this embodiment, image category division is achieved by traversing the similarity matrix. The similarity matrix can be traversed row-wise or column-wise; this embodiment does not restrict this.

[0097] Optionally, the step "comparing multiple similarities corresponding to the same unclassified image feature in the image features with a preset similarity threshold" may include:

[0098] The similarity matrix is ​​traversed by rows or columns. During each traversal, if the same image feature corresponding to the traversed row or column already has an image category, the next row or column is traversed. If the same image feature corresponding to the traversed row or column is not classified, the similarity in the traversed row or column is compared with a preset similarity threshold.

[0099] For ease of understanding, the following explanation will use row traversal as an example. The column traversal scheme can be derived by referring to the row traversal scheme.

[0100] Optionally, the first row of the similarity matrix is ​​traversed, and the similarity of the first row is compared with a preset similarity threshold to determine the target similarity in the first row that is not lower than the preset similarity threshold. The image features corresponding to the target similarity are classified into one category. The rows and columns corresponding to the classified image features are deleted from the similarity block matrix. The similarity of the next row of the new similarity matrix is ​​compared with the preset similarity threshold to determine the target similarity that is not lower than the preset similarity threshold. The image features corresponding to the target similarity are classified into one category. The rows and columns corresponding to the classified image features are deleted from the similarity block matrix. The process is repeated, and the process of comparing the similarity of the next row of the new similarity matrix with the preset similarity threshold is returned until the last row of the matrix is ​​traversed.

[0101] In one example, rows and columns corresponding to already classified image features may not be deleted. If the image features corresponding to a row of the matrix being traversed have already been classified, then that row of the matrix is ​​skipped and the next row of the matrix is ​​traversed.

[0102] For example, in this embodiment, the image features obtained in step 101 are flattened to obtain flattened image features. For instance, a 512×16×8 dimensional image feature is flattened to 65536 dimensions. Then, similarity is calculated between each pair of flattened image features to obtain a similarity matrix A. Each row of the similarity matrix A is traversed; if aij > a threshold, then the i-th image feature and the j-th image feature belong to the same category. Simultaneously, to prevent transitivity, image features whose categories have been determined will be deleted in subsequent traversal calculations, i.e., the corresponding rows and columns will be skipped.

[0103] Figure 6 The similarity matrix calculated from seven original images is given, where A0-A6 correspond to image features 0-6. This application is based on... Figure 6Here's an example of clustering. Starting with the first row A0, using 0.92 as the similarity threshold, only A01 has a similarity greater than 0.92, so image features 0 and 1 belong to the same class. Next, since image feature 1 has already been classified, the second row A1 is skipped, and the next row, A2, is examined. In row A2, the similarities of A23 and A25 are both not lower than 0.92, so image features 2, 3, and 5 are classified into one class. Since image feature 3 has already been classified, A3 is skipped, and row A4 is traversed. Image feature 4 belongs to its own class. Since image feature 5 has already been classified, A5 is skipped, and row A6 is traversed. When examining A6, although the similarity of A61 is higher than 0.92, image feature 1 has already been classified, so image feature 6 is grouped independently.

[0104] Without such restrictions, image features 0, 1, and 6 would be grouped together. However, the similarity between features 0 and 6 would only be 0.906. With a massive dataset, this transitivity would persist indefinitely, leading to error accumulation, increasing the differences between similar samples, and ultimately resulting in extremely poor clustering. If the similarity between any two features is directly restricted to be above a threshold, image features 2, 3, and 5 would be assigned to different categories, increasing the number of categories and reducing the effectiveness of image deduplication. Therefore, the blocking method in this application strikes a good balance between category quality and the number of categories, thereby improving overall efficiency.

[0105] Optionally, there are no restrictions on the method of calculating similarity in this embodiment, including but not limited to pre-similarity, Euclidean distance, etc.

[0106] Step 104: Obtain the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap.

[0107] Optionally, the heatmap and coefficient of variation corresponding to the original image can be output by the upstream task; or, as needed, feature extraction can be performed on the original image to obtain an image descriptor, and then a heatmap can be obtained based on the image descriptor mapping, followed by calculation of the anomaly coefficient on the heatmap.

[0108] Optionally, in one example, step 104 may include: obtaining a feature map of the original image; obtaining an image descriptor of the original image; mapping based on the image descriptor to obtain a heatmap of each original image; and calculating the coefficient of variation corresponding to each heatmap.

[0109] An image descriptor is a feature vector containing various information about an image. To visualize the local attention of descriptors within an image and to increase the numerical differences of features, we map them to a heatmap for subsequent calculations. Image descriptors are used to represent homogeneous features in an image or sub-image. SIFT (Scale Invariant Feature Transform), HOG (Histogram of Oriented Gradients), LBP (Local Binary Patterns), and their variants are the most effective and commonly used descriptors. Of course, the image descriptor in this embodiment can also include texture feature maps, normal feature maps, etc. Essentially, an image descriptor is a type of image feature, and therefore can also be extracted using deep networks. For example, an image descriptor can also be an anomaly score matrix output by a bug detection algorithm.

[0110] Optionally, the image descriptor in this embodiment can be determined according to actual needs, such as the characteristics of the image, the application scenario of the deduplicated image, or the needs of subsequent tasks. The key is that the selected feature map can represent the key information of the original image.

[0111] Optionally, if a downstream task exists, the image descriptor of the original image is matched with the image feature focus dimension of the original image by the downstream task of the current image deduplication task.

[0112] In one example, the step "obtain the image descriptor of the original image" may include:

[0113] Obtain the target application scenario identifier corresponding to the deduplication result of the original image, and determine the feature identifier to be extracted in the target application scenario based on the target application scenario identifier. The feature identifier to be extracted is used to indicate the target type of the image descriptor to be extracted from the original image.

[0114] Based on feature extraction identifiers, image descriptors of target types are extracted from the original image.

[0115] In another example, the step "obtain the image descriptor of the original image" may also include:

[0116] The downstream task identifier applied to the deduplication result of the original image is obtained. Based on the downstream task identifier, the image feature attention identifier of the downstream task on the original image is determined. The image feature attention identifier is used to indicate the target type of the image descriptor that the downstream task is interested in on the original image.

[0117] Based on image feature attention markers, image descriptors of target types are extracted from the original image.

[0118] In this embodiment, anomaly detection can be implemented using any feasible image anomaly detection algorithm, and there are no restrictions on this in this embodiment.

[0119] Step 105: Divide the coefficient of variation of the heatmap corresponding to the original image under each image category into coefficient of variation intervals to obtain at least one coefficient of variation interval under each image category.

[0120] Step 106: Select a representative heatmap from the heatmaps of each coefficient of variation interval, and determine the deduplicated image based on the original image corresponding to the representative heatmap of each coefficient of variation interval.

[0121] Steps 104-106 represent the steps for image selection. The following will combine... Figure 10 The steps for selecting representative images are explained in detail.

[0122] 2) Representative image selection

[0123] When designing a representative image selection algorithm, the key features of the representative data can be characterized based on the specific application scenario, and then the algorithm can be designed for calculation and selection. One possible selection method is to choose the image with the highest anomaly score within the same category, or to select randomly.

[0124] It is understandable that neither heatmaps calculated based on image descriptors nor those output by upstream tasks can achieve 100% anomaly detection accuracy. For example, with heatmaps, there will always be some labeling discrepancies between the heatmap output by the upstream task and the anomaly score. A simple selection strategy may lead to missed bugs, affecting the final bug detection rate. Therefore, this application can comprehensively analyze the statistical features and anomaly scores of the input heatmaps to perform a second round of partitioning for each image category, and then select several reasonable representative heatmaps from these. This ensures the retention of key representative images for all bugs, reducing repetitive work for inspectors while preventing missed bugs. Figure 7 As shown, the representative selection algorithm includes the steps of segmenting the data to obtain multiple intervals and selecting anomaly representatives within each interval.

[0125] This application uses image bug detection as an example of an upstream task for illustration. See [link to documentation]. Figure 10 The upstream task takes the original image as input, which is then processed by a bug detection algorithm to obtain an anomaly score matrix. This matrix is ​​then mapped using a heatmap to create a visual heatmap. The upstream output includes the heatmap and the anomaly score (the score of the most abnormal part of the image). A heatmap is a statistical graph that uses color-coding to display data; darker, warmer colors represent larger values. The image bug detection algorithm outputs the anomaly score for each block of the image, which is then visualized as a heatmap. For example... Figure 10 The heatmap section represents the mapping result of the image bug detection prediction score. The anomaly score, which is the score of the lower right corner of the heatmap, is 50.

[0126] Optionally, in this embodiment, the coefficient of variation can be calculated based on the heatmap. The coefficient of variation is a normalized measure of the dispersion of a probability distribution, defined as the ratio of the standard deviation of the outlier scores to the mean. A smaller value indicates less data fluctuation and greater data stability. The coefficient of variation measures the spread of outliers in the image, and therefore its value can directly reflect the overall numerical differences in the image. Figure 8 The heatmaps shown have a coefficient of variation of 1.69 on the left and 4.57 on the right.

[0127] Optionally, when dividing the coefficient of variation into intervals, the coefficients of variation can be clustered to obtain at least one interval. Alternatively, the coefficients of variation corresponding to each image category can be divided into intervals based on a preset difference in coefficients of variation or the number of intervals.

[0128] In one example, when selecting an interval to represent a heatmap, it can be chosen randomly or more accurately based on anomaly coefficients.

[0129] Optionally, the step "selecting a representative heatmap from the heatmaps of each coefficient of variation interval" may include: selecting at least one representative heatmap from the heatmaps of each coefficient of variation interval based on the coefficient of variation of the heatmaps in each coefficient of variation interval.

[0130] In one example, "selecting a representative heatmap from the heatmaps of each coefficient of variation interval" may include: selecting at least one representative heatmap from the heatmaps of each coefficient of variation interval based on the mean anomaly score of the heatmaps in each coefficient of variation interval. In this embodiment, the mean anomaly score of the heatmap may be obtained by averaging the anomaly scores in the anomaly score matrix of the heatmap.

[0131] For example, a heatmap where the mean of outlier scores in each coefficient of variation interval is greater than the target mean of outlier scores can be selected as the representative heatmap for the interval. Alternatively, a preset number of target mean outlier scores with the highest mean of outlier scores in each coefficient of variation interval can be determined, and the heatmap corresponding to the target mean of outlier scores can be selected as the representative heatmap for the interval.

[0132] For example, based on the average outlier scores of the heatmaps within the same coefficient of variation interval, an interval score threshold can be determined for that interval. The interval score threshold is the product of the highest average outlier score within the same coefficient of variation interval and a preset multiple. Heatmaps with average outlier scores not lower than the corresponding interval score threshold in each coefficient of variation interval are selected as candidate heatmaps for that interval. The candidate heatmap with the smallest coefficient of variation in each interval is then selected as the representative heatmap for that interval. The preset multiple can be selected between 0 and 1.

[0133] For example, the most anomalous image is selected from each interval as the representative heatmap for that interval. The evaluation criteria for anomalous images are a high anomalous score and a large area covered by red (indicated by a high degree of red coverage). Therefore, the mean and coefficient of variation of the heatmap are calculated; the mean measures the anomalous score, and the coefficient of variation measures the degree of anomalous spread. The process for selecting the representative heatmap for an interval is as follows: Figure 9 As shown, features whose mean outlier score within an interval is higher than 0.95 times the maximum mean outlier score within the interval are placed into a candidate sequence. Then, the feature with the widest heatmap coverage in the candidate heatmap of the candidate sequence is selected as the representative, that is, the feature with the smallest coefficient of variation is selected as the representative heatmap of the interval.

[0134] Optionally, this embodiment also provides a scheme for obtaining deduplication results, such as receiving a deduplication result acquisition request containing a deduplication task identifier; based on the deduplication task identifier and the aforementioned preset message queue, acquiring the deduplicated image corresponding to the deduplication task identifier, and sending the deduplicated image to the sending end of the deduplication result acquisition request.

[0135] Optionally, the preset message queue may store the deduplicated image itself, or the identifier of the deduplicated image, storage address, and other information that can be used to obtain the deduplicated image. After receiving the deduplication task identifier, such as the task ID, the deduplicated image and status information can be obtained from the preset message queue based on the task ID.

[0136] In one example, after receiving a deduplication task identifier, the client can subscribe to that identifier in a preset message queue. Once the preset message queue stores information such as the deduplication result for that task, the client retrieves the deduplication result and status information, which are then sent back to the client. For example, see [link to example]. Figure 3 RabbitMQ queues are based on client subscriptions, and the server sends deduplication results, task IDs, and status information to the client.

[0137] The image deduplication method of this embodiment can extract image features from the original image; calculate the similarity between each pair of image features to obtain the similarity between the image features; perform image clustering based on a preset similarity threshold and the obtained similarity to obtain at least one image category; obtain the heatmap and coefficient of variation corresponding to the original image; divide the heatmap under the same image category into coefficient of variation intervals to obtain at least one coefficient of variation interval; select a representative heatmap from the heatmaps of each coefficient of variation interval to determine the deduplicated image. The image features enable good distinguishability between images during clustering, which is beneficial to ensuring the clustering effect. The interval division can divide similar images into the same interval and images with large differences into different intervals, which is beneficial to ensuring that the selected representative heatmap of the interval retains the effective information of the original image, maximizes the data value of the original data, and avoids the situation where the representative heatmaps of the intervals are highly similar, thus ensuring the deduplication effect.

[0138] Furthermore, deep clustering algorithms have no restrictions on input data, such as the size and distribution of the dataset. They do not require prior knowledge of the dataset, nor do they need data annotation. They can quickly and unsupervisedly cluster all data to obtain multiple image categories. The introduction of deep networks enables the algorithm to extract more discriminative image features from complex images, while the reduction in feature dimensionality also accelerates subsequent computations. At the same time, the algorithm does not require pre-training and is compatible with the original image format, thus it can be easily ported to any scenario and excellently complete clustering tasks.

[0139] This embodiment also provides an image deduplication device, which can be integrated into an electronic device, such as a computer device, which can be a terminal, server or other device.

[0140] For example, in this embodiment, the method of this application embodiment will be described in detail using a server specifically integrated with an image deduplication device as an example.

[0141] For example, such as Figure 11 As shown, the image deduplication device may include:

[0142] Feature extraction unit 1101 is used to extract features from multiple original images to be deduplicated, and obtain the image features of the original images;

[0143] Similarity unit 1102 is used to calculate the similarity between image features pairwise to obtain the similarity between image features;

[0144] The classification unit 1103 is used to perform image clustering based on a preset similarity threshold and the similarity between image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category;

[0145] The acquisition unit 1104 is used to acquire the heat map corresponding to the original image and the coefficient of variation corresponding to the heat map;

[0146] The interval division unit 1105 is used to divide the coefficient of variation intervals of the heat map corresponding to the original image under each image category, so as to obtain at least one coefficient of variation interval under each image category.

[0147] The representative selection unit 1106 is used to select a representative heatmap from the heatmaps of each coefficient of variation interval, and to determine the deduplicated image based on the original image corresponding to the representative heatmap of each coefficient of variation interval.

[0148] In one example, a classification unit is used to compare multiple similarities corresponding to the same unclassified image feature in the image features with a preset similarity threshold;

[0149] If among the multiple similarities corresponding to the same unclassified image feature, there is a target similarity not lower than a preset similarity threshold, the unclassified image features in the image features corresponding to the target similarity are classified into the same image category to obtain at least one image category.

[0150] In one example, the similarity between image features is recorded by a similarity matrix, with each row and column corresponding to a different original image. A classification unit is used to traverse the similarity matrix by row or column. During each traversal, if the same image feature corresponding to the traversed row or column already has an image category, the next row or column is traversed. If the same image feature corresponding to the traversed row or column is not classified, the similarity in the traversed row or column is compared with a preset similarity threshold.

[0151] In one example, the acquisition unit is used to acquire the image descriptor of the original image; to obtain the heatmap of each original image by mapping based on the image descriptor; and to calculate the coefficient of variation corresponding to each heatmap.

[0152] In one example, the image descriptor of the original image is matched with the image feature focus dimension of the original image by the downstream task of the current image deduplication task.

[0153] In one example, the acquisition unit is used to acquire the target application scenario identifier corresponding to the deduplication result of the original image, determine the feature identifier to be extracted in the target application scenario based on the target application scenario identifier, and the feature identifier to be extracted is used to indicate the target type of the image descriptor to be extracted from the original image; based on the feature extraction identifier, the image descriptor of the target type is extracted from the original image.

[0154] In one example, the acquisition unit is used to acquire the downstream task identifier applied to the deduplication result of the original image, determine the image feature attention identifier of the downstream task to the original image based on the downstream task identifier, the image feature attention identifier is used to indicate the target type of the image descriptor that the downstream task is interested in for the original image; and extract the image descriptor of the target type from the original image based on the image feature attention identifier.

[0155] In one example, the image deduplication device also includes an upstream task docking unit for receiving the heatmap output by the upstream task for the original image and the coefficient of variation corresponding to the heatmap.

[0156] In one example, the upstream task may also output only a heatmap, and the coefficient of variation can be calculated by the device in this embodiment based on the heatmap.

[0157] In one example, represents the selected unit, used to select at least one interval to represent the heatmap in each interval of the heatmap based on the coefficient of variation of the heatmap in each interval of the coefficient of variation.

[0158] In one example, the selected unit is used to determine the interval score threshold of the coefficient of variation interval based on the average abnormal score of the heatmaps in the same coefficient of variation interval. The interval score threshold is the product of the highest average abnormal score in the same coefficient of variation interval and a preset multiple. Heatmaps with average abnormal scores not lower than the corresponding interval score threshold in each coefficient of variation interval are selected as candidate heatmaps for the coefficient of variation interval. In each coefficient of variation interval, the candidate heatmap with the smallest coefficient of variation is selected as the representative heatmap of the interval.

[0159] In one example, the device further includes a receiving unit and a storage unit;

[0160] The receiving unit is used to obtain deduplication requests for multiple original images and send the deduplication task identifier corresponding to the deduplication request to the sending end of the deduplication request.

[0161] The feature extraction unit is used to asynchronously call the deduplication algorithm to extract features from multiple original images to be deduplicated, thereby obtaining the image features of the original images.

[0162] The storage unit is used to store the correspondence between the deduplicated images and the deduplication task identifiers into a preset message queue.

[0163] In one example, the receiving unit is also used to receive a deduplication result retrieval request containing a deduplication task identifier;

[0164] The device also includes a sending unit, which is used to obtain the deduplicated image corresponding to the deduplication task identifier based on the deduplication task identifier and a preset message queue, and send the deduplicated image to the sending end of the deduplication result acquisition request.

[0165] Using the apparatus in this application embodiment, image features can be extracted from the original image; similarity calculation can be performed between pairs of image features; image categories can be obtained by clustering based on a preset similarity threshold and the similarity between images; heatmaps and coefficients of variation corresponding to the original image can be obtained; the heatmaps under the same image category can be divided into coefficients of variation intervals to obtain at least one coefficient of variation interval; a representative heatmap of each interval can be selected from the heatmaps of each coefficient of variation interval to determine the deduplicated image. The image features enable good distinguishability between images during clustering, which is beneficial to ensuring the clustering effect. The interval division can divide similar images into the same interval and images with large differences into different intervals, which is beneficial to ensuring that the selected representative heatmap of the interval retains the effective information of the original image, maximizes the data value of the original data, and avoids the situation where the representative heatmaps of the intervals are highly similar, thus ensuring the deduplication effect.

[0166] Accordingly, this application also provides an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Alternatively, the electronic device can be a server.

[0167] like Figure 12 As shown, Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1200 includes a processor 1201 with one or more processing cores, a memory 1202 with one or more computer-readable storage media, and a computer program stored in the memory 1202 and executable on the processor. The processor 1201 and the memory 1202 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0168] The processor 1201 is the control center of the electronic device 1200. It connects various parts of the electronic device 1200 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 1202, and by calling data stored in the memory 1202, it executes various functions of the electronic device 1200 and processes data. The processor 1201 may be a CPU, GPU, network processor (NP), etc., and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0169] In this embodiment, the processor 1201 in the electronic device 1200 loads the instructions corresponding to the processes of one or more applications into the memory 1202 according to the following steps, and the processor 1201 runs the applications stored in the memory 1202 to realize various functions, such as:

[0170] Feature extraction is performed on multiple original images to be deduplicated to obtain the image features of the original images;

[0171] The similarity between image features is obtained by calculating the similarity between each pair of image features;

[0172] Image clustering is performed based on a preset similarity threshold and the similarity between image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category;

[0173] Obtain the heatmap corresponding to the original image, and the coefficient of variation corresponding to the heatmap;

[0174] Divide the coefficient of variation of the heatmap corresponding to the original image under each image category into a coefficient of variation interval to obtain at least one coefficient of variation interval under each image category.

[0175] Select a representative heatmap from the heatmaps of each coefficient of variation interval, and determine the deduplicated image based on the original image corresponding to the representative heatmap of each coefficient of variation interval.

[0176] In one example, image clustering is performed based on a preset similarity threshold and the similarity between image features to obtain at least one image category, including:

[0177] Compare multiple similarities corresponding to the same unclassified image feature in the image features with a preset similarity threshold;

[0178] If among the multiple similarities corresponding to the same unclassified image feature, there is a target similarity not lower than a preset similarity threshold, the unclassified image features in the image features corresponding to the target similarity are classified into the same image category to obtain at least one image category.

[0179] In one example, the similarity between image features is recorded using a similarity matrix, where the rows and columns of the similarity matrix correspond one-to-one with the original image.

[0180] The similarity scores of the same unclassified original image within the original image are compared with a preset similarity threshold, including:

[0181] The similarity matrix is ​​traversed by row or column. During each traversal, if the same image feature corresponding to the traversed row or column already has an image category, the next row or column is traversed. If the same image feature corresponding to the traversed row or column is not classified, the similarity in the traversed row or column is compared with a preset similarity threshold.

[0182] In one example, the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap are obtained, including:

[0183] Obtain the image descriptor of the original image;

[0184] Heatmaps of each original image are obtained by mapping based on image descriptors;

[0185] Calculate the coefficient of variation for each heatmap.

[0186] In one example, the feature map of the original image is obtained, including:

[0187] Obtain the target application scenario identifier corresponding to the deduplication result of the original image, and determine the feature identifier to be extracted in the target application scenario based on the target application scenario identifier. The feature identifier to be extracted is used to indicate the target type of the image descriptor to be extracted from the original image.

[0188] Based on feature extraction identifiers, image descriptors of target types are extracted from the original image.

[0189] In one example, the feature map of the original image is obtained, including:

[0190] The downstream task identifier applied to the deduplication result of the original image is obtained. Based on the downstream task identifier, the image feature attention identifier of the downstream task on the original image is determined. The image feature attention identifier is used to indicate the target type of the image descriptor that the downstream task is interested in on the original image.

[0191] Based on image feature attention markers, image descriptors of target types are extracted from the original image.

[0192] In one example, the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap are obtained, including:

[0193] Receive the heatmap and corresponding coefficient of variation output by the upstream task for the original image.

[0194] In one example, the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap are obtained, including:

[0195] Receive the heatmap output from the upstream task for the original image, and calculate the corresponding coefficient of variation based on the heatmap.

[0196] In one example, an interval is selected from the heatmaps representing the coefficient of variation intervals, including:

[0197] Based on the coefficient of variation of the heatmap in each coefficient of variation interval, at least one interval is selected to represent the heatmap in each coefficient of variation interval.

[0198] In one example, based on the coefficient of variation of the heatmap in each coefficient of variation interval, at least one interval is selected from the heatmaps of each coefficient of variation interval to represent the heatmap, including:

[0199] Based on the average outlier score of the heatmap in the same coefficient of variation interval, the interval score threshold of the coefficient of variation interval is determined. The interval score threshold is the product of the highest average outlier score in the same coefficient of variation interval and a preset multiple.

[0200] Heatmaps in which the mean of outlier scores in each coefficient of variation interval is not lower than the corresponding interval score threshold are selected as candidate heatmaps for the coefficient of variation interval.

[0201] In each range of coefficient of variation, the candidate heatmap with the smallest coefficient of variation is selected as the representative heatmap for that range.

[0202] In one example, the method also includes:

[0203] Obtain deduplication requests for multiple original images and send the deduplication task identifier corresponding to the deduplication request to the sending end of the deduplication request;

[0204] Feature extraction is performed on multiple original images to be deduplicated, resulting in image features of the original images, including:

[0205] The deduplication algorithm is asynchronously invoked to extract features from multiple original images to be deduplicated, thereby obtaining the image features of the original images.

[0206] The method also includes:

[0207] The correspondence between the deduplicated images and the deduplication task identifiers is stored in a preset message queue.

[0208] In one example, it also includes:

[0209] Receive a request to retrieve deduplication results, which includes a deduplication task identifier;

[0210] Based on the deduplication task identifier and the preset message queue, obtain the deduplicated image corresponding to the deduplication task identifier, and send the deduplicated image to the sender of the deduplication result retrieval request.

[0211] Therefore, the electronic device provided in this embodiment can bring the following technical effects: it can extract image features from the original image; perform pairwise similarity calculation based on the image features; it can cluster images to obtain image categories based on a preset similarity threshold and the similarity between image features; it can obtain the heatmap and coefficient of variation corresponding to the original image; it can divide the heatmap under the same image category into at least one coefficient of variation interval; it can select the interval representative heatmap from the heatmaps of each coefficient of variation interval to determine the deduplicated image. The image features make the images have good distinguishability during clustering, which is conducive to ensuring the clustering effect. The interval division can divide similar images into the same interval and images with large differences into different intervals, which is conducive to ensuring that the selected interval representative heatmap retains the effective information of the original image, maximizes the data value of the original data, and avoids the situation where the interval representative heatmaps are highly similar, thus ensuring the deduplication effect.

[0212] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0213] Optional, such as Figure 12As shown, the electronic device 1200 also includes: a touch display screen 1203, a radio frequency circuit 1204, an audio circuit 1205, an input unit 1206, and a power supply 1207. The processor 1201 is electrically connected to the touch display screen 1203, the radio frequency circuit 1204, the audio circuit 1205, the input unit 1206, and the power supply 1207. Those skilled in the art will understand that... Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0214] The touch display screen 1203 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 1203 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), and generate corresponding operation commands, which then execute the corresponding program. Optionally, the touch panel may include a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1201. It can also receive and execute commands from the processor 1201. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 1201 to determine the type of touch event. Subsequently, the processor 1201 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 1203 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 1203 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 1203 can also be used as part of the input unit 1206 to achieve input functions.

[0215] The radio frequency circuit 1204 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0216] Audio circuit 1205 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 1205 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 1205, converted back into audio data, and then processed by processor 1201 before being transmitted via radio frequency circuit 1204 to, for example, another electronic device, or output to memory 1202 for further processing. Audio circuit 1205 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0217] The input unit 1206 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0218] Power supply 1207 is used to supply power to various components of electronic device 1200. Optionally, power supply 1207 can be logically connected to processor 1201 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 1207 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0219] although Figure 12 As not shown in the diagram, the electronic device 1200 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0220] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0221] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0222] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute any of the image deduplication methods provided in embodiments of this application. For example, the computer program can execute the steps of the following image deduplication method:

[0223] Feature extraction is performed on multiple original images to be deduplicated to obtain the image features of the original images;

[0224] The similarity between image features is obtained by calculating the similarity between each pair of image features;

[0225] Image clustering is performed based on a preset similarity threshold and the similarity between image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category;

[0226] Obtain the heatmap corresponding to the original image, and the coefficient of variation corresponding to the heatmap;

[0227] Divide the coefficient of variation of the heatmap corresponding to the original image under each image category into a coefficient of variation interval to obtain at least one coefficient of variation interval under each image category.

[0228] Select a representative heatmap from the heatmaps of each coefficient of variation interval, and determine the deduplicated image based on the original image corresponding to the representative heatmap of each coefficient of variation interval.

[0229] In one example, image clustering is performed based on a preset similarity threshold and the similarity between image features to obtain at least one image category, including:

[0230] Compare multiple similarities corresponding to the same unclassified image feature in the image features with a preset similarity threshold;

[0231] If among the multiple similarities corresponding to the same unclassified image feature, there is a target similarity not lower than a preset similarity threshold, the unclassified image features in the image features corresponding to the target similarity are classified into the same image category to obtain at least one image category.

[0232] In one example, the similarity between image features is recorded using a similarity matrix, where the rows and columns of the similarity matrix correspond one-to-one with the original image.

[0233] The similarity scores of the same unclassified original image within the original image are compared with a preset similarity threshold, including:

[0234] The similarity matrix is ​​traversed by row or column. During each traversal, if the same image feature corresponding to the traversed row or column already has an image category, the next row or column is traversed. If the same image feature corresponding to the traversed row or column is not classified, the similarity in the traversed row or column is compared with a preset similarity threshold.

[0235] In one example, the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap are obtained, including:

[0236] Obtain the image descriptor of the original image;

[0237] Heatmaps of each original image are obtained by mapping based on image descriptors;

[0238] Calculate the coefficient of variation for each heatmap.

[0239] In one example, the feature map of the original image is obtained, including:

[0240] Obtain the target application scenario identifier corresponding to the deduplication result of the original image, and determine the feature identifier to be extracted in the target application scenario based on the target application scenario identifier. The feature identifier to be extracted is used to indicate the target type of the image descriptor to be extracted from the original image.

[0241] Based on feature extraction identifiers, image descriptors of target types are extracted from the original image.

[0242] In one example, the feature map of the original image is obtained, including:

[0243] The downstream task identifier applied to the deduplication result of the original image is obtained. Based on the downstream task identifier, the image feature attention identifier of the downstream task on the original image is determined. The image feature attention identifier is used to indicate the target type of the image descriptor that the downstream task is interested in on the original image.

[0244] Based on image feature attention markers, image descriptors of target types are extracted from the original image.

[0245] In one example, obtaining the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap includes: receiving the heatmap output by the upstream task for the original image and the coefficient of variation corresponding to the heatmap, or receiving the heatmap output by the upstream task for the original image and calculating the corresponding coefficient of variation based on the heatmap.

[0246] In one example, selecting a representative heatmap from the heatmaps of each coefficient of variation interval includes: selecting at least one representative heatmap from the heatmaps of each coefficient of variation interval based on the coefficient of variation of the heatmaps in each coefficient of variation interval.

[0247] In one example, based on the coefficient of variation of the heatmap in each coefficient of variation interval, at least one interval is selected from the heatmaps of each coefficient of variation interval to represent the heatmap, including:

[0248] Based on the average outlier score of the heatmap in the same coefficient of variation interval, the interval score threshold of the coefficient of variation interval is determined. The interval score threshold is the product of the highest average outlier score in the same coefficient of variation interval and a preset multiple.

[0249] Heatmaps in which the mean of outlier scores in each coefficient of variation interval is not lower than the corresponding interval score threshold are selected as candidate heatmaps for the coefficient of variation interval.

[0250] In each range of coefficient of variation, the candidate heatmap with the smallest coefficient of variation is selected as the representative heatmap for that range.

[0251] In one example, the method also includes: obtaining deduplication requests for multiple original images and sending the deduplication task identifier corresponding to the deduplication request to the sending end of the deduplication request;

[0252] Feature extraction is performed on multiple original images to be deduplicated, resulting in image features of the original images, including:

[0253] The deduplication algorithm is asynchronously invoked to extract features from multiple original images to be deduplicated, thereby obtaining the image features of the original images.

[0254] The method also includes storing the correspondence between the deduplicated images and the deduplication task identifiers in a preset message queue.

[0255] In one example, it also includes: receiving a deduplication result retrieval request containing a deduplication task identifier; obtaining the deduplicated image corresponding to the deduplication task identifier based on the deduplication task identifier and a preset message queue, and sending the deduplicated image to the sender of the deduplication result retrieval request.

[0256] As can be seen, the computer program can be loaded by the processor to execute any of the image deduplication methods provided in the embodiments of this application, thereby bringing the following technical effects: it can extract image features from the original image; perform pairwise similarity calculation based on the image features; cluster the images based on a preset similarity threshold and the similarity between image features to obtain image categories; obtain the heatmap and coefficient of variation corresponding to the original image; divide the heatmap under the same image category into at least one coefficient of variation interval; select the interval representative heatmap from the heatmaps of each coefficient of variation interval to determine the deduplicated image. The image features make the images have good distinguishability during clustering, which is conducive to ensuring the clustering effect and reducing the number of images. The interval division can divide similar data into the same interval and large differences into different intervals, separating images with different data characteristics. This is conducive to retaining at least one image of each data characteristic after deduplication, making the most of the data value of the original data and ensuring the deduplication effect.

[0257] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0258] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0259] Since the computer program stored in the computer-readable storage medium can execute any of the image deduplication methods provided in the embodiments of this application, the beneficial effects that any of the image deduplication methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0260] According to one aspect of this application, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0261] In the above embodiments of the image deduplication device, computer-readable storage medium, electronic device, and computer program product, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and beneficial effects of the image deduplication device, computer-readable storage medium, computer program product, electronic device, and their corresponding units described above can be referred to the description of the image deduplication method in the above embodiments, and will not be repeated here.

[0262] The foregoing has provided a detailed description of an image deduplication method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image deduplication method, characterized in that, include: Feature extraction is performed on multiple original images to be deduplicated to obtain the image features of the original images; The similarity between each pair of the image features is calculated to obtain the similarity between the image features; Image clustering is performed based on a preset similarity threshold and the similarity between the image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category; Obtain the heatmap corresponding to the original image, and the coefficient of variation corresponding to the heatmap; The coefficient of variation of the heatmap corresponding to the original image under each image category is divided into coefficient of variation intervals to obtain at least one coefficient of variation interval under each image category. Based on the average abnormal score of the heatmap in the same coefficient of variation interval, the interval score threshold of the coefficient of variation interval is determined. The interval score threshold is the product of the highest average abnormal score in the same coefficient of variation interval and a preset multiple. Heatmaps in which the mean of the outlier scores in each of the coefficient of variation intervals is not lower than the corresponding interval score threshold are selected as candidate heatmaps for the coefficient of variation intervals. In each of the aforementioned coefficient of variation intervals, the candidate heatmap with the smallest coefficient of variation is selected as the interval representative heatmap. Based on the original image corresponding to the interval representative heatmap of each coefficient of variation interval, the deduplicated image is determined.

2. The image deduplication method according to claim 1, characterized in that, The image clustering based on a preset similarity threshold and the similarity between the image features to obtain at least one image category includes: The similarity scores corresponding to the same unclassified image feature in the image features are compared with a preset similarity threshold. If among the multiple similarities corresponding to the same unclassified image feature, there is a target similarity not lower than the preset similarity threshold, the unclassified image features in the image features corresponding to the target similarity are classified into the same image category to obtain at least one image category.

3. The image deduplication method according to claim 2, characterized in that, The similarity between the image features is recorded by a similarity matrix, and the rows and columns of the similarity matrix correspond one-to-one with the original image. The step of comparing multiple similarities corresponding to the same unclassified image feature in the image features with a preset similarity threshold includes: The similarity matrix is ​​traversed by rows or columns. During each traversal, if the same image feature corresponding to the traversed row or column already has an image category, the next row or column is traversed. If the same image feature corresponding to the traversed row or column is not classified, the similarity in the traversed row or column is compared with a preset similarity threshold.

4. The image deduplication method according to claim 1, characterized in that, The step of obtaining the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap includes: Obtain the image descriptor of the original image; Heatmaps of each original image are obtained by mapping based on the image descriptor. Calculate the coefficient of variation for each of the aforementioned heatmaps.

5. The image deduplication method according to claim 1, characterized in that, The step of obtaining the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap includes: Receive the heatmap output by the upstream task terminal for the original image and the coefficient of variation corresponding to the heatmap.

6. The image deduplication method according to any one of claims 1-5, characterized in that, Also includes: Obtain a deduplication request for multiple original images, and send the deduplication task identifier corresponding to the deduplication request to the sending end of the deduplication request; The step of extracting features from multiple original images to be deduplicated, to obtain image features of the original images, includes: The deduplication algorithm is asynchronously invoked to extract features from the multiple original images to be deduplicated, thereby obtaining the image features of the original images; The method further includes: The correspondence between the deduplicated images and the deduplication task identifiers is stored in a preset message queue.

7. An image deduplication device, characterized in that, include: The feature extraction unit is used to extract features from multiple original images to be deduplicated, thereby obtaining the image features of the original images. A similarity unit is used to calculate the similarity between each pair of the image features to obtain the similarity between the image features. A classification unit is used to perform image clustering based on a preset similarity threshold and the similarity between the image features to obtain at least one image category, wherein the image features of the same original image exist in only one image category; An acquisition unit is used to acquire the heatmap corresponding to the original image and the coefficient of variation corresponding to the heatmap; The interval division unit is used to divide the coefficient of variation of the heatmap corresponding to the original image under each image category into a coefficient of variation interval, so as to obtain at least one coefficient of variation interval under each image category. The representative selection unit is used to determine the interval score threshold of the coefficient of variation interval based on the average abnormal score of the heatmaps in the same coefficient of variation interval. The interval score threshold is the product of the highest average abnormal score in the same coefficient of variation interval and a preset multiple. The heatmaps in each coefficient of variation interval whose average abnormal score is not lower than the corresponding interval score threshold are determined as candidate heatmaps of the coefficient of variation interval. The candidate heatmap with the smallest coefficient of variation in each coefficient of variation interval is selected as the interval representative heatmap. Based on the original image corresponding to the interval representative heatmap of each coefficient of variation interval, the deduplicated image is determined.

8. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps of the image deduplication method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image deduplication method according to any one of claims 1 to 6.

Images