Image clustering method, system and device and storage medium

Abstract

The invention provides an image clustering method, system and device and a storage medium. The method comprises the steps of obtaining feature vectors and image quality scores of a plurality of images; calculating a first distance between the feature vectors of every two images; according to image quality scores corresponding to the feature vectors, adjusting the first distance to obtain a second distance between the feature vectors; wherein when one of the image quality scores corresponding to the two feature vectors is larger than or equal to a first quality threshold value, the other one is smaller than the first quality threshold value, and a first distance between the two feature vectors is smaller than a first distance threshold value, the first distance between the two feature vectors is reduced, and a corresponding second distance is obtained; and clustering the image according to a second distance between the feature vectors to obtain a plurality of clusters. Through adoption of the method and the device, the clustering recall rate of low-quality samples is improved on the premise of ensuring the image clustering precision.

Description

Background Art

[0002] Image clustering refers to the technology of grouping a large number of images according to specific targets in the images. For example, when the specific target is a face, face clustering is used to group a large number of face images according to their potential identities. Face clustering usually uses the feature vector representation of the face to measure the similarity between faces and cluster similar face images together.

[0003] There are two main methods for face clustering: traditional clustering methods and learning-based methods. Most traditional clustering methods have assumptions about the distribution of data (for example, k-means requires that the data distribution shape is convex, etc.), but the data distribution in the real world is complex and often cannot fully meet these assumptions, thus affecting the accuracy of traditional clustering methods.

[0004] Learning-based methods first use a small portion of the data to train a clustering model in a supervised manner, and then test its performance on the rest of the data. Common learning-based clustering methods include CDP, LGCN, LTC, GCN (V + E), STAR-FC, Ada-NETS, etc. Among them, CDP integrates and aggregates the features extracted by different clustering models, but the computational cost of the integrated aggregation strategy is very high. L-GCN first uses graph convolutional networks (GCNs) to predict the links in the instance pivot subgraph, and then uses the connected components as clusters. Both LTC and GCN (V + E) use two-stage GCNs to cluster the entire K-NN (K-Nearest Neihbor, K nearest neighbor) graph. Specifically, LTC generates a series of subgraphs as proposals and detects face clusters on them, while GCN (V + E) learns confidence and connectivity through GCNs. In order to solve the inefficiency of GCNs, STAR-FC proposes a local graph learning strategy to simultaneously address the challenges of large-scale training and efficient reasoning. To address the noisy connectivity problem in the K-NN graph constructed in the feature space, Ada-NETS proposes an adaptive neighbor discovery strategy.

[0005] In summary, existing learning-based methods have achieved significant objective verification by focusing on developing deep models to learn better feature vectors to represent the similarity between two images. However, in existing application scenarios, some difficult samples are encountered when clustering images. Difficult samples refer to low-quality samples that exist when the monitoring equipment is shooting. The above methods do not take into account the treatment of difficult samples, and therefore fail to solve the clustering problem of difficult samples in complex monitoring scenarios.

[0006] In order to eliminate the influence of low-quality images on clustering accuracy, the prior art generally excludes low-quality images from clustering samples without clustering them. However, many useful samples are lost, which has an adverse effect on subsequent image analysis and processing based on clustered images. Therefore, the prior art fails to solve the clustering problem of low-quality images. Summary of the invention

[0007] In view of the problems in the prior art, the purpose of this application is to provide an image clustering method, system, device and storage medium, which improves the clustering recall rate of low-quality samples while ensuring the image clustering accuracy.

[0008] The present application provides an image clustering method, comprising the following steps:

[0009] Obtain feature vectors and image quality scores for multiple images;

[0010] Calculate the first distance between the feature vectors of every two images;

[0011] According to the image quality scores corresponding to the feature vectors, the first distance is adjusted to obtain a second distance between the feature vectors; wherein, when one of the image quality scores corresponding to the two feature vectors is greater than or equal to a first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, the first distance between the two feature vectors is reduced to obtain a corresponding second distance;

[0012] The images are clustered according to the second distance between the feature vectors to obtain a plurality of clusters.

[0013] In some embodiments, adjusting the first distance according to the image quality score corresponding to the feature vector to obtain the second distance between the feature vectors further includes:

[0014] When the image quality scores corresponding to the two feature vectors are both smaller than the first quality threshold, the first distance between the two feature vectors is increased to obtain a corresponding second distance.

[0015] In some embodiments, after calculating the first distance between the feature vectors of every two images, the method further includes: obtaining a first nearest neighbor vector set of each feature vector according to the first distance and the first nearest neighbor number;

[0016] Adjusting the first distance includes: adjusting a first distance between the feature vector and a first nearest neighbor vector in its first nearest neighbor vector set.

[0017] In some embodiments, clustering the images according to the second distance between the feature vectors comprises the following steps:

[0018] Obtaining a second nearest neighbor vector set for each feature vector according to the second distance and the second nearest neighbor quantity;

[0019] Calculating the probability that the feature vector and the second nearest neighbor vector in the second nearest neighbor vector set belong to the same target category, and obtaining a probability calculation result;

[0020] The images are clustered according to the probability calculation results, and images belonging to the same target category are clustered into the same cluster.

[0021] In some embodiments, after clustering the images according to the second distance between the feature vectors to obtain a plurality of clusters, the method further includes the following steps:

[0022] According to the number of images in each cluster, the clusters are divided into clusters and discrete clusters;

[0023] Based on the image quality scores of the images in the clusters, the images in the clusters are divided into high-quality images and low-quality images;

[0024] When the image quality score of the image in the discrete cluster is greater than or equal to the second quality threshold, and the similarity between the image in the discrete cluster and the high-quality image in a cluster is higher than the first similarity threshold, merging the discrete cluster with the cluster;

[0025] When the image quality score of the image in the discrete cluster is less than a second quality threshold, and the similarity between the image in the discrete cluster and the image in a cluster is higher than a second similarity threshold, the discrete cluster is merged with the cluster, and the second similarity threshold is less than the first similarity threshold.

[0026] In some embodiments, after clustering the image according to the second distance between the feature vectors to obtain a plurality of clusters, the method further includes the following step of merging the clusters in the clusters:

[0027] Based on the image quality scores of the images in the clusters, the images in the clusters are divided into high-quality images and low-quality images;

[0028] The clusters are sequentially used as designated clusters, and for a designated cluster, a third nearest neighbor vector set of high-quality images of the designated cluster is selected according to the number of third nearest neighbors;

[0029] If the image corresponding to the third nearest neighbor vector set is not in the specified cluster, and the similarity between the high-quality image in the cluster where the third nearest neighbor image is located and the high-quality image in the specified cluster is greater than a third similarity threshold, the cluster where the image corresponding to the third nearest neighbor is located is merged with the specified cluster.

[0030] In some embodiments, selecting the third nearest neighbor vector set of the high-quality images of the designated cluster according to the third nearest neighbor quantity comprises the following steps:

[0031] For the high-quality image of the specified cluster, sort the other images according to the second distance between the feature vectors, and select the third nearest neighbor vector set of the high-quality image of the specified cluster according to the order of the other images and the number of the third nearest neighbors; and / or,

[0032] For the high-quality images of the specified cluster, other images are sorted according to the probability that the feature vectors belong to the same target category, and the third nearest neighbor vector set of the high-quality images of the specified cluster is selected according to the order of other images and the number of third nearest neighbors.

[0033] The embodiment of the present application further provides an image clustering system, which is applied to the image clustering method, and the system includes:

[0034] A data acquisition module, used for acquiring feature vectors and image quality scores of multiple images;

[0035] A distance calculation module, used for calculating a first distance between feature vectors of every two images;

[0036] a distance adjustment module, configured to adjust the first distance according to the image quality scores corresponding to the feature vectors to obtain a second distance between the feature vectors; wherein, when one of the image quality scores corresponding to the two feature vectors is greater than or equal to the first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, the first distance between the two feature vectors is reduced to obtain a corresponding second distance;

[0037] The image clustering module is used to cluster the image according to the second distance between the feature vectors to obtain a plurality of clusters.

[0038] The present application also provides an image clustering device, including:

[0039] processor;

[0040] a memory storing executable instructions of the processor;

[0041] The processor is configured to perform the steps of the image clustering method by executing the executable instructions.

[0042] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the image clustering method are implemented.

[0043] By adopting this application, the beneficial effects obtained are as follows:

[0044] By adopting the image clustering method of the present application, when acquiring the feature vector of an image, the image quality score of the image is also acquired. After obtaining the first distance between every two feature vectors in the image to be clustered, the first distance is further adjusted according to the first distance and the image quality score. When the image quality score corresponding to one of the two feature vectors is low and the first distance between the two feature vectors is small, the first distance between the two feature vectors is further reduced to eliminate the influence of the low image quality score on the distance calculation of the two feature vectors, thereby facilitating subsequent more accurate image clustering based on distance, improving the accuracy of image clustering, and improving the recall rate of low-quality images. More image information can be provided in the image analysis scenario in which the clustered image is applied after clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0046] Figure 1 is a flow chart of an image clustering method according to an embodiment of the present application;

[0047] Figure 2 is a detailed flow chart of an image clustering method according to an embodiment of the present application;

[0048] Figure 3 is a flowchart of calculating the probability that images belong to the same target category according to an embodiment of the present application;

[0049] Figure 4 is a flow chart of merging discrete clusters and clustered clusters according to an embodiment of the present application;

[0050] Figure 5 is a flow chart of merging between clusters according to an embodiment of the present application;

[0051] Figure 6 is a schematic diagram of an image clustering system according to an embodiment of the present application;

[0052] Figure 7 is a structural schematic diagram of an image clustering device according to an embodiment of the present application;

[0053] Figure 8 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not used as a limitation on the number and importance of specific features.

[0055] In order to solve the defects of low-quality sample processing in the image clustering method in the prior art, the present application provides an image clustering method for clustering multiple images according to the target categories to which the targets contained therein belong, and obtaining multiple clusters, wherein the images in each cluster contain the same target category. Taking face image clustering as an example, each target category corresponds to a person, and the images in each cluster obtained by clustering are images containing the same person.

[0056] like Figure 1 As shown, in one embodiment, the image clustering method includes the following steps:

[0057] S100: Obtain feature vectors and image quality scores of multiple images;

[0058] Wherein, a plurality of images that need to be subjected to image clustering processing are respectively input into a trained feature extraction model to obtain a feature vector of each image, wherein the input of the feature extraction model is an image and the output is a feature vector representing the image features;

[0059] Input each image into a trained quality assessment model to obtain an image quality score for each image. The input of the quality assessment model is the image, and the output is the image quality score of the image.

[0060] S200: Calculate a first distance between feature vectors of every two images;

[0061] Here, the first distance may be the Euclidean distance, cosine distance, etc. between the feature vectors of the two images, which can represent the similarity between the two images. The larger the first distance between the feature vectors of the two images, the lower the similarity between the two images.

[0062] S300: adjusting the first distance according to the image quality score corresponding to the feature vector to obtain a second distance between the feature vectors;

[0063] Wherein, when one of the image quality scores corresponding to the two feature vectors is greater than or equal to the first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, the first distance between the two feature vectors is reduced to obtain a corresponding second distance;

[0064] Therefore, when one of the image quality scores corresponding to the two feature vectors is greater than or equal to the first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, since one of the feature vectors has a lower image quality score and has a higher similarity with the other feature vector, considering that the low quality factor of the image with the lower score may cause the calculated similarity with the other feature vector to be lower, it is necessary to reduce the first distance between the two feature vectors, that is, to increase the similarity between the two feature vectors, so as to be more conducive to subsequent more accurate clustering based on the distance;

[0065] S400: Clustering the images according to the second distance between the feature vectors to obtain multiple clusters. In this embodiment, clustering is performed according to the second distance, and images with a smaller second distance can be directly divided into a cluster. In another embodiment, clustering is performed according to the second distance. The nearest neighbor vector set corresponding to each feature vector (the set of the first k feature vectors with the smallest second distance) can be first screened according to the second distance, and then the probability that each feature vector in the nearest neighbor vector set belongs to the same target category as the feature vector is calculated. The images are clustered according to this probability, and images with a higher probability of belonging to the same target category are divided into the same cluster.

[0066] By adopting the image clustering method of the present application, when the feature vector of the image is obtained through step S100, the image quality score of the image is also obtained. After the first distance between every two feature vectors in the image to be clustered is obtained through step S200, the first distance is further adjusted according to the first distance and the image quality score through step S300. When the image quality score corresponding to one of the two feature vectors is low and the first distance between the two feature vectors is small, the first distance between the two feature vectors is further reduced to eliminate the influence of the low image quality score on the distance calculation of the two feature vectors, thereby facilitating more accurate image clustering according to the distance in the subsequent step S400, improving the accuracy of image clustering, and improving the recall rate of low-quality images. More image information can be provided in the image analysis scenario where the clustered image is applied after clustering.

[0067] Before executing the above step S100, it is necessary to first train the feature extraction model and the quality assessment model based on the sample image. Specifically, training the feature extraction model includes, for example, the following steps:

[0068] Prepare an image training set, which includes sample images in various scenarios and sample images of multiple target categories (for example, in face image clustering, prepare multiple face images of different people). The images corresponding to each target category are marked with the same ID. The sample images in the image training set can be collected images or images in a public set.

[0069] Perform target detection on sample images in the image training set to obtain target detection frames;

[0070] Based on the target detection frame, locate the key points of the target;

[0071] According to the position of the key points, the objects in the sample images in the image training set are normalized to the same size;

[0072] The normalized sample image is input into the feature extraction model for training, the network parameters in the feature extraction model are updated, and the feature extraction model is obtained. During training, a loss function is constructed based on the ID recognized by the model and the marked ID. The loss function is, for example, an ArcFace loss function.

[0073] Training a quality assessment model may include the following steps:

[0074] The normalized sample images are sampled according to the target category, with N sample images (N>1) for each target category. The sample images are evenly divided into three sample sets a, b, and c, and the image of the c sample set is expanded;

[0075] Input the sample images of sample sets a, b, and c into the feature extraction model respectively to obtain the image feature set fea a1 、fea a2 、fea b1 、fea b2 、fea c1 、fea c2 ; Here, it is only exemplified that there are two samples in each sample set. In actual applications, the number of sample images in each sample set can be set as needed;

[0076] The sample images of sample sets a, b, and c are input into the quality assessment model respectively to obtain the image quality score set qua a1 ,qua a2 ,qua b1 ,qua b2 ,qua c1 ,qua c2 ;

[0077] The image feature sets and quality score sets of sample sets a, b, and c are weighted fused one by one to obtain the fused feature set fea a 、fea b 、fea c ;

[0078] The classification loss and feature distribution distillation loss are calculated based on the fusion feature set of weighted fusion. a 、fea c The distance distribution is used to approximate fea a 、fea b The distance distribution of the feature extraction model is obtained by calculating the classification loss, the feature distribution distillation loss is obtained by calculating the DDL loss, and the quality assessment model is trained based on the classification loss and the feature distribution distillation loss. During the training process, the parameters of the feature extraction model are frozen, and the parameters of the quality assessment model are updated to obtain the quality assessment model of the training number.

[0079] The training methods of the feature extraction model and the quality assessment model here are only examples. In other embodiments, other methods may be used to train the feature extraction model and the quality assessment model. For example, a quality score label is added to the sample image, the feature vector of the sample image is input into the quality assessment model, and a loss function is constructed based on the quality score and label output by the quality assessment model to train the quality assessment model. The above-mentioned loss function types are also only examples. Other optional training methods and optional loss functions may also be applicable to the present application and are within the scope of protection of the present application.

[0080] In the above step S100, feature vectors and image quality scores of multiple images are obtained. The images can be directly input into a feature extraction model to obtain feature vectors, and the feature vectors are input into a quality assessment model to obtain image quality scores. Alternatively, target detection and key point detection are first performed on the images, and the images are normalized according to the key point detection. The images are then input into a feature extraction model to obtain feature vectors, and then the feature vectors are input into a quality assessment model to obtain image quality scores.

[0081] like Figure 2 As shown, in this embodiment, after the step S200: calculating the first distance between the feature vectors of every two images, the step further includes:

[0082] S201: Obtain the first nearest neighbor vector set of each feature vector according to the first distance and the first nearest neighbor number k1; the first nearest neighbor vector set includes the top k1 feature vectors with the smallest first distance to the feature vector, and each feature vector and its first nearest neighbor vector set form an initial feature sequence [x,x 1 ,x 2 ,...,x k1;

[0083] In step S300, adjusting the first distance includes: adjusting the first distance between the feature vector and the first nearest neighbor vector in the first nearest neighbor vector set of the feature vector.

[0084] In this embodiment, adjusting the first distance in step S300 specifically includes the following steps:

[0085] When the image quality scores corresponding to two feature vectors are both less than the first quality threshold, increase the first distance between the two feature vectors to obtain a corresponding second distance; for example, multiply the first distance by a coefficient c1, where c1 > 1; since the images corresponding to the two feature vectors are both low-quality images, the first distance calculated by these two feature vectors is not credible, and the distance value between them is enlarged to avoid having an adverse impact on the clustering of other images;

[0086] When one of the image quality scores corresponding to two feature vectors is greater than or equal to the first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, decrease the first distance between the two feature vectors to obtain a corresponding second distance; multiply the first distance by a coefficient c2, where 0 < c2 < 1; by decreasing the first distance between the two feature vectors, the distance difference caused by quality can be reduced, and the number of low-quality images participating in clustering can be increased, so as to improve the clustering recall rate of low-quality images while ensuring the clustering accuracy.

[0087] If the image quality scores of two feature vectors are both greater than or equal to the first quality threshold, the first distance between them remains unchanged. If one of the image quality scores of two feature vectors is greater than or equal to the first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is greater than or equal to the first distance threshold, the first distance between them also remains unchanged.

[0088] As Figure 2 shown, step S400: clustering the images according to the second distance between the feature vectors includes the following steps:

[0089] S410: obtaining the second nearest neighbor vector set of each feature vector according to the second distance and the second nearest neighbor number k2;

[0090] Specifically, sort the feature vectors in the first nearest neighbor vector set from small to large according to the second distance, and select the top k2 feature vectors to join the second nearest neighbor vector set; here, the second nearest neighbor number k2 is less than the first nearest neighbor number k2; the obtained feature vector and its second nearest neighbor vector set are represented as [x, x 1 , x 2 ,..., xk2 ];

[0091] S420: Calculate the probability that each feature vector and the second nearest neighbor vector in the second nearest neighbor vector set belong to the same target category, and obtain a probability calculation result;

[0092] Specifically, for each eigenvector, the probability calculation result is [p 1 ,p 2 ,...p k2 ], where p 1 That is, the probability that a feature vector and the first nearest neighbor vector in its second nearest neighbor vector set belong to the same target category, and so on. i That is, the probability that a feature vector and the i-th nearest neighbor vector in its second nearest neighbor vector set belong to the same target category, i∈[1,k2];

[0093] S430: clustering the images according to the probability calculation result, and clustering the images belonging to the same target category into the same cluster;

[0094] In this embodiment, the probability calculation result [p 1 ,p 2 ,...p k2 ], the feature vectors with probability less than the preset probability threshold are filtered out, and the remaining feature vectors and corresponding probabilities are input into the Infomap algorithm for solution, thereby obtaining the clustering result output by the Infomap algorithm. The Infomap algorithm is a community detection algorithm that clusters images according to target categories to obtain multiple clusters, each of which includes images belonging to the same target category.

[0095] like Figure 3 As shown, in this embodiment, step S420 includes the following steps:

[0096] S421: Each eigenvector and the second nearest neighbor vector in the second nearest neighbor vector set are combined to form an initial eigenvector sequence;

[0097] The initial feature vector sequence is expressed as [x,x 1 ,x 2 ,...,x k2 ];

[0098] S422: inputting the initial feature vector sequence into the trained feature conversion model to obtain a converted feature vector sequence output by the feature conversion model;

[0099] The transformed feature vector sequence is expressed as [x′,x 1 ′,x 2 ′,...,x k2 ′];

[0100] The feature conversion model is, for example, a convolutional neural network model, which is used to convert the initial feature vector sequence so that feature vectors belonging to the same target category can be closer, and feature vectors that do not belong to the same target category have lower similarity;

[0101] S423: Combining the converted feature vector sequences in pairs to obtain a combined feature vector sequence;

[0102] The combined feature vector sequence is expressed as [x′x 1 ′,x′x 1 ′,x′x 2 ′,...,x′x k2 ′];

[0103] S424: Input the combined feature vector sequence into the trained probability prediction model, calculate the probability that each feature vector and the second nearest neighbor vector in the second nearest neighbor vector set belong to the same target category, and obtain a probability calculation result;

[0104] The probability calculation result is expressed as [p 1 ,p 2 ,...p k2 ];

[0105] The probability prediction model is, for example, a multi-layer perceptron (MLP) model, which is used to predict the probability of belonging to the same target category based on a combined feature vector sequence.

[0106] The process of training the above feature conversion model and probability prediction model includes the following steps:

[0107] Collect sample images, obtain the second nearest neighbor vector set of the sample images, and then construct an initial feature vector sequence corresponding to the sample images, wherein each feature vector of the sample images is provided with a target category label;

[0108] The initial feature vector sequence corresponding to the sample image is input into the feature conversion model to obtain the converted feature vector sequence, and the converted feature vector sequence is obtained by pairwise combination;

[0109] The combined feature vector sequence corresponding to the sample image is input into the probability prediction model to obtain the probability calculation result, and a loss function (such as a cross entropy loss function) is constructed based on the probability calculation result and the target category label. The model parameters of the feature conversion model and the probability prediction model are trained based on the loss optimization to obtain the trained feature conversion model and the probability prediction model.

[0110] like Figure 2 As shown, after step S400, the step S500 may further include: merging discrete clusters and merging aggregated clusters. Figure 4As shown, step S500 includes the following steps:

[0111] S510: Dividing the clusters into clustered clusters and discrete clusters according to the number of images in each cluster;

[0112] A cluster is a cluster that contains more than one image, and a discrete cluster is a cluster that contains a single image.

[0113] S520: Based on the image quality scores of the images in the cluster, the images in the cluster are divided into high-quality images and low-quality images;

[0114] Specifically, a quality threshold is set, and images in the cluster whose quality scores are greater than or equal to the quality threshold are regarded as high-quality images, and images whose quality scores are less than the quality threshold are regarded as low-quality images. The quality threshold may be equal to the first quality threshold or may be different from the first quality threshold.

[0115] S530: When the image quality score of the image in the discrete cluster is greater than or equal to the second quality threshold, and the similarity between the image in the discrete cluster and the high-quality image in a cluster is higher than the first similarity threshold, merging the discrete cluster with the cluster;

[0116] The second quality threshold may be equal to the first quality threshold, or may be different from the first quality threshold;

[0117] S540: when the image quality score of the image in the discrete cluster is less than the second quality threshold, and the similarity between the image in the discrete cluster and the image in an aggregated cluster is higher than the second similarity threshold, merging the discrete cluster with the aggregated cluster, and the second similarity threshold is less than the first similarity threshold;

[0118] Specifically, when the image quality score of the image in the discrete cluster is less than the second quality threshold, firstly determine whether the similarity between the image in the discrete cluster and the high-quality image in the aggregated cluster is higher than the second similarity threshold, if yes, merge the discrete cluster with the aggregated cluster, if not, further determine whether the similarity between the image in the discrete cluster and the low-quality image in the aggregated cluster is higher than the second similarity threshold, if the similarity between the image in the discrete cluster and the low-quality image in the aggregated cluster is higher than the second similarity threshold, merge the discrete cluster with the aggregated cluster;

[0119] In this embodiment, the second similarity threshold is smaller than the first similarity threshold. When the image quality in the discrete cluster is high, the image in the discrete cluster needs to have a high similarity with the high-quality image in the clustered cluster in order to merge the discrete cluster with the clustered cluster. When the image quality in the discrete cluster is low, considering the adverse effect of low-quality images on the similarity, the requirement for the similarity between the images in the discrete cluster and the images in the clustered cluster is lowered. At this time, the discrete cluster and the clustered cluster can be merged if the similarity between the images in the discrete cluster and the images in the clustered cluster is higher than the lower second similarity threshold, thereby increasing the possibility of low-quality images being clustered into the clustered cluster and improving the clustering recall rate of low-quality images.

[0120] When judging whether the similarity between two images satisfies a similarity threshold, it can be set that the similarity between a discrete cluster and a high-quality image in a cluster satisfies the similarity threshold, or it can be set that the similarity between a discrete cluster and a plurality of high-quality images in a cluster satisfies the similarity threshold.

[0121] like Figure 2 As shown, after step S400, the step S600 may further include: a step of merging clusters. Figure 5 As shown, step S600 includes the following steps:

[0122] S610: Based on the image quality scores of the images in the clusters, the images in the clusters are divided into high-quality images and low-quality images;

[0123] Specifically, a quality threshold is set, and images in the cluster whose quality scores are greater than or equal to the quality threshold are regarded as high-quality images, and images whose quality scores are less than the quality threshold are regarded as low-quality images. The quality threshold may be equal to the first quality threshold or may be different from the first quality threshold.

[0124] S620: sequentially taking clusters as designated clusters, for a designated cluster, selecting a third nearest neighbor vector set of high-quality images of the designated cluster according to the third nearest neighbor number k3;

[0125] The third nearest neighbor vector set includes feature vectors of top k3 images closest to the high-quality image in the specified cluster;

[0126] S630: if the image corresponding to the third nearest neighbor vector set is not in the designated cluster, and the similarity between the high-quality image in the cluster where the third nearest neighbor image is located and the high-quality image in the designated cluster is greater than a third similarity threshold, merging the cluster where the image corresponding to the third nearest neighbor is located with the designated cluster;

[0127] Specifically, it is determined in turn whether the image corresponding to the feature vector in the third nearest neighbor vector set is in the specified cluster. If an image is not in the specified cluster, the similarity between the image and the high-quality image in the specified cluster is calculated. If the similarity is greater than the third similarity threshold, the two clusters are merged. Here, the third similarity threshold can be the same as or different from the first similarity threshold. If the image corresponding to the feature vector in the third nearest neighbor vector set is already in the specified cluster, no further processing is required.

[0128] Here, it is judged whether the similarity between the image and the high-quality images in the specified cluster is greater than the third similarity threshold. It can be required that the similarity between the image and a high-quality image in the specified cluster is greater than the third similarity threshold, or it can be required that the similarity between the image and multiple high-quality images in the specified cluster are greater than the third similarity threshold.

[0129] Wherein, in step S620, selecting the third nearest neighbor vector set of the high-quality image of the designated cluster according to the third nearest neighbor number includes the following two optional implementations:

[0130] (1) For the high-quality image of the specified cluster, sort the other images according to the second distance between the feature vectors, and select the third nearest neighbor vector set of the high-quality image of the specified cluster according to the order of the other images and the number of the third nearest neighbors;

[0131] Here, the second distance is the second distance between the feature vectors of the images obtained by the above step S300. According to the second distance between the high-quality image and other images in the specified cluster, the other images are sorted in order from small to large according to the distance, and the top k3 nearest neighbor feature vectors are selected from the front to the back to obtain the third nearest neighbor vector set.

[0132] (2) For the high-quality images of the specified cluster, sort the other images according to the probability that the feature vectors belong to the same target category, and select the third nearest neighbor vector set of the high-quality images of the specified cluster according to the order of the other images and the number of the third nearest neighbors;

[0133] The probability here is the probability that the two feature vectors obtained by the probability prediction model in the above step S424 belong to the same target category. The probabilities between the feature vectors of the high-quality image and the feature vectors of other images are obtained respectively, and the other images are sorted in descending order according to the probability, and the top k3 nearest neighbor feature vectors are selected from front to back to obtain the third nearest neighbor vector set.

[0134] When the step S600 is used to merge clusters, the above-mentioned implementation (1) or implementation (2) can be selected to merge clusters. Alternatively, the above-mentioned implementation (1) can be first used to merge clusters for the first time, and then the above-mentioned implementation (2) can be used to merge clusters for the second time, that is, the third nearest neighbor vector set (including the top k3 feature vectors with the smallest second distance) is first selected according to the second distance for the first merge, and whether to merge clusters is determined according to the similarity between the image in the third nearest neighbor vector set and the high-quality image in the specified cluster. Then, the third nearest neighbor vector set (including the top k3 feature vectors with the largest probability) is reselected for the second merge according to the probability, and whether to merge clusters is determined according to the similarity between the image in the third nearest neighbor vector set and the high-quality image in the specified cluster. Alternatively, the above-mentioned implementation (2) can be first used to merge clusters for the first time, and then the above-mentioned implementation (1) can be used to merge clusters for the second time. All of the above implementations fall within the protection scope of the present application.

[0135] like Figure 6 As shown, the embodiment of the present application also provides an image clustering system, which is applied to the image clustering method, and the system includes:

[0136] A data acquisition module M100 is used to acquire feature vectors and image quality scores of multiple images;

[0137] The distance calculation module M200 is used to calculate the first distance between the feature vectors of every two images;

[0138] The distance adjustment module M300 is used to adjust the first distance according to the image quality scores corresponding to the feature vectors to obtain the second distance between the feature vectors; wherein, when one of the image quality scores corresponding to the two feature vectors is greater than or equal to the first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, the first distance between the two feature vectors is reduced to obtain the corresponding second distance;

[0139] The image clustering module M400 is used to cluster the images according to the second distance between the feature vectors to obtain a plurality of clusters.

[0140] By adopting the image clustering system of the present application, when the feature vector of the image is obtained through the data acquisition module M100, the image quality score of the image is also obtained. After the first distance between every two feature vectors in the image to be clustered is obtained through the distance calculation module M200, the first distance is further adjusted according to the first distance and the image quality score through the distance adjustment module M300. When the image quality score corresponding to one of the two feature vectors is low and the first distance of the two feature vectors is small, the first distance of the two feature vectors is further reduced to eliminate the influence of the low image quality score on the distance calculation of the two feature vectors, thereby facilitating more accurate image clustering according to distance in the subsequent image clustering module M400, improving the accuracy of image clustering, and improving the recall rate of low-quality images, and more image information can be provided in the image analysis scenario where the clustered image is applied after clustering.

[0141] An embodiment of the present application also provides an image clustering device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to perform the steps of the image clustering method by executing the executable instructions.

[0142] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0143] Refer to the following Figure 7 The electronic device 600 according to this embodiment of the present application is described. Figure 7 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Figure 7 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0145] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the above image clustering method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0146] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0147] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0148] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0149] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] By adopting the image clustering device provided by the present application, the processor executes the image clustering method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned image clustering method.

[0151] The embodiment of the present application also provides a computer-readable storage medium for storing a program, which implements the steps of the image clustering method when executed by a processor. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present application described in the above image clustering method section of this specification.

[0152] refer to Figure 8 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0153] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0155] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0156] By adopting the computer-readable storage medium provided by the present application, the program stored therein implements the steps of the image clustering method when being executed, thereby obtaining the beneficial effects of the above-mentioned image clustering method.

[0157] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. An image clustering method, characterized in that: The steps include: Obtain feature vectors and image quality scores for multiple images; Calculate the first distance between the feature vectors of every two images; According to the image quality scores corresponding to the feature vectors, the first distance is adjusted to obtain a second distance between the feature vectors; wherein, when one of the image quality scores corresponding to the two feature vectors is greater than or equal to a first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, the first distance between the two feature vectors is reduced to obtain a corresponding second distance; The images are clustered according to the second distance between the feature vectors to obtain a plurality of clusters.

2. The image clustering method according to claim 1, characterized in that: According to the image quality score corresponding to the feature vector, the first distance is adjusted to obtain a second distance between the feature vectors, further comprising: When the image quality scores corresponding to the two feature vectors are both smaller than the first quality threshold, the first distance between the two feature vectors is increased to obtain a corresponding second distance.

3. The image clustering method according to claim 1, characterized in that: After calculating the first distance between the feature vectors of every two images, the method further includes: obtaining a first nearest neighbor vector set of each feature vector according to the first distance and the first nearest neighbor number; Adjusting the first distance includes: adjusting a first distance between the feature vector and a first nearest neighbor vector in its first nearest neighbor vector set.

4. The image clustering method according to claim 1, characterized in that: Clustering the images according to the second distance between the feature vectors comprises the following steps: Obtaining a second nearest neighbor vector set for each feature vector according to the second distance and the second nearest neighbor quantity; Calculating the probability that the feature vector and the second nearest neighbor vector in the second nearest neighbor vector set belong to the same target category, and obtaining a probability calculation result; The images are clustered according to the probability calculation results, and images belonging to the same target category are clustered into the same cluster.

5. The image clustering method according to claim 1, characterized in that: After clustering the images according to the second distance between the feature vectors to obtain a plurality of clusters, the method further includes the following steps: According to the number of images in each cluster, the clusters are divided into clusters and discrete clusters; Based on the image quality scores of the images in the clusters, the images in the clusters are divided into high-quality images and low-quality images; When the image quality score of the image in the discrete cluster is greater than or equal to the second quality threshold, and the similarity between the image in the discrete cluster and the high-quality image in a cluster is higher than the first similarity threshold, merging the discrete cluster with the cluster; When the image quality score of the image in the discrete cluster is less than a second quality threshold, and the similarity between the image in the discrete cluster and the image in a cluster is higher than a second similarity threshold, the discrete cluster is merged with the cluster, and the second similarity threshold is less than the first similarity threshold.

6. The image clustering method according to claim 1, characterized in that: After clustering the image according to the second distance between the feature vectors to obtain a plurality of clusters, the method further includes the following step of merging the clusters in the clusters: Based on the image quality scores of the images in the clusters, the images in the clusters are divided into high-quality images and low-quality images; The clusters are sequentially used as designated clusters, and for a designated cluster, a third nearest neighbor vector set of high-quality images of the designated cluster is selected according to the number of third nearest neighbors; If the image corresponding to the third nearest neighbor vector set is not in the specified cluster, and the similarity between the high-quality image in the cluster where the third nearest neighbor image is located and the high-quality image in the specified cluster is greater than a third similarity threshold, the cluster where the image corresponding to the third nearest neighbor is located is merged with the specified cluster.

7. The image clustering method according to claim 6, characterized in that: The step of selecting the third nearest neighbor vector set of the high-quality image of the designated cluster according to the third nearest neighbor quantity comprises the following steps: For the high-quality image of the specified cluster, sort the other images according to the second distance between the feature vectors, and select the third nearest neighbor vector set of the high-quality image of the specified cluster according to the order of the other images and the number of the third nearest neighbors; and / or, For the high-quality images of the specified cluster, other images are sorted according to the probability that the feature vectors belong to the same target category, and the third nearest neighbor vector set of the high-quality images of the specified cluster is selected according to the order of other images and the number of third nearest neighbors.

8. An image clustering system, characterized in that: The image clustering method applied to any one of claims 1 to 7, wherein the system comprises: A data acquisition module, used for acquiring feature vectors and image quality scores of multiple images; A distance calculation module, used for calculating a first distance between feature vectors of every two images; a distance adjustment module, configured to adjust the first distance according to the image quality scores corresponding to the feature vectors to obtain a second distance between the feature vectors; wherein, when one of the image quality scores corresponding to the two feature vectors is greater than or equal to the first quality threshold and the other is less than the first quality threshold, and the first distance between the two feature vectors is less than the first distance threshold, the first distance between the two feature vectors is reduced to obtain a corresponding second distance; The image clustering module is used to cluster the image according to the second distance between the feature vectors to obtain a plurality of clusters.

9. An image clustering device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to perform the steps of the image clustering method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the image clustering method according to any one of claims 1 to 7 are implemented.

Images