A deep image clustering method, system and device based on active selection constraints

Abstract

The present invention belongs to the field of image clustering, and specifically relates to a deep image clustering method, system, and device based on active selection constraints. It includes the following steps: S1: construct a twin network containing two subnetworks with shared weights; S2: add a clustering network after the twin network to obtain the required deep image clustering model; S3: obtain multiple original images to form the required data set; S4: randomly select sample images in the data set, manually add constraint information, and then input them into the deep image clustering model in sequence to complete the clustering process; S5: according to the corresponding clustering results of all samples in the data set, actively select sample images with weak link relationships in each category to construct pairwise constraints and re-cluster; S6: set the number of iterations to iteratively train the network model. The present invention solves the problem that the quality of pairwise constraints between samples in the data set of the existing semi-supervised clustering method is difficult to ensure, which leads to poor clustering accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of image clustering, and in particular relates to a deep image clustering method, system and device based on active selection constraints. Background Art

[0002] Clustering is a fundamental problem in unsupervised learning, and many classic clustering methods have been proposed in recent decades, such as K-Means, spectral clustering, and subspace clustering. These methods aim to divide the data into several clusters so that the data in the same cluster will be close to each other, and those in different clusters will be far away from each other. Since conventional methods usually divide the data in the original feature space, which may not reveal the intrinsic structure of the data, the performance may be limited.

[0003] In order to overcome the problem that conventional clustering methods are prone to performance degradation when processing high-dimensional data, technicians have proposed a deep clustering method combined with deep learning. The deep clustering method applies neural networks to learn embedded representations, which can better characterize the intrinsic structure of the data and then be used to learn the potential representation of the data. Due to the powerful representation ability of neural networks, deep clustering often achieves better performance. However, deep clustering is an unsupervised method after all, and since there are no labels, this method can easily be misled by some noisy or difficult data, which in turn leads to a significant reduction in clustering accuracy.

[0004] In order to overcome the shortcomings of unsupervised deep clustering methods, technicians expect to adjust the clustering process by obtaining some prior information. This method is called semi-supervised deep clustering. In many practical applications, although the labels of the data are difficult to obtain, the relationship between paired data is relatively easy to obtain. For example, in face recognition tasks, we usually don’t know the names of the people in the images (ie, the labels), but we can easily judge whether the two people in the two pictures are the same person. Therefore, many semi-supervised methods apply pairwise constraints (ie, whether the two data are in the same class) as supervisory information to guide clustering.

[0005] Although the performance of the semi-supervised clustering method has been improved, the clustering performance of this method is usually highly dependent on pairwise constraints. How to properly select such pairwise constraints becomes the key to restricting the improvement of clustering method accuracy. In addition, the process of constructing pairwise constraints between data will also greatly expand the data processing scale of deep learning networks and clustering networks, reducing the processing efficiency of clustering problems. Therefore, how to optimize the semi-supervised clustering method and improve the data processing efficiency and clustering accuracy of clustering problems has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0006] To solve the problems that the pairwise constraints between samples in the data set in the existing semi-supervised clustering methods are difficult to effectively construct, the data processing rate of the clustering process is low, and the clustering accuracy is poor; the present invention provides a deep image clustering method, system, and device based on active selection constraints.

[0007] The present invention is implemented by the following technical solutions:

[0008] A deep image clustering method based on active selection constraints includes the following steps:

[0009] S1: Construct a twin network containing two sub-networks with shared weights. The sub-network in the twin network is used to learn the feature information in the input sample image and then extract the embedded representation of the sample image. For the feature information obtained by the two sub-networks, the twin network uses Euclidean distance calculation and outputs the difference between the two, and uses the contrast loss function to optimize the network parameters of the twin network.

[0010] S2: Add a clustering network to the backend of the twin network to obtain the required deep image clustering model. The clustering network predicts the category and cluster center corresponding to each sample image based on the embedded representation of the sample image extracted by the twin network.

[0011] S3: Acquire multiple original images, add color channel attributes to the two-dimensional original images to obtain three-dimensional image data; use each three-dimensional image data as a sample to form a required data set.

[0012] S4: Randomly select any two images in the data set as sample images, and manually add constraint information to the two sample images. The constraint information is used to characterize whether the two sample images belong to the same class. The constructed sample images with pairwise constraint information are sequentially input into the deep image clustering model until all sample images in the data set complete the feature learning and clustering process. Among them, when the constraint information is set to 1, it indicates that the two sample images belong to the same class; when the constraint information is 0, it indicates that the two sample images do not belong to the same class.

[0013] In the present invention, the clustering processing process of the clustering network is as follows:

[0014] S41: Use the K-Means algorithm to determine the clustering results of the data set samples and the cluster centers corresponding to each category.

[0015] S42: Use Student's t distribution to calculate the similarity between each sample image and each cluster center, and then obtain the distribution confidence q corresponding to each sample image ij ; Among them, q ij Represents the sample image x i The probability of belonging to the jth category. The distribution confidence reflects the clustering distribution status of all sample images.

[0016] S43: Assign confidence q to each sample image ij ; The following formula is used to calculate the more obvious clustering trend

[0017] Polarization distribution confidence P ij :

[0018]

[0019] Among them, P ij Represents the sample image x i The probability of belonging to the jth class.

[0020] The calculated polarization distribution confidence P in the present invention ij Relative distribution confidence q ij For each sample image, the distribution state is more sharp, that is, the clustering result is more clear. However, the polarization distribution confidence P ij Relative distribution confidence q of clustering reliability of different samples ij Lower.

[0021] S44: Use KL divergence as the loss function to optimize the distribution confidence q in the clustering process ij and polarization distribution confidence P ij The difference between the two gives the confidence q ij Update; based on the updated distribution confidence q ij Get the clustering results of the data set.

[0022] S5: According to the corresponding clustering results of all samples in the data set determined in the previous step, sample images with weak link relationships in each category are actively selected for re-clustering. The re-clustering process is as follows:

[0023] S51: Obtaining the updated distribution confidence q of the clustering features representing each sample image ij , and determine the number of categories in the clustering results and the number of samples in each category.

[0024] S52: Based on the confidence q ij The value of is used as the center distance between each sample image and the cluster center, and the sample images in each category are sorted in order from near to far distance from the cluster center to obtain an image sequence.

[0025] S53: setting a sampling ratio, calculating the number of samples N for each category according to the sampling ratio, and extracting the first item and the last N-1 items in the image sequence respectively.

[0026] S54: Taking the first item extracted from each category as a sample image and each subsequent item extracted as another sample image, a new pairwise constrained sample image is constructed, and after adding constraint information, the new image is input into the deep image clustering model for re-clustering.

[0027] S6: Set an iteration number; take each round of re-clustering in step S54 as a sign of iteration completion, and repeat the process of steps S4-S5 until the preset iteration number is reached; and then complete the clustering process of all sample images in the data set.

[0028] As a further improvement of the present invention, in step S1, both sub-networks in the twin network use CNN modules as the backbone network, and the backbone networks of the two CNN modules include three 3×3 convolutional layers with a depth of 64, pooling layers, three 3×3 convolutional layers with a depth of 128, pooling layers, three 3×3 convolutional layers with a depth of 256, pooling layers, 1×1 convolutional layers with a depth of 10, pooling layers, and the final fully connected layer in order. The input of the backbone network is the sample image, and the output of the backbone network is the embedded representation of the extracted sample image.

[0029] Among them, in the CNN module of the twin network, all convolutional layers use RELU as the active function; the step size is 1; for the first three pooling layers, 2×2 maximum pooling with a step size of 2 is used; for the last pooling layer, 6×6 average pooling with a step size of 2 is used.

[0030] As a further improvement of the present invention, in the learning process of the twin network extracting respective embedded representations for paired constrained sample images, the goal is to make the outputs of sample images belonging to the same category close to each other, and to make the outputs of sample images belonging to different categories far away from each other.

[0031] Based on the above goals, the loss function L(w) of the twin network in each iteration is set as follows:

[0032]

[0033] In the above formula, M represents a predefined must-connect set, and the paired elements in set M belong to the same category; C represents a predefined cannot-connect set, and the paired elements in set C belong to different categories; w represents the model weight vector for each iteration; x i and x j are two sample images input into the twin network; f(x i ;θ) and f(x j ; θ) represents the embedding representation corresponding to the two sample images extracted by the twin network; θ represents the network parameters of the twin network; y ijis the constraint information of the input paired sample images; δ represents a preset threshold; when y ij =1, that is, the input sample image x i and x j belong to the same category, then the corresponding f(x i ;θ) and f(x j ; θ) should become smaller; when y ij =0, that is, the input sample image x i and x j belong to different categories, then the corresponding f(x i ;θ) and f(x j ; θ) should become larger and smaller than the threshold δ.

[0034] As a further improvement of the present invention, in step S3, the data set is divided into multiple subsets according to the sample size in the data set, and then in step S4, paired constrained sample images are constructed with different samples in each subset and input into the deep image clustering model until all sample images in each subset in the data set complete the feature learning and clustering process.

[0035] Considering that the traditional method of randomly constructing paired constraints between sample images may have a large degree of repetition rate, when processing the same data set, the data scale that needs to be processed by the deep learning network and the clustering network is greatly increased. In order to reduce the data reuse rate when constructing paired constraint sample images and reduce the data processing amount in each round of iteration, the present invention divides the data set into equal parts.

[0036] After the present invention divides the original data set into different subsets, the number of sample images in each subset is greatly reduced. Therefore, in the subsequent random construction of paired constraints, the reuse rate of sample images in each subset is significantly reduced, and the reuse rate of sample images in the entire data set is also significantly reduced. In the actual application process, the subset division data can be reasonably set according to the scale of the original data set. When the scale of the original data set is small, it is not necessary to divide the subsets and directly perform the overall processing. When the scale of the metadata set is large, it is necessary to divide the subsets. And the larger the scale of the original data set, the more the number of subset divisions should be.

[0037] As a further improvement of the present invention, in step S42, the distribution confidence q of each sample image in the data set is ij The calculation formula is as follows:

[0038]

[0039] In the above formula, μ j represents the cluster center of the jth category; α represents a working parameter related to the distribution confidence, and α is set to 1.

[0040] As a further improvement of the present invention, in step S44, KL divergence is used as the loss function to optimize the distribution confidence q in the clustering process. ij and polarization distribution confidence P ij The difference between the two, the split configuration confidence q ij During the update process, the loss function L 2 The setup is as follows:

[0041]

[0042] In the above formula, n represents the number of sample images in the data set; c represents the number of clustering processing categories of the data set;

[0043] Based on the optimized updated distribution confidence q ij ; The minimized network parameters θ and cluster centers μ are obtained through the stochastic gradient descent method j ; Sample image x obtained after the loss function converges i The cluster center is the distribution confidence q ij The subscript of the maximum value of each row of data.

[0044] As a further improvement of the present invention, each sample image in the data set obtained in step 3 also includes a category label for characterizing the true category of the image; in step S6, after completing the final clustering process, the final clustering result is also compared with the true category label of each sample image, and then the clustering result is evaluated by clustering accuracy.

[0045] The present invention also includes a deep image clustering system based on active selection constraints, which uses the deep image clustering method based on active selection constraints as described above to quickly cluster large-scale sample images. The clustering process includes an initialization clustering stage for all sample images and a re-clustering stage based on some sample images after active selection. The deep image clustering system includes: a data set acquisition module, a pairwise constraint construction module, a feature learning module, a clustering module, and an iteration control module.

[0046] Among them, the data set acquisition module is used to obtain the sample images to be clustered, and add color channel information to the two-dimensional data in the sample images. When the sample image is a color image, the number of color channels is set to 3; when the sample image is a grayscale image, the number of color channels is 1; after adding the color channel information, the sample images in the data set are upgraded to three-dimensional data; and then a data set to be clustered consisting of all three-dimensional sample images is obtained.

[0047] The pairwise constraint construction module is used to select sample images in the dataset and construct pairwise constraints between any two images. Every two sample images with a pairwise constraint relationship are used as the input of a feature learning module. There are two strategies for constructing constraint information between sample images: (1) In the initial clustering stage, sample images are selected from the dataset by random selection, and constraint information is manually annotated for every two sample images until all samples have completed the construction of pairwise constraint relationships. (2) In the re-clustering stage, new pairwise constraints are constructed based on some sample images selected by an active selection model.

[0048] The feature learning module uses a twin network constructed with a CNN network as a subnetwork. The two subnetworks in the twin network share weights. The backbone network is used to learn the feature information in the input sample image and then extract the embedded representation of the sample image. For the feature information obtained by the two subnetworks, the twin network uses Euclidean distance calculation and outputs the difference between the two, and uses the contrast loss function to optimize the parameters of the twin network.

[0049] The clustering module is used to: (1) obtain the embedded representation of each sample image in the data set extracted by the feature learning module, and then calculate the clustering result of the data set; the clustering result includes the number of categories corresponding to the data set and the cluster center of each category. (2) Calculate the distribution confidence and polarization distribution confidence that characterize the cluster distribution characteristics based on the similarity between the sample image and the cluster center. (3) Use KL divergence as the loss function and adjust and update the distribution confidence based on the polarization distribution confidence.

[0050] The active selection module is used to: (1) obtain the clustering results of the clustering module when initializing the clustering, and then determine the distribution confidence of each sample image and the cluster center corresponding to each category. (2) Sort all sample images in each category according to their distance from the cluster center based on the distribution confidence. (3) Calculate the number of sample images that need to be re-clustered in each category according to the preset sampling ratio. (4) According to the number of samples and the sorting relationship of each category, extract the sample images corresponding to the cluster center as one category, and several sample images far away from the cluster center as another category; use the two categories of sample images as sample images to construct new pairwise constraints in the re-clustering stage.

[0051] The iterative control module is used to control the paired constraint construction module, feature learning module, clustering model and active selection module to iteratively cluster the data set provided by the data set acquisition module according to the preset number of iterations. During the iteration process, the parameter optimization of each functional module is completed, and the final clustering result of the data set is obtained.

[0052] The present invention also includes a deep image clustering device based on active selection constraints, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the deep image clustering method based on active selection constraints are implemented as described above.

[0053] The technical solution provided by the present invention has the following beneficial effects:

[0054] The deep image clustering method based on active selection constraints proposed in this invention constructs a twin network using a convolutional neural network as the backbone to effectively learn and extract the deep feature information and embedded representation of complex images. At the same time, a clustering layer is inserted into the twin network to reveal the clustering structure of the data, thereby obtaining the clustering results.

[0055] In order to improve the clustering accuracy of the network model, the present invention combines the initial manual annotation with the output results of the first-level clustering layer to automatically select the paired constraints between the data, thereby realizing iterative processing of the model, continuously learning data features and optimizing clustering results, and realizing efficient semi-supervised deep image clustering processing. It should be emphasized that in the active constraint selection strategy of the present invention, uncertainty and information volume are fully considered to evaluate paired data so that the selected constraints are valuable for clustering. At the same time, the data processing method is also optimized, and the overhead caused by duplicate data in the data processing process is reduced by setting subsets, which effectively improves the work efficiency of the network model in processing clustering problems.

[0056] The present invention seamlessly connects the three tasks of feature learning, clustering processing and constraint active selection contained in the semi-supervised deep clustering problem into a unified framework, so that the three tasks can promote each other. Finally, the final clustering result is obtained through the clustering layer. It can not only combine the automated and efficient processing of the data set, but also give play to the supervisory role of artificial prior information on the network model, and improve the clustering accuracy of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of the steps of a deep image clustering method based on active selection constraints provided in Example 1 of the present invention.

[0058] Figure 2 This is a module diagram of the CNN module inside the twin network in Example 1 of the present invention.

[0059] Figure 3 This is a distribution diagram of each sample image in the initial clustering result in Example 1 of the present invention.

[0060] Figure 4 This is a location distribution diagram of cluster centers in each category in the initial clustering result in Example 1 of the present invention.

[0061] Figure 5 This is a link relationship diagram between objects with manual selection constraints in Example 1 of the present invention.

[0062] Figure 6 This is a system framework diagram of a deep image clustering system based on active selection constraints provided in Example 2 of the present invention.

[0063] Figure 7 This is a module schematic diagram of a deep image clustering system based on active selection constraints provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0065] Example 1

[0066] This embodiment provides a deep image clustering method based on active selection constraints, such as Figure 1 As shown, the deep image clustering method includes the following steps:

[0067] S1: Construct a twin network containing two sub-networks with shared weights. The sub-network in the twin network is used to learn the feature information in the input sample image and then extract the embedded representation of the sample image. For the feature information obtained by the two sub-networks, the twin network uses Euclidean distance calculation and outputs the difference between the two, and uses the contrast loss function to optimize the network parameters of the twin network.

[0068] like Figure 2 As shown in the figure, both sub-networks in the twin network use CNN modules as the backbone network. The backbone networks of the two CNN modules include three 3×3 convolutional layers with a depth of 64, pooling layers, three 3×3 convolutional layers with a depth of 128, pooling layers, three 3×3 convolutional layers with a depth of 256, pooling layers, 1×1 convolutional layers with a depth of 10, pooling layers, and the final fully connected layer. The input of the backbone network is the sample image, and the output of the backbone network is the embedded representation of the extracted sample image.

[0069] Among them, in the CNN module of the twin network, all convolutional layers use RELU as the active function; the step size is 1; for the first three pooling layers, 2×2 maximum pooling with a step size of 2 is used; for the last pooling layer, 6×6 average pooling with a step size of 2 is used.

[0070] During the twin network processing: it is necessary to construct a must-link set M and a cannot-link set C. In the first iteration, for efficiency reasons, we randomly select k pairs of data to manually annotate to obtain the initial M and C. In the next iteration, M and C will be constructed through dynamic selection based on the previous clustering results.

[0071] Among them, for the two samples x input into the twin network i and x j If: (x i ,x j )∈M,y ij =1 and if (xi,xj)∈C, yij=0. Then it is obvious that ij The value of indicates whether xi and xj are in the same cluster.

[0072] When the network input x i and x j After entering the two CNN modules with shared weights, use f(x i ;θ) and f(x j ; θ) represent the input x i and x j The output of the corresponding CNN module is the embedded representation of the sample extracted by the CNN model. Among them, θ is the learnable network parameter in the CNN module.

[0073] For the twin network, if y ij =1, that is, x i and x j belong to the same cluster, f(x i ;θ) and f(x j ;θ) should be as close as possible if y ij =0, f(x i ;θ) and f(x j ; θ) should be far away from each other. That is, in the learning process of extracting the respective embedded representations of paired constrained sample images by the twin network, the goal is to make the outputs of sample images belonging to the same category close to each other, and to make the outputs of sample images belonging to different categories far away from each other. Based on the above goals, the loss function L(w) of the twin network in each iteration is set as follows:

[0074]

[0075] In the above formula, M represents a predefined must-connect set, and the paired elements in set M belong to the same category; C represents a predefined cannot-connect set, and the paired elements in set C belong to different categories; w represents the model weight vector for each iteration; x i and x jare two sample images input into the twin network; f(x i ;θ) and f(x j ; θ) represents the embedding representation corresponding to the two sample images extracted by the twin network; θ represents the network parameters of the twin network; y ij is the constraint information of the input sample image; δ represents a preset threshold; when y ij =1, that is, the input sample image x i and x j belong to the same category, then the corresponding f(x i ;θ) and f(x j ; θ) should become smaller; when y ij =0, that is, the input sample image x i and x j belong to different categories, then the corresponding f(x i ;θ) and f(x j ; θ) should become larger and smaller than the threshold δ.

[0076] S2: Add a clustering network to the backend of the twin network to obtain the required deep image clustering model. The clustering network predicts the category and cluster center corresponding to each sample image based on the embedded representation of the sample image extracted by the twin network.

[0077] In the clustering network, the initial categories and cluster centers corresponding to the data set to be clustered are calculated by the K-Means algorithm. The cluster distribution state of each sample in the data set is obtained by calculating the distribution confidence of each sample.

[0078] S3: Acquire multiple original images, add color channel attributes to the two-dimensional original images to obtain three-dimensional image data; use each three-dimensional image data as a sample to form a required data set.

[0079] The original image used in this embodiment is a two-dimensional data, which includes pixel information in the width and height directions of the image. In order to be recognized by the convolutional neural network (CNN), this embodiment performs dimensionality upscaling on the original image and adds color channel information for each image. When the sample image is a grayscale image, the number of color channels is 1; after adding the color channel information, the sample image in the data set is upscaled to three-dimensional data.

[0080] S4: Randomly select any two images in the data set as sample images, and manually add constraint information to the two sample images. The constraint information is used to characterize whether the two sample images belong to the same class. The constructed sample images with pairwise constraint information are sequentially input into the deep image clustering model until all sample images in the data set complete the feature learning and clustering process. Among them, when the constraint information is set to 1, it indicates that the two sample images belong to the same class; when the constraint information is 0, it indicates that the two sample images do not belong to the same class.

[0081] In the present invention, the clustering processing process of the clustering network is as follows:

[0082] S41: Using the K-Means algorithm, a plurality of initialization categories corresponding to the data set are determined, and a cluster center corresponding to each initialization category is obtained.

[0083] S42: Use Student's t distribution to calculate the similarity between each sample image and each cluster center, and then obtain the distribution confidence q corresponding to each sample image ij ; Among them, q ij Represents the sample image x i The probability of belonging to the jth category. The distribution confidence reflects the clustering distribution status of all sample images.

[0084] In step S42, the distribution confidence q of each sample image in the data set is ij The calculation formula is as follows:

[0085]

[0086] In the above formula, μ j represents the cluster center of the jth category; α represents a working parameter related to the distribution confidence, and α is set to 1.

[0087] S43: Assign confidence q to each sample image ij ; The following formula is used to calculate the more obvious clustering trend

[0088] Polarization distribution confidence P ij :

[0089]

[0090] Among them, P ij Represents the sample image x i The probability of belonging to the jth class.

[0091] The calculated polarization distribution confidence P in the present invention ij Relative distribution confidence q ijFor each sample image, the distribution state is more sharp, that is, the clustering result is more clear. However, the polarization distribution confidence P ij Relative distribution confidence q of clustering reliability of different samples ij Lower.

[0092] S44: Use KL divergence as the loss function to optimize the distribution confidence q in the clustering process ij and polarization distribution confidence P ij The difference between the two gives the confidence q ij Update; based on the updated distribution confidence q ij Get the clustering results of the data set.

[0093] In step S44, KL divergence is used as the loss function to optimize the distribution confidence q in the clustering process. ij and polarization distribution confidence P ij The difference between the two, the split configuration confidence q ij During the update process, the loss function L 2 The setup is as follows:

[0094]

[0095] In the above formula, n represents the number of sample images in the data set; c represents the number of clustering processing categories of the data set;

[0096] In this embodiment, when KL divergence is used as the loss function for optimization, the distribution confidence q can be reduced without significantly reducing the distribution confidence q ij Based on the reliability of ij Appropriately allocate confidence P to polarization ij Closer; updated distribution confidence q ij It has a clearer clustering effect.

[0097] Based on the optimized updated distribution confidence q ij ; The minimized network parameters θ and cluster centers μ are obtained through the stochastic gradient descent method j ; Sample image x obtained after the loss function converges i The cluster center is the distribution confidence q ij The subscript of the maximum value of each row of data.

[0098] S5: According to the corresponding clustering results of all samples in the data set determined in the previous step, sample images with weak link relationships in each category are actively selected for re-clustering. The re-clustering process is as follows:

[0099] S51: Obtaining the updated distribution confidence q of the clustering features representing each sample image ij, and determine the number of categories in the clustering results and the number of samples in each category.

[0100] S52: Based on the confidence q ij The value of is used as the center distance between each sample image and the cluster center, and the sample images in each category are sorted in order from near to far distance from the cluster center to obtain an image sequence.

[0101] S53: setting a sampling ratio, calculating the number of samples N for each category according to the sampling ratio, and extracting the first item and the last N-1 items in the image sequence respectively.

[0102] S54: Taking the first item extracted from each category as a sample image and each subsequent item extracted as another sample image, a new pairwise constrained sample image is constructed, and after adding constraint information, the new image is input into the deep image clustering model for re-clustering.

[0103] The following combination Figure 3-5 The implementation process of the active selection constraint in this embodiment is explained more clearly. After the initialization clustering process is completed, the number of categories and the category distribution of each sample can be obtained according to the clustering result. Figure 3 The initial clustering result of a data set with 25 samples is shown. Each solid circle in the figure represents a sample data. Different color grayscales distinguish different categories. Figure 3 It can be found that all samples are divided into 3 categories, of which the category in the upper left corner includes 11 samples; the category in the upper right corner includes 8 samples, and the category below includes 6 samples. According to the distribution confidence calculated in the clustering result, the distance relationship between each sample in each category and the cluster center can be determined. The sample closest to the cluster center is defined as the centroid of each category (i.e., different clusters), and the cluster center in each category is determined as Figure 4 The solid points with a circle in the middle and outside. Next, based on the distance between each sample and the cluster center, the points close to the center and the points far from the center in each category can be determined.

[0104] Considering that in order to obtain better clustering performance, the ideal data input into the deep image clustering model as supervised data should satisfy the following properties:

[0105] (1) With uncertainty: If the model can easily determine x i and x j Whether they belong to the same cluster, there is no need to query manual annotations; therefore, it is necessary to select uncertain paired data for labeling, and the obtained artificial constraint information is most helpful for feature learning in the clustering process.

[0106] (2) More information: It is easy to verify that there are many more constraints that cannot be linked than constraints that must be linked. However, the goal of clustering is to determine which instances are in the same cluster. Therefore, constraints that must be linked provide more information than constraints that cannot be linked, and they are more helpful for clustering than constraints that cannot be linked.

[0107] In summary, in the process of actively selecting constraints, samples closest to the edge of the cluster and samples at the center of the cluster should be selected to form new pairwise constraints to relearn and cluster the deep clustering model.

[0108] like Figure 5 As shown in the figure, according to the difference in the number of samples in the three categories, we get the number of samples of the newly added pairwise constraints generated by different categories. The number of categories in the upper left corner is large, and the newly added pairwise constraints are determined to be 3 groups, while the number of samples in the other two categories is relatively small. In this embodiment, the newly added pairwise constraints are determined to be 2 groups. Specifically, one sample in each category located in the cluster is selected as an object, and another object close to the edge of the category is selected to form the following Figure 5 The constraint information between each pair of constraints constructed should be manually queried and marked. For example, Figure 5 The solid line between two samples in the figure means "must be linked", that is, the two samples belong to the same class; while the dotted line means "cannot be linked", that is, the two samples do not belong to the same class. These data with pairwise constraint information will be used as input data for the deep image clustering model in the re-clustering stage.

[0109] S6: Set an iteration number; take each round of re-clustering in step S54 as a sign of iteration completion, and repeat the process of steps S4-S5 until the preset iteration number is reached; and then complete the clustering process of all sample images in the data set.

[0110] The iterative process can fine-tune the network parameters θ of the twin network and obtain more accurate clustering centers {μ 1 ,····,μ c}; Therefore, in the application process of the deep image clustering model, a reasonable number of iterations can be set as needed, and the processing process of the deep image clustering model can be repeatedly iterated using the data set to directly meet the training requirements of the network model.

[0111] In each round of iteration, in order to save data processing budget, it is usually checked whether each pairwise constraint has been processed, and if so, it is reselected.

[0112] In the method provided in this example, each sample image in the data set also includes a category label for characterizing the true category of the image. For example, in a pile of face images used for training, each face image may have a category label that reflects the person corresponding to the face in the image. Therefore, after the final clustering process is completed in this embodiment, the final clustering result can also be compared with the true category label of each sample image, and then the performance of the deep image clustering method based on active selection constraints provided in this embodiment can be evaluated by clustering accuracy.

[0113] It should be noted that the above content introduces a strategy for selecting information-rich but uncertain pairs for labeling in this embodiment. However, in each round of iteration, for all candidate pairs, we should also check whether they have been selected in the previous iteration. If they have been selected, they should not be processed again, otherwise it will generate a large computational overhead and occupy the effective processing budget. And it will cause the entire operation process of the network model to take a long time. At the same time, the process also needs to traverse the previous selection set multiple times.

[0114] Considering that the traditional method of randomly constructing pairwise constraints between sample images may have a large degree of repetition, when processing the same data set, the scale of data that needs to be processed by the deep learning network and clustering network is greatly increased. In order to solve this problem, this embodiment proposes a simpler but effective method to avoid checking whether a pair has been selected before. The specific method is: divide the data set into multiple subsets according to the sample size in the data set, and then construct pairwise constrained sample images with different samples in each subset, and input them into the deep image clustering model until all sample images in each subset in the data set complete the feature learning and clustering process.

[0115] After the original data set is divided into different subsets in this embodiment, the number of sample images in each subset is greatly reduced. Therefore, in the subsequent random construction of paired constraints, the reuse rate of sample images in each subset is significantly reduced, and the reuse rate of sample images in the entire data set is also significantly reduced. In actual application, the subset division data can be reasonably set according to the scale of the original data set. When the scale of the original data set is small, it is not necessary to divide the subsets and directly perform overall processing. When the scale of the metadata data set is large, subset division is required. And the larger the scale of the original data set, the more subsets should be divided. In this embodiment, the number of subset divisions can be set to the number of iterations of the network model. That is: assuming that the number of pairs (or the number of iterations) is T, we first divide the original data set X into T non-overlapping subsets: X 1 ....X T , at the Tth iteration, only in X T Select constraints on.

[0116] In this embodiment, all convolutional layers use RELU as the active function, the convolution kernel size is 3×3, and the step size is 1. For the first three pooling layers, 2×2 maximum pooling with a step size of 2 is used. For the last pooling layer, 6×6 average pooling with a step size of 2 is used. The dimension embedding of the output is 10. When training the twin network with pairwise constraints, the preset threshold δ is fixed to 1, the batch size (Batchsize) is adjusted to 64, and the training round (epoch) is set to 100. RMSProp is used as the optimizer with a learning rate of 0.001. When fine-tuning the network using the clustering layer, we use SGD with a learning rate of 0.001 as the optimizer.

[0117] Example 2

[0118] This embodiment provides a deep image clustering system based on active selection constraints. The deep image clustering system adopts the deep image clustering method based on active selection constraints provided in Example 1 to quickly cluster large-scale sample images.

[0119] like Figure 6 As shown, the clustering process includes an initialization clustering stage for all sample images and a re-clustering stage based on some sample images after active selection.

[0120] like Figure 7 As shown, the deep image clustering system includes: a data set acquisition module, a pairwise constraint construction module, a feature learning module, a clustering module, and an iteration control module.

[0121] Among them, the data set acquisition module is used to obtain the sample images to be clustered, and add color channel information to the two-dimensional data in the sample images. When the sample image is a color image, the number of color channels is set to 3; when the sample image is a grayscale image, the number of color channels is 1; after adding the color channel information, the sample images in the data set are upgraded to three-dimensional data; and then a data set to be clustered consisting of all three-dimensional sample images is obtained.

[0122] The pairwise constraint construction module is used to select sample images in the dataset and construct pairwise constraints between any two images. Every two sample images with a pairwise constraint relationship are used as the input of a feature learning module. There are two strategies for constructing constraint information between sample images: (1) In the initial clustering stage, sample images are selected from the dataset by random selection, and constraint information is manually annotated for every two sample images until all samples have completed the construction of pairwise constraint relationships. (2) In the re-clustering stage, new pairwise constraints are constructed based on some sample images selected by an active selection model.

[0123] The feature learning module uses a twin network constructed with a CNN network as a subnetwork. The two subnetworks in the twin network share weights. The backbone network is used to learn the feature information in the input sample image and then extract the embedded representation of the sample image. For the feature information obtained by the two subnetworks, the twin network uses Euclidean distance calculation and outputs the difference between the two, and uses the contrast loss function to optimize the parameters of the twin network.

[0124] The clustering module is used to: (1) obtain the embedded representation of each sample image in the data set extracted by the feature learning module, and then calculate the clustering result of the data set; the clustering result includes the number of categories corresponding to the data set and the cluster center of each category. (2) Calculate the distribution confidence and polarization distribution confidence that characterize the cluster distribution characteristics based on the similarity between the sample image and the cluster center. (3) Use KL divergence as the loss function and adjust and update the distribution confidence based on the polarization distribution confidence.

[0125] The active selection module is used to: (1) obtain the clustering results of the clustering module when initializing the clustering, and then determine the distribution confidence of each sample image and the cluster center corresponding to each category. (2) Sort all sample images in each category according to their distance from the cluster center based on the distribution confidence. (3) Calculate the number of sample images that need to be re-clustered in each category according to the preset sampling ratio. (4) According to the number of samples and the sorting relationship of each category, extract the sample images corresponding to the cluster center as one category, and several sample images far away from the cluster center as another category; use the two categories of sample images as sample images to construct new pairwise constraints in the re-clustering stage.

[0126] The iterative control module is used to control the paired constraint construction module, feature learning module, clustering model and active selection module to iteratively cluster the data set provided by the data set acquisition module according to the preset number of iterations. During the iteration process, the parameter optimization of each functional module is completed, and the final clustering result of the data set is obtained.

[0127] Example 3

[0128] This embodiment provides a deep image clustering device based on active selection constraints, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the deep image clustering method based on active selection constraints are implemented as described above.

[0129] The computer device may be a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. The computer device of this embodiment includes at least but is not limited to: a memory and a processor that can communicate with each other through a system bus.

[0130] In this embodiment, the memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0131] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of a computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data to implement the processing of the deep image clustering method based on active selection constraints in the aforementioned embodiment, so as to accurately cluster all image samples in the data set to be clustered into different categories.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep image clustering method based on active selection constraints, characterized by: The steps include: S1: Construct a twin network including two sub-networks with shared weights; the sub-network in the twin network is used to learn the feature information in the input sample image and then extract the embedded representation of the sample image; for the feature information obtained by the two sub-networks, the twin network uses Euclidean distance calculation and outputs the difference between the two, and uses contrast loss function to optimize the network parameters of the twin network; S2: Add a clustering network to the back end of the twin network to obtain the required deep image clustering model; the clustering network predicts the category and cluster center corresponding to each sample image according to the embedded representation of the sample image extracted by the twin network; S3: Acquire multiple original images, add color channel attributes to the two-dimensional original images to obtain three-dimensional image data; use each three-dimensional image data as a sample to form a required data set; S4: randomly selecting any two images in the data set as sample images, and manually adding constraint information to the two sample images, wherein the constraint information is used to characterize whether the two sample images belong to the same class; and sequentially inputting the constructed sample images with pairwise constraint information into the deep image clustering model until all sample images in the data set complete feature learning and clustering processes; The clustering process of the clustering network is as follows: S41: Using the k-means algorithm to determine the clustering results of the data set samples and the cluster centers corresponding to each category; S42: Use Student's t distribution to calculate the similarity between each sample image and each cluster center, and then obtain the distribution confidence q corresponding to each sample image ij ; Among them, q ij Represents the sample image x i The probability of belonging to the jth category; S43: Assign confidence q to each sample image ij ; The following formula is used to calculate the confidence value P of the polarization distribution with a more obvious clustering trend ij : Where n represents the number of sample images in the data set; c represents the number of clustering processing categories of the data set; S44: Use KL divergence as the loss function to optimize the distribution confidence q in the clustering process ij and polarization distribution confidence P ij The difference between the two gives the confidence q ij Update; obtain the clustering result of the data set based on the updated distribution confidence; S5: According to the corresponding clustering results of all samples in the data set determined in the previous step, sample images with weak link relationships in each category are actively selected for re-clustering. The re-clustering process is as follows: S51: Obtaining the updated distribution confidence q of the clustering features representing each sample image ij , and determine the number of categories in the clustering results and the number of samples in each category; S52: Assign confidence q based on the distribution ij The value of is used as the center distance between each sample image and the cluster center, and the sample images in each category are sorted in the order of the distance from the cluster center from near to far, so as to obtain an image sequence; S53: setting a sampling ratio, calculating the number of samples N for each category according to the sampling ratio, and extracting the first item and the last N-1 items in the image sequence respectively; S54: taking the first item extracted from each category as a sample image and each subsequent item extracted as another sample image, constructing a new pair-wise constrained sample image, and inputting the new image into the deep image clustering model for re-clustering after adding constraint information; S6: Set an iteration number; take each round of re-clustering in step S54 as a sign of iteration completion, and repeat the process of steps S4-S5 until the preset iteration number is reached; and then complete the clustering process of all sample images in the data set.

2. The deep image clustering method based on active selection constraints as claimed in claim 1, characterized in that: In step S1, the two sub-networks in the twin network both use CNN modules as the backbone network. The backbone networks of the two CNN modules include, in order, three 3×3 convolutional layers with a depth of 64, a pooling layer, three 3×3 convolutional layers with a depth of 128, a pooling layer, three 3×3 convolutional layers with a depth of 256, a pooling layer, a 1×1 convolutional layer with a depth of 10, a pooling layer, and a final fully connected layer; the input of the backbone network is the sample image, and the output of the backbone network is the embedded representation of the extracted sample image.

3. The deep image clustering method based on active selection constraints as claimed in claim 2, characterized in that: In the CNN module in the twin network, all convolutional layers use RELU as the active function; the step size is 1; for the first three pooling layers, 2×2 maximum pooling with a step size of 2 is used; for the last pooling layer, 6×6 average pooling with a step size of 2 is used.

4. The deep image clustering method based on active selection constraints as claimed in claim 1, characterized in that: In the learning process of extracting respective embedding representations for paired constrained sample images, the twin network makes the outputs of sample images belonging to the same category close to each other, and makes the outputs of sample images belonging to different categories far away from each other; The loss function L(w) of the twin network in each iteration is set as follows: In the above formula, M represents a predefined must-connect set, and the paired elements in set M belong to the same category; C represents a predefined cannot-connect set, and the paired elements in set C belong to different categories; w represents the model weight vector for each iteration; x i and x j are two sample images input into the twin network; f(x i ;θ) and f(x j ; θ) represents the embedding representation corresponding to the two sample images extracted by the twin network; θ represents the network parameters of the twin network; y ij is the constraint information of the input paired sample images; δ represents a preset threshold; when y ij =1, that is, the input sample image x i and x j belong to the same category, then the corresponding f(x i ;θ) and f(x j ; The Euclidean distance between θ) should be as small as possible; when y ij =0, that is, the input sample image x i and x j belong to different categories, then the corresponding f(x i ;θ) and f(x j ; θ) should become larger and smaller than the threshold δ.

5. The deep image clustering method based on active selection constraints as claimed in claim 1, characterized in that: In step S3, the data set is divided into multiple subsets according to the sample size in the data set, and then in step S4, paired constrained sample images are constructed with different samples in each subset and input into the deep image clustering model until all sample images in each subset in the data set complete the feature learning and clustering process.

6. The deep image clustering method based on active selection constraints as claimed in claim 1, characterized in that: In step S42, the distribution confidence q of each sample image in the data set is ij The calculation formula is as follows: In the above formula, μ j represents the cluster center of the jth category; α represents a working parameter related to the distribution confidence, and α is set to 1.

7. The deep image clustering method based on active selection constraints as claimed in claim 1, characterized in that: In step S44, KL divergence is used as the loss function to optimize the distribution confidence q in the clustering process. ij and polarization distribution confidence P ij The difference between the two gives the confidence q ij During the update process, the loss function L2 is set as follows: In the above formula, n represents the number of sample images in the data set; c represents the number of clustering processing categories of the data set; Based on the optimized updated distribution confidence q ij ; The minimized network parameters θ and cluster centers μ are obtained through the stochastic gradient descent method j ; Sample image x obtained after the loss function converges i The cluster category to which it belongs is the distribution confidence q ij The subscript of the maximum value of each row of data.

8. The deep image clustering method based on active selection constraints as claimed in claim 1, characterized in that: Each sample image in the data set obtained in step 3 also includes a category label for representing the true category of the image; In step S6, after the final clustering process is completed, the final clustering result is also compared with the actual category label of each sample image, and the clustering result is evaluated by clustering accuracy.

9. A deep image clustering system based on active selection constraints, characterized in that: The deep image clustering system adopts the deep image clustering method based on active selection constraints as described in any one of claims 1 to 8 to quickly cluster large-scale sample images; the clustering process includes an initialization clustering stage for all sample images and a re-clustering stage based on some sample images after active selection; the deep image clustering system includes: The data set acquisition module is used to acquire sample images to be clustered and add color channel information to the two-dimensional data in the sample images. When the sample images are color images, the number of color channels is set to 3; when the sample images are grayscale images, the number of color channels is 1; after adding the color channel information, the sample images in the data set are upgraded to three-dimensional data; and then the data set to be clustered consisting of all three-dimensional sample images is obtained; A pairwise constraint construction module is used to select sample images in a data set and construct pairwise constraints between any two images therein; every two sample images with a pairwise constraint relationship are used as the input of a feature learning module; there are two strategies for constructing constraint information between sample images: (1) In the initial clustering stage, sample images are selected from the data set by random selection, and constraint information is manually annotated for every two sample images until all samples have completed the construction of pairwise constraint relationships; (2) In the re-clustering stage, new pairwise constraints are constructed based on some sample images selected by an active selection model; A feature learning module, which uses a twin network constructed by a CNN network as a subnetwork; the two subnetworks in the twin network share weights; the subnetwork is used to learn the feature information in the input sample image and then extract the embedded representation of the sample image; for the feature information obtained by the two subnetworks, the twin network uses Euclidean distance calculation and outputs the difference between the two, and uses a contrast loss function to optimize the parameters of the twin network; A clustering module, which is used to: (1) obtain the embedded representation of each sample image in the data set extracted by the feature learning module, and then calculate the clustering result of the data set; the clustering result includes the number of categories corresponding to the data set and the cluster center of each category; (2) calculate the distribution confidence and polarization distribution confidence that characterize the cluster distribution characteristics according to the similarity between the sample image and the cluster center; (3) use KL divergence as a loss function, and adjust and update the distribution confidence according to the polarization distribution confidence; An active selection module is used to: (1) obtain the clustering module's initialization clustering results, and then determine the distribution confidence of each sample image and the cluster center corresponding to each category; (2) sort all sample images in each category according to their distance relationship with the cluster center based on the distribution confidence; (3) calculate the number of sample images that need to be re-clustered in each category according to a preset sampling ratio; (4) according to the number of samples and the sorting relationship of each category, extract the sample images corresponding to the cluster center as one category, and several sample images far from the cluster center as another category; use the two categories of sample images as new pairwise constrained sample images for constructing the re-clustering stage; and An iterative control module is used to control the paired constraint construction module, feature learning module, clustering model and active selection module to perform iterative clustering processing on the data set provided by the data set acquisition module according to a preset number of iterations, complete parameter optimization of each functional module during the iteration process, and obtain the final clustering result of the data set.

10. A deep image clustering device based on active selection constraints, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the deep image clustering method based on active selection constraints as described in any one of claims 1 to 8 are implemented.

Images