Deep clustering network
By designing a deep clustering network, defining a unified objective function and using a non-maximum suppression sampling method, the problems of poor clustering performance and insufficient stability in the prior art are solved, and high-performance and stable clustering effects are achieved.
Patent Information
- Application Number
- CN202411912564.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-04-22
AI Technical Summary
The existing deep clustering network based on variational autoencoding includes reconstruction terms and clustering terms in the objective function, resulting in conflicting optimization goals, poor clustering performance, and using Monte Carlo sampling in decoding, resulting in poor clustering stability.
A deep clustering network is designed, including input layer, encoding layer, clustering layer and output layer. By defining a unified objective function, including clustering terms, equilibrium constraints and discriminant constraints, the non-maximum suppression sampling method is used to cluster, reduce parameters and achieve coordinated optimization.
The performance and stability of the clustering algorithm are improved, the time required for the clustering algorithm to run is reduced, the coordinated optimization of feature learning and clustering tasks is realized, and the stability of the clustering effect is improved.
Smart Images
Figure CN119862910A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning, and in particular to a deep clustering network. Background Art
[0002] Deep learning theory and technology have developed rapidly in recent years, especially in the field of image processing. In clustering tasks, it is mainly integrated with unsupervised networks such as autoencoders and variational autoencoders to achieve a significant improvement in performance.
[0003] However, the current deep clustering algorithm based on variational autoencoder has two shortcomings:
[0004] 1. These unsupervised networks do not have a targeted design for clustering modules and often require the introduction of additional parameters outside the network. Therefore, it is impossible to achieve the coordinated optimization of feature learning and clustering tasks through the overall parameter update of the network;
[0005] 2. Although the decoding part of the variational autoencoder network gives it the ability to generate, the existing deep clustering network objective function based on variational autoencoders contains both reconstruction and clustering terms. The two are contradictory in network optimization. The optimization goal of the decoding part and the clustering task cannot be unified, resulting in poor clustering performance. Monte Carlo sampling is widely used in decoding, resulting in poor clustering stability. Therefore, there is an urgent need for an algorithm that unifies clustering tasks and feature learning tasks and completely unifies parameters in the network. Summary of the invention
[0006] In view of the defects in the prior art, the present invention provides a deep clustering network to reduce the time required for the operation of the existing clustering algorithm and improve the performance and stability of the clustering algorithm.
[0007] A deep clustering network comprises an input layer, an encoding layer, a clustering layer and an output layer, wherein:
[0008] Input layer, used to receive N data to be processed;
[0009] The encoding layer is used to calculate the N data to be processed respectively to obtain the low-dimensional feature distribution corresponding to each data to be processed, and perform a first sampling operation on the low-dimensional feature distribution corresponding to each data to be processed to obtain N low-dimensional data, each low-dimensional data corresponds to each data to be processed one by one;
[0010] The clustering layer is used to calculate the N low-dimensional data respectively to obtain the cluster category distribution corresponding to each low-dimensional data, and perform a second sampling operation on the cluster category distribution corresponding to each low-dimensional data by the non-maximum suppression sampling method to obtain the clustering results of the N low-dimensional data. The clustering result of each low-dimensional data corresponds one-to-one to the clustering result of each data to be processed;
[0011] The output layer is used to output the clustering results of N data to be processed.
[0012] Furthermore, it also includes: defining an objective function of a deep clustering network, wherein the objective function of the deep clustering network includes clustering terms, balance constraints, and discrimination constraints.
[0013] Furthermore, the clustering items are obtained by analyzing the general derivation formula of the objective function.
[0014] The general derivation of the objective function is represented by the following expression:
[0015]
[0016] Where x is the data to be processed, KL represents the Kullback-Leibler divergence, z represents the low-dimensional data, q(z|x) represents the low-dimensional feature distribution corresponding to the data to be processed, c represents the category of the data to be processed, p(z|c) represents the distribution obeyed by the low-dimensional data, q(c|z) represents the cluster category distribution corresponding to the low-dimensional data, N is the total number of data to be processed, and D = ∫ z ∏ c p(z|c) q(c|z) dz.
[0017] Furthermore, the N low-dimensional data obey a normal distribution with k means as clustering layer network parameters and a variance of 1.
[0018] Furthermore, it also includes: adjusting the hyperparameters of the two regularization constraint terms, the balance constraint and the discriminant constraint in the objective function through the Bayesian parameter adjustment method.
[0019] Furthermore, it also includes: performing a first sampling operation on the low-dimensional feature distribution corresponding to each data to be processed by a reparameterized sampling method to obtain N low-dimensional data. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the specific embodiments of the present invention, the following will briefly introduce the drawings required for the specific embodiments or the prior art description. In all the drawings, each element or part is not necessarily drawn according to the actual scale.
[0021] Figure 1 A schematic diagram of the structure of a deep clustering network provided by an embodiment of the present invention;
[0022] Figure 2 A schematic diagram of the projection of the feature distribution of the deep clustering algorithm and other clustering algorithms provided in an embodiment of the present invention on a two-dimensional plane. DETAILED DESCRIPTION
[0023] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.
[0024] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.
[0025] In one embodiment, if Figure 1 As shown, a deep clustering network is provided, including an input layer, an encoding layer, a clustering layer and an output layer, wherein:
[0026] 1. Input layer, used to receive N data to be processed;
[0027] The data to be processed can be image data, text data, or any vectorized data such as single-cell mRNA sequencing data.
[0028] 2. The coding layer is used to calculate the N data to be processed respectively to obtain the low-dimensional feature distribution corresponding to each data to be processed, and perform the first sampling operation on the low-dimensional feature distribution corresponding to each data to be processed to obtain N low-dimensional data, each low-dimensional data corresponds to each data to be processed one by one;
[0029] The coding layer is a multi-layer network, and the function of the coding layer is to learn the low-dimensional feature distribution corresponding to each data to be processed (learn the parameters such as the mean and variance of the low-dimensional feature distribution), and then randomly sample from the low-dimensional feature distribution corresponding to each data to be processed to obtain N low-dimensional data. Preferably, the first sampling operation can be performed on the low-dimensional feature distribution corresponding to each data to be processed by a reparameterized sampling method to obtain N low-dimensional data.
[0030] Since the low-dimensional feature distribution of each low-dimensional data is the same as the corresponding low-dimensional feature distribution of the data to be processed, it can be considered that the two are of the same category.
[0031] The coding layer can use the idea and implementation of the variational autoencoder network coding part to calculate N data to be processed respectively, and obtain the low-dimensional feature distribution corresponding to each data to be processed. The difference is that the deep clustering network of the present invention and the variational autoencoder network coding part have different distribution assumptions of low-dimensional data.
[0032] Preferably, the distribution assumption of the present invention is to assume that N low-dimensional data obey a normal distribution with k means as clustering layer network parameters and a variance of 1. The deep clustering network of the present invention will calculate the received N data to be processed respectively to obtain the low-dimensional feature distribution corresponding to each data to be processed. The low-dimensional feature distribution corresponding to each data to be processed should belong to one of the k low-dimensional feature distributions, and then perform the first sampling operation on the low-dimensional feature distribution corresponding to each data to be processed to obtain N low-dimensional data. This method unifies the distribution parameters of N low-dimensional data in the network, that is, the clustering layer network parameters are regarded as the distribution parameters of N low-dimensional data. This design can reduce parameters and achieve collaborative optimization while also conforming to the idea of clustering, such as the idea of K-means: calculating the distance from the data to the center of the category to determine which category the data belongs to.
[0033] The distribution assumption of the encoding part of the variational autoencoder network is that the N low-dimensional data all obey a single standard normal distribution with a mean of 0 and a variance of 1. That is, the variational autoencoder network will only perform the first sampling operation in the same low-dimensional feature distribution (a standard normal distribution with a mean of 0 and a variance of 1), and cannot effectively distinguish low-dimensional data of different categories. This method contradicts the idea of clustering.
[0034] The present invention regards k means as k clustering layer network parameters, that is, the clustering layer network parameters are regarded as the distribution parameters of N low-dimensional data. This design can achieve collaborative optimization of the entire deep clustering network, and finally obtain a deep clustering network with more stable performance and better clustering effect.
[0035] 3. Clustering layer, which is used to calculate N low-dimensional data respectively to obtain the cluster category distribution corresponding to each low-dimensional data, and perform a second sampling operation on the cluster category distribution corresponding to each low-dimensional data by non-maximum suppression sampling method to obtain the clustering results of N low-dimensional data, which correspond one-to-one to the clustering results of each data to be processed;
[0036] Preferably, the clustering layer is a fully connected network layer, which is used to implement the clustering task.
[0037] Preferably, it also includes defining an objective function of a deep clustering network, wherein the objective function of the deep clustering network includes clustering terms, balance constraints, and discrimination constraints.
[0038] Preferably, the clustering term is obtained by analyzing the general derivation of the objective function:
[0039] The general derivation of the objective function is represented by the following expression:
[0040]
[0041] Among them, x is the data to be processed, KL represents the Kullback-Leibler divergence, z represents the low-dimensional data, q(z|x) represents the low-dimensional feature distribution corresponding to the data to be processed, c represents the category of the data to be processed, p(z|c) represents the distribution obeyed by the low-dimensional data (continuing from the previous text, it can be assumed here that the low-dimensional data obeys a normal distribution with k means as the clustering layer network parameters and a variance of 1), q(c|z) represents the cluster category distribution corresponding to the low-dimensional data, N is the total number of data to be processed, and D = ∫ z ∏ c p(z|c) q(c|z) dz.
[0042] Specifically, different distribution assumptions can be substituted into the above general derivation of the objective function to obtain specific clustering items through analysis.
[0043] For example, the present invention assumes that N low-dimensional data obey a normal distribution with k means as clustering layer network parameters and a variance of 1. Substituting this distribution assumption into the above general derivation of the objective function can obtain a specific clustering term expression:
[0044]
[0045] in: μ represents the mean of each data to be processed, θ represents the cluster center, i.e., the network parameter of the clustering layer, σ represents the variance of each data to be processed, r represents the data dimension, i.e., the constant, E q Express expectations.
[0046] Preferably, in the absence of prior knowledge, it is reasonable to assume that the number of samples contained in each category of the overall data is roughly the same. The balance constraint in the deep clustering network is to constrain the clustering result to be optimized in the direction of a relatively average number of samples contained in each category, and interact with the clustering terms to avoid model performance degradation. For a single sample data, the deep clustering network uses a k-dimensional vector to represent the clustering category distribution, and each element in the vector represents the probability of belonging to a certain category. In the optimization, the discriminant constraint is to further increase the gap between the values of the elements in a reasonable optimization direction to achieve a clear category orientation.
[0047] Continuing from the above, specifically, the objective function is clustering term expression + balance constraint expression + discriminant constraint expression.
[0048] The equilibrium constraint expression is as follows:
[0049]
[0050] in, It's entropy.
[0051] Calculation formula:
[0052]
[0053] in:
[0054]
[0055]
[0056] Discriminant constraint expression:
[0057]
[0058] Calculation formula:
[0059]
[0060] Among them, p j is a vector, p ij is one of the elements.
[0061] Finally, the parameters of the entire deep clustering network are optimized through back propagation, that is, the objective function is differentiated to find the direction in which the objective function value becomes smaller, and then the parameters of the deep clustering network are updated in the direction of decrease at a reasonable learning rate until the objective function converges. The parameters of the deep clustering network are no longer updated, and the clustering category distribution corresponding to each of the N low-dimensional data can be calculated through the parameters of the entire deep clustering network. Finally, the clustering category distribution corresponding to each low-dimensional data is subjected to a second sampling operation through the non-maximum suppression sampling method, and the clustering results of the N low-dimensional data are output, which correspond one to one to the clustering results of each data to be processed.
[0062] The present invention does not have a data generation network when performing clustering tasks, reduces parameters, avoids the Monte Carlo sampling method used in the generation network, and enhances the stability of the clustering effect; the present invention has a clustering layer, realizes collaborative optimization, and improves performance. Preferably, the hyperparameters of the two regularization constraints (balance constraint and discriminant constraint) in the objective function are adjusted by the Bayesian parameter adjustment method.
[0063] 4. The output layer is used to output the clustering results of N data to be processed.
[0064] In one embodiment, the deep clustering network of the present invention is constructed using the programming language Python and the deep learning framework Pytorch.
[0065] The present invention is an improvement based on VAE (Variational Autoencoder), and for the first time proposes to obtain the final clustering result by sampling the cluster category distribution.
[0066] like Figure 2, demonstrating the superiority of the present invention over general clustering algorithms in feature space distribution. This experiment randomly selects 10,000 samples from the handwriting dataset MNIST and uses the T-SNE algorithm to perform two-dimensional visualization of its spatial distribution. It is not difficult to find that: in the original space, the data of each category is difficult to distinguish, such as Figure 2 (a); After extracting features through VAE (Variational Autoencoder), data of different categories have a preliminary clustering in the feature space, such as Figure 2 (b) After clustering by the VaDE algorithm (a deep learning model that combines variational autoencoders and clustering methods) based on VAE, data of different categories are clearly distinguished in the feature space, such as Figure 2 (c); The algorithm adopted by the deep clustering network of the present invention, due to its reasonable distribution assumption and parameter setting, achieves a feature distribution in which the same category is more clustered and different categories are more separated, such as Figure 2 (d), such features are obviously more beneficial to the subsequent clustering task.
[0067] The algorithm adopted by the deep clustering network of the present invention has two samplings: one is to perform a first sampling on the low-dimensional feature distribution of the data to be processed to obtain low-dimensional data z, and the other is to perform a second sampling on the clustering category distribution of the low-dimensional data z to obtain the clustering result c.
[0068] The present invention reduces the time required for running the existing clustering algorithm, improves the clustering performance, and enhances the stability of the clustering algorithm performance.
[0069] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of the claims and specification of the present invention.
Claims
1. A deep clustering network, characterized in that: It includes input layer, encoding layer, clustering layer and output layer, where: The input layer is used to receive N data to be processed; The encoding layer is used to calculate the N data to be processed respectively to obtain the low-dimensional feature distribution corresponding to each data to be processed, and perform a first sampling operation on the low-dimensional feature distribution corresponding to each data to be processed to obtain N low-dimensional data, each low-dimensional data corresponds to each data to be processed one by one; The clustering layer is used to calculate the N low-dimensional data respectively to obtain the cluster category distribution corresponding to each low-dimensional data, and perform a second sampling operation on the cluster category distribution corresponding to each low-dimensional data by a non-maximum suppression sampling method to obtain the clustering results of the N low-dimensional data, and the clustering result of each low-dimensional data corresponds one-to-one to the clustering result of each data to be processed; The output layer is used to output the clustering results of N data to be processed.
2. A deep clustering network as claimed in claim 1, characterized in that: Also includes: An objective function of the deep clustering network is defined, wherein the objective function of the deep clustering network includes a clustering term, a balance constraint, and a discrimination constraint.
3. A deep clustering network as claimed in claim 2, characterized in that: Also includes: The clustering term is obtained by analyzing the general derivation of the objective function. The general derivation of the objective function is represented by the following expression: Where x is the data to be processed, KL represents the Kullback-Leibler divergence, z represents the low-dimensional data, q(z|x) represents the low-dimensional feature distribution corresponding to the data to be processed, c represents the category of the data to be processed, p(z|c) represents the distribution obeyed by the low-dimensional data, q(c|z) represents the cluster category distribution corresponding to the low-dimensional data, N is the total number of data to be processed, and D = ∫ z ∏ c p(z|c) q(c|z) dz.
4. A deep clustering network as claimed in claim 3, characterized in that: The N low-dimensional data obey a normal distribution with k means being the clustering layer network parameters and a variance of 1.
5. A deep clustering network as claimed in claim 4, characterized in that: Also includes: The hyperparameters of the two regularization constraints, the balance constraint and the discrimination constraint in the objective function, are adjusted by the Bayesian parameter adjustment method.
6. A deep clustering network as claimed in claim 5, characterized in that: Also includes: The first sampling operation is performed on the low-dimensional feature distribution corresponding to each data to be processed by the reparameterized sampling method to obtain N low-dimensional data.