Clustering method and apparatus based on variational autoencoder

Abstract

The application discloses a clustering method and device based on a variational autoencoder, and the method comprises the following steps: acquiring input data x; acquiring corresponding discrete category hidden variable y and continuous Gaussian hidden variable z by combining the input data x with the variational autoencoder; acquiring corresponding joint probability distribution q(z, y|x) by combining the input data x, the discrete category hidden variable y and the continuous Gaussian hidden variable z with the variational autoencoder; and determining the category of the input data x according to the probability distribution of the discrete category hidden variable y by directly decomposing the joint probability distribution q(z, y|x) by using the variational autoencoder. The method is improved on the basis of the variational autoencoder, the mean field approximation is abandoned, the probability distribution of the discrete category hidden variable is acquired by directly decomposing the joint probability distribution, and then the category of the input data is determined, so that the clustering effect and accuracy are improved.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese patent application number 202011165489.X, filed by Tsinghua University on October 27, 2020, entitled "Clustering Method and Apparatus Based on Variational Autoencoder". Technical Field

[0003] This application relates to the field of deep learning technology, and in particular to a clustering method and apparatus based on variational autoencoders. Background Technology

[0004] Currently, unsupervised clustering is widely used in various practical fields, such as image or image feature clustering. Related technologies utilize variational autoencoder models for data clustering. For example, LTVAE greedily learns the tree structure between latent variables using a tree model on the latent variables, and then uses this complex prior-constrained variational autoencoder model for clustering. GMVAE and VaDE both use mean-field approximation to handle the relationships between latent variables to achieve data clustering. However, LTVAE uses a complex tree model to iteratively optimize the structural relationships between latent variables, resulting in poor clustering performance. GMVAE and VaDE use mean-field approximation, but this approximation may not hold true in real-world data, leading to lower clustering accuracy. Summary of the Invention

[0005] This application proposes a clustering method and apparatus based on variational autoencoders. This method improves upon variational autoencoders by abandoning the mean-field approximation and using direct decomposition of the joint probability distribution to obtain the probability distribution of discrete class latent variables, thereby determining the class of the input data and improving the clustering effect and accuracy.

[0006] The first aspect of this application proposes a clustering method based on a variational autoencoder, comprising: acquiring input data x; acquiring corresponding discrete categorical latent variables y and continuous Gaussian latent variables z based on the input data x and the variational autoencoder; acquiring a joint probability distribution q(z,y|x) by combining the input data x, the discrete categorical latent variables y, and the continuous Gaussian latent variables z with the variational autoencoder; directly decomposing the joint probability distribution q(z,y|x) using the variational autoencoder, and determining the category of the input data x based on the probability distribution of the discrete categorical latent variables y.

[0007] The clustering method based on variational autoencoders in this application involves: acquiring input data x; obtaining the corresponding discrete categorical latent variable y and continuous Gaussian latent variable z based on the input data x and a variational autoencoder; obtaining the joint probability distribution q(z,y|x) by combining the input data x, the discrete categorical latent variable y, and the continuous Gaussian latent variable z with the variational autoencoder; directly decomposing the joint probability distribution q(z,y|x) using the variational autoencoder; and determining the category of the input data x based on the probability distribution of the discrete categorical latent variable y. This method improves upon the variational autoencoder by abandoning the mean-field approximation and using direct decomposition of the joint probability distribution to obtain the probability distribution of the discrete categorical latent variable, thereby determining the category of the input data and improving the clustering effect and accuracy.

[0008] A second aspect of this application proposes a clustering device based on a variational autoencoder, comprising: a first acquisition module for acquiring input data x; a second acquisition module for combining the input data x with a variational autoencoder to acquire corresponding discrete categorical latent variables y and continuous Gaussian latent variables z; a third acquisition module for combining the input data x, the discrete categorical latent variables y, and the continuous Gaussian latent variables z with the variational autoencoder to acquire a corresponding joint probability distribution q(z,y|x); and a clustering module for directly decomposing the joint probability distribution q(z,y|x) using the variational autoencoder and determining the category of the input data x based on the probability distribution of the discrete categorical latent variables y.

[0009] The clustering device based on a variational autoencoder in this application embodiment acquires input data x; combines the input data x with a variational autoencoder to obtain the corresponding discrete categorical latent variable y and continuous Gaussian latent variable z; combines the input data x, the discrete categorical latent variable y, and the continuous Gaussian latent variable z with the variational autoencoder to obtain the corresponding joint probability distribution q(z,y|x); and directly decomposes the joint probability distribution q(z,y|x) using the variational autoencoder, determining the category of the input data x based on the probability distribution of the discrete categorical latent variable y. This device improves upon the variational autoencoder by abandoning the mean-field approximation and using direct decomposition of the joint probability distribution to obtain the probability distribution of the discrete categorical latent variable, thereby determining the category of the input data, thus improving the clustering effect and accuracy.

[0010] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0011] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0012] Figure 1 This is a flowchart illustrating a clustering method based on a variational autoencoder according to an embodiment of this application;

[0013] Figure 2 This is a schematic diagram of the model structure of a variational autoencoder according to an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of a clustering device based on a variational autoencoder according to an embodiment of this application. Detailed Implementation

[0015] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0016] The following description, with reference to the accompanying drawings, describes a clustering method and apparatus based on variational autoencoders according to embodiments of this application. It should be noted that the execution entity of the clustering method based on variational autoencoders in these embodiments is a clustering apparatus based on variational autoencoders. This clustering apparatus can be applied to any computer device, enabling the computer device to perform clustering functions based on variational autoencoders. The computer device can be a personal computer (PC), cloud device, mobile device, server, etc. Mobile devices can be, for example, mobile phones, tablets, personal digital assistants, wearable devices, in-vehicle devices, etc., hardware devices with various operating systems, touchscreens, and / or displays.

[0017] Figure 1 This is a flowchart illustrating a clustering method based on a variational autoencoder according to an embodiment of this application.

[0018] like Figure 1 As shown, the specific implementation process of this clustering method based on variational autoencoders is as follows:

[0019] Step 101: Obtain the input data x.

[0020] In this embodiment, data x can be multiple feature data in the image or original image data (original pixel value matrix), for example, data x = {x1, x2, ... x} nAs an example, feature extraction can be performed on an image to obtain feature data from the image, and this feature data can be used as input data x. It should be noted that the input data x can be a random vector in a high-dimensional space.

[0021] Step 102: Combine the input data x with a variational autoencoder to obtain the corresponding discrete categorical latent variable y and continuous Gaussian latent variable z.

[0022] It's important to understand that a variational autoencoder can consist of three parts: an encoder, a hidden layer representation, and a decoder. The encoder encodes data into a hidden layer representation vector, and the decoder recovers the data from the hidden layer representation vector.

[0023] Optionally, the encoder in the variational autoencoder can encode the input data x into a hidden layer representation vector, and use this hidden layer representation vector as a discrete categorical latent variable y. It should be noted that the discrete categorical latent variable y can be understood as the specific Gaussian variable selected by the variational autoencoder model, for example, the i-th Gaussian variable.

[0024] Next, in the embodiments of this application, the encoder in the variational autoencoder can obtain the corresponding continuous Gaussian latent variable z based on the input data x and the discrete categorical latent variable y. It should be noted that the continuous Gaussian latent variable z follows a normal distribution, and the continuous Gaussian latent variable z depends on the input data x and the discrete categorical latent variable y. The continuous Gaussian latent variable z can characterize the information of the input data x.

[0025] In order to obtain accurate latent variables, in this embodiment of the application, before combining the input data x with the variational autoencoder to obtain the corresponding discrete categorical latent variable y and continuous Gaussian latent variable z, the variational autoencoder can be trained first.

[0026] As an example, Monte Carlo objectives are used as the loss function, combined with variance annealing, to train the variational autoencoder model.

[0027] In this embodiment, a Monte Carlo objective can be used as the training objective, which is a tighter variational lower bound than ELBO (the original training objective of the variational autoencoder). Using the Monte Carlo objective as the loss function and directly training the model parameters with the global gradient eliminates the need for reparameterization of the latent variables y and z. The specific formula is as follows:

[0028]

[0029] Among them, L M (x; θ, φ) represents the Monte Carlo objective of the training. q represents the expectation of the joint sample obtained from sampling. φ(z,y|x) represents the learnable posterior distribution, p θ (x,y (m) ,z (m) ) and q φ (z (m) ,y (m) |x) represents the probability density obtained after the m-th group of (y, z) joint samples.

[0030] It should be noted that model training ends when the loss function value remains almost unchanged. Typically, the model is trained for a given number of rounds (e.g., 500), and the optimal model is selected using early stopping based on the loss function value on the validation set.

[0031] To stabilize model training and achieve optimal clustering results, variance annealing can be used to stabilize model training, thereby obtaining the best clustering results. That is, in the early stages of training, a large lower bound is added to the variance of the Gaussian posterior distribution to limit its probability density, thus preventing the Monte Carlo objective from becoming too large when the variance of the importance sampling estimator is large (leading to loss function divergence). As model training progresses, this lower bound is gradually shrunk to a smaller constant value, thus ensuring that the overall training objective remains unchanged. Specifically, this can be expressed by the following formula: σ' = SoftPlus(σ) + ε, where σ' and σ are the new variance and the original variance, respectively, and ε is the shrinkable lower bound.

[0032] It's worth noting that variance annealing not only significantly stabilizes model training but also helps the model learn more separable implicit representations (especially for raw image data). Ultimately, the clustering task can be completed by determining the category of each input x using the discrete categorical latent variable y corresponding to the largest q(y|x).

[0033] For example, y is a discrete categorical latent variable, such as y belonging to {1,…,k}. For each category y, q(y|x) is a probability density value. Since the number of categories k is finite, the largest value q(y=y0|x) can be selected from the corresponding q(y|x), then y0 is the category corresponding to the input data x.

[0034] Furthermore, it should be noted that occasional poor initialization may affect the final performance of the model. In this embodiment, a simple pre-training initialization can be used to address this issue. As an example, a simple standard autoencoder (with the same network structure as the variational autoencoder model) can be pre-trained, followed by training a Gaussian mixture model on the hidden layers of the standard autoencoder. The network parameters of the autoencoder are used to initialize the corresponding network layers in the variational autoencoder model, and the means of each distribution of the Gaussian mixture model are used to initialize the means of the Gaussian mixture prior in the variational autoencoder model (instead of random initialization). The model initialized using this method can achieve more stable clustering results.

[0035] Step 103: Combine the input data x, the discrete categorical latent variable y, and the continuous Gaussian latent variable z with a variational autoencoder to obtain the corresponding joint probability distribution q(z,y|x).

[0036] It is necessary to understand that, such as Figure 2 As shown, the variational autoencoder can use two neural networks to establish two probability models: one is used for variational inference of the input data to obtain the probability distribution of the latent variables, called the variational model (network parameter φ); the other is used to reconstruct the approximate probability distribution of the generated input data based on the probability distribution of the latent variables, called the generative model (network parameter θ).

[0037] In the embodiments of this application, such as Figure 2 As shown, the input data x, discrete categorical latent variable y, and continuous Gaussian latent variable z are used to obtain the joint probability distribution q(z,y|x) of latent variables z and y through the variational model corresponding to the variational autoencoder.

[0038] Step 104: Use a variational autoencoder to directly decompose the joint probability distribution q(z,y|x), and determine the category of the input data x based on the probability distribution of the discrete categorical latent variable y.

[0039] In this embodiment of the application, the joint probability distribution q(z,y|x) in the variational autoencoder can be decomposed using the following formula:

[0040] q(z,y|x)=q(z|y,x)q(y|x)

[0041] Here, q(y|x) represents the probability distribution of the discrete categorical latent variable y, and q(z|y,x) represents the probability distribution of the continuous Gaussian latent variable z. It should be noted that q(z,y|x) = q(z|y,x)q(y|x) is an exact decomposition in the probabilistic sense, requiring no additional approximation. Regardless of whether z and x are conditionally independent given x, this decomposition always accurately represents all possible joint probability distributions q(z,y|x).

[0042] It should be noted that, in the embodiments of this application, a joint probability distribution q(z,y|x) can be used to approximate the true posterior probability distribution p(z,y|x). Since the overall prior probability distribution p(z,y) is a combination of several Gaussian probability distributions, each Gaussian probability distribution corresponds to a category. In the variational autoencoder model, the KL divergence term in the loss function constrains the posterior probability distribution q(z,y) to approximate the prior probability distribution p(z,y), thereby forming a separate representation of the posterior probability distribution q(z,y) in the latent space. The z corresponding to each category x follows a Gaussian probability distribution, thus forming a cluster in the latent space. Since z is the implicit feature expression of each category of data, and the discrete category variable y can represent each data category, the category of the input data x can be determined according to the probability distribution of the discrete category latent variable y.

[0043] The clustering method based on variational autoencoders in this application involves: acquiring input data x; combining the input data x with a variational autoencoder to obtain the corresponding discrete categorical latent variable y and continuous Gaussian latent variable z; combining the input data x, the discrete categorical latent variable y, and the continuous Gaussian latent variable z with the variational autoencoder to obtain the corresponding joint probability distribution q(z,y|x); directly decomposing the joint probability distribution q(z,y|x) using the variational autoencoder, and determining the category of the input data x based on the probability distribution of the discrete categorical latent variable y. This method improves upon the variational autoencoder by using a non-reparameterization method and variance annealing for model training, abandoning the mean-field approximation, and directly decomposing the joint probability distribution to obtain the probability distribution of the discrete categorical latent variable, thereby determining the category of the input data. This significantly improves the clustering effect and accuracy while reducing the model training overhead.

[0044] To achieve the above embodiments, this application also proposes a clustering device based on a variational autoencoder.

[0045] Figure 3 This is a schematic diagram of a clustering device based on a variational autoencoder according to an embodiment of this application.

[0046] like Figure 3 As shown, the clustering device 300 based on variational autoencoder includes: a first acquisition module 310, a second acquisition module 320, a third acquisition module 330, and a clustering module 340.

[0047] The system comprises the following modules: a first acquisition module 310 for acquiring input data x; a second acquisition module 320 for combining the input data x with a variational autoencoder to acquire the corresponding discrete categorical latent variable y and continuous Gaussian latent variable z; a third acquisition module 330 for combining the input data x, the discrete categorical latent variable y, and the continuous Gaussian latent variable z with a variational autoencoder to acquire the corresponding joint probability distribution q(z,y|x); and a clustering module 340 for decomposing the joint probability distribution q(z,y|x) using a variational autoencoder and determining the category of the input data x based on the probability distribution of the discrete categorical latent variable y.

[0048] As one possible implementation of this application, the joint probability distribution q(z,y|x) is decomposed using a variational autoencoder, specifically expressed as follows:

[0049] q(z,y|x)=q(z|y,x)q(y|x)

[0050] Where q(y|x) represents the probability distribution of the discrete categorical latent variable y, and q(z|y,x) represents the probability distribution of the continuous Gaussian latent variable z.

[0051] As one possible implementation of this application, the clustering device 300 based on the variational autoencoder further includes: a training module, used to train the variational autoencoder model by using a Monte Carlo target as a loss function and combining variance annealing.

[0052] As one possible implementation of this application, a Monte Carlo target is used as the loss function, specifically expressed as follows:

[0053]

[0054] Among them, L M (x; θ, φ) represents the Monte Carlo objective of the training. Let q represent the expectation of the joint sample obtained from sampling. φ (z,y|x) represents the learnable posterior distribution, p θ (x,y (m) ,z (m) ) and q φ (z (m) ,y (m) |x) represents the probability density of the m-th group (y, z) joint sample.

[0055] The clustering device based on variational autoencoder in this application embodiment acquires input data x; combines the input data x with a variational autoencoder to obtain the corresponding discrete categorical latent variable y and continuous Gaussian latent variable z; combines the input data x, discrete categorical latent variable y, and continuous Gaussian latent variable z with the variational autoencoder to obtain the corresponding joint probability distribution q(z,y|x); and directly decomposes the joint probability distribution q(z,y|x) using the variational autoencoder, determining the category of the input data x based on the probability distribution of the discrete categorical latent variable y. This device improves upon the variational autoencoder by using a non-reparameterization method and variance annealing for model training, abandoning the mean-field approximation, and directly decomposing the joint probability distribution to obtain the probability distribution of the discrete categorical latent variable, thereby determining the category of the input data. This significantly improves the clustering effect and accuracy while reducing model training overhead.

[0056] To implement the above embodiments, this application also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the clustering method based on variational autoencoders as proposed in the foregoing embodiments of this application.

[0057] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the clustering method based on variational autoencoders as proposed in the foregoing embodiments of this application.

[0058] To implement the above embodiments, this application also proposes a computer program product that, when the instruction processor in the computer program product is executed, performs the clustering method based on variational autoencoder as proposed in the above embodiments of this application.

[0059] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0061] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0062] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0063] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0064] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0065] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0066] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method of image clustering based on variational autoencoder, characterized in that, The method comprises the following steps: acquiring an image, feature data extracted from the image, or an original pixel value matrix of the image as input data x; acquiring a corresponding discrete category latent variable y by combining the input data x with a trained variational autoencoder, and acquiring a corresponding continuous Gaussian latent variable z by combining the input data x and the discrete category latent variable y with the trained variational autoencoder, wherein a Monte Carlo target is used as a loss function, and a variance annealing is combined to train the variational autoencoder to obtain the trained variational autoencoder; acquiring a corresponding joint probability distribution q(z, y|x) by combining the input data x, the discrete category latent variable y, and the continuous Gaussian latent variable z with the trained variational autoencoder; directly decomposing the joint probability distribution q(z, y|x) by using the trained variational autoencoder, and determining the category of the input data x according to the probability distribution of the discrete category latent variable y, wherein a formula for directly decomposing the joint probability distribution q(z, y|x) by using the trained variational autoencoder is as follows: q(z, y|x) = q(z|y, x)q(y|x) wherein q(y|x) represents the probability distribution of the discrete category latent variable y, and q(z|y, x) represents the probability distribution of the continuous Gaussian latent variable z.

2. The method of claim 1, wherein, The Monte Carlo target is used as the loss function, and a formula for the Monte Carlo target is as follows: where L M (x; 0, f) denotes the trained Monte Carlo objective, denotes the expectation over the sampled joint samples, q φ (z, y | x) denotes the learnable posterior distribution, p θ (x, y (m) , z (m) ) and q φ (z (m) , y (m) | x) denote the probability density obtained after the m-th set of (y, z) joint samples.

3. An apparatus for image clustering based on variational autoencoder, characterized in that, The method comprises the following steps: a first acquisition module is configured to acquire an image, feature data extracted from the image, or an original pixel value matrix of the image as input data x; a second acquisition module is configured to acquire a corresponding discrete category latent variable y by combining the input data x with a trained variational autoencoder, and acquire a corresponding continuous Gaussian latent variable z by combining the input data x and the discrete category latent variable y with the trained variational autoencoder, wherein a Monte Carlo target is used as a loss function, and a variance annealing is combined to train the variational autoencoder to obtain the trained variational autoencoder; a third acquisition module is configured to acquire a corresponding joint probability distribution q(z, y|x) by combining the input data x, the discrete category latent variable y, and the continuous Gaussian latent variable z with the trained variational autoencoder; a clustering module is configured to directly decompose the joint probability distribution q(z, y|x) by using the trained variational autoencoder, and determine the category of the input data x according to the probability distribution of the discrete category latent variable y, wherein a formula for directly decomposing the joint probability distribution q(z, y|x) by using the trained variational autoencoder is as follows: q(z, y|x) = q(z|y, x)q(y|x) wherein q(y|x) represents the probability distribution of the discrete category latent variable y, and q(z|y, x) represents the probability distribution of the continuous Gaussian latent variable z.

4. The apparatus of claim 3, wherein, The Monte Carlo target is used as the loss function, and a formula for the Monte Carlo target is as follows: where L M (x; 0, f) denotes the trained Monte Carlo objective, denotes the expectation over the sampled joint samples, q φ (z, y | x) denotes the learnable posterior distribution, p θ (x, y (m) , z (m) ) and q φ (z (m) , y (m) | x) denote the probability density of the m-th set of (y, z) joint samples.

Images