A deep unsupervised representation learning method based on hidden layer coefficient of variation

Abstract

The application discloses a deep unsupervised representation learning method based on a hidden layer coefficient of variation, and the input data X is subjected to a training step 1 GRBM training and n step covRBM model training in sequence, so that output data is obtained; the covRBM model is composed of a visual layer v, a hidden layer h and a reconstructed visual layer and a reconstructed hidden layer which are sequentially performed; a first target of the covRBM model training is to maximize a log-likelihood function of a visual unit in the RBM, and a second target is to maximize a coefficient of variation value corresponding to a hidden layer feature. The method of the application introduces the coefficient of variation feature value of the hidden layer into the learning process of the restricted Boltzmann machine, thereby guiding the feature distribution of the hidden layer in an unsupervised manner, overcoming the purposeless learning defect of the traditional restricted Boltzmann machine, increasing the interpretability, and improving the representation ability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a deep unsupervised representation learning method based on hidden layer coefficient of variation. BACKGROUND

[0002] In the field of machine learning, representation learning is a set of techniques that convert raw data into a form that can be effectively exploited by machine learning. The purpose of representation learning is to simplify complex raw data and extract better data representation from raw data, so that subsequent tasks can be accomplished with less effort. At present, supervised representation learning has achieved great success in the field of machine learning. However, labeled data is scarce in many practical applications. In recent years, some researches have focused on unsupervised representation learning methods. However, obtaining a proper feature distribution without any label is still a difficult point in machine learning.

[0003] Restricted Boltzmann Machine (RBM) is a popular energy-based unsupervised representation learning model, which aims to explore proper hidden features. RBM is a probability-based energy model. The standard RBM is composed of a layer of visible nodes and a layer of hidden nodes, and is also called shallow neural network due to its two-layer structure. The learning of RBM can be regarded as an encoding-decoding process. For the training sample of the visible layer input and the randomly initialized connection weight W and the visible layer bias α, the 0,1 value of the hidden layer unit is obtained through the sigmoid activation function, and this forward process is the encoding process. Conversely, the data reconstruction process from the hidden layer to the visible layer is the decoding process. For each training sample, the goal of training is to minimize the difference between the encoded and decoded visible layer output and the visible layer input. Based on this goal, the maximum likelihood function is used as the loss function, and the learning parameters of RBM are obtained through iterative optimization.

[0004] RBM is a simple neural network for unsupervised reconstruction of raw data, which only learns by mining its own characteristics without other knowledge guidance, lacks interpretability and has poor expressiveness. Therefore, it is particularly important to guide the learning process of RBM and improve the expressiveness by adding some additional strategies. SUMMARY

[0005] In view of the problem that the restricted Boltzmann machine only learns by mining its own characteristics without other knowledge guidance, lacks interpretability and has poor expressiveness, the present application provides a deep unsupervised representation learning method based on hidden layer coefficient of variation.

[0006] The application provides a deep unsupervised representation learning method based on a hidden layer coefficient of variation, which is a deep unsupervised architecture based on representation learning and is composed of one GRBM and n covRBM (coefficient of variation restricted Boltzmann machine) stacks. The visible layer (x1-x M ) is real value data, and all the hidden layers (H0-H n ) are composed of binary data. The input data X is subjected to one-step GRBM training and n-step covRBM model training in sequence, and output data is obtained. The covRBM model is the core of the application.

[0007] The covRBM model is composed of a visible layer v, a hidden layer h and a reconstructed visible layer A reconstructed hidden layer in sequence. The first target of the covRBM model training is to maximize the log-likelihood function of the visible units in the RBM, and the second target is to maximize the coefficient of variation value corresponding to the hidden layer features.

[0008] The objective function of the covRBM model is as follows:

[0009]

[0010] In the formula, θ is a model parameter, represents is the visible layer data of the covRBM model, v i represents the i-th row vector of , N represents the number of samples, is the hidden layer data of the covRBM model, h k represents the k-th hidden feature, represents the coefficient of variation value corresponding to the k-th hidden feature, λ is a proportional coefficient in (0, 1), and is used to describe the proportion of the two training targets in the loss function.

[0011] The coefficient of variation value corresponding to the k-th hidden feature is calculated according to the following formula:

[0012]

[0013] In the formula, h k =h k1 , h k2 ,…, h kj ,…, h kN ] represents the k-th hidden feature with N samples, and h kj represents the j-th sample in the k-th hidden feature.

[0014] The problem of learning covRBM model is to obtain the optimal or near-optimal parameters. For the first learning objective, the contrastive divergence can be used to approximate the gradient of log-likelihood. For the second learning objective, the gradient descent method is used to solve the optimization problem, i.e., to solve the composite function with respect to the model parameters θ = {w ij , α i , β j}. Here, w ij denotes the connection weights between v i and h j , v i denotes the i-th visible unit, h j denotes the j-th hidden unit; α i denotes the bias of the i-th visible unit, and β j denotes the bias of the j-th hidden unit.

[0015] Based on this, the update rules of covRBM model parameters are as follows:

[0016]

[0017]

[0018] Δα i = λε(<v i > data - <v i > recon )

[0019]

[0020] where λ ∈ (0, 1) is a proportionality coefficient to characterize the proportion of the two training objectives in the loss function; ε denotes the learning rate; <~> denotes the expected distribution, data denotes the forward process, and recon denotes the reconstruction process; h kj denotes the j-th sample in the k-th hidden feature; and v ki denotes the i-th sample in the k-th visible feature.

[0021] The execution process of a piece of data v d in the dataset v in the covRBM model is as follows:

[0022] (1) Forward process: data v d = [v d1 , v d2 , …, v dj , …, v dN ] is transmitted into the RBM as the visible layer, and the operation is performed to obtain the hidden layer

[0023] (2) data reconstruction: the hidden layer data The sigmoid (hW + a) operation is sequentially performed to obtain the reconstructed data

[0024] (3) coefficient of variation guided learning parameter process: calculate the coefficient of variation value of the hidden layer data And maximize it, that is, calculate the change value of the parameters Δω ij , Δβ j And Δα i And update the parameters;

[0025] Repeat steps (1)-(3) above until the loss function of the covRBM model converges, that is, the training is completed.

[0026] Compared with the prior art, the present application has the advantages of:

[0027] The present application incorporates the eigenvalue of the coefficient of variation of the hidden layer into the learning process of the restricted Boltzmann machine, thereby guiding the feature distribution of the hidden layer in an unsupervised manner, overcoming the shortcomings of the traditional restricted Boltzmann machine without purpose learning, increasing the interpretability, and improving the representation ability of the model.

[0028] The learning method of the present application has significant advantages over the original data and the shallow GRBM model on nine data sets, which reflects the effectiveness and superiority of the method of the present application.

[0029] Other advantages, objects and features of the present application will be partially embodied by the following description, and will be partially understood by those skilled in the art through research and practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The model structure diagram of the deep unsupervised representation learning based on the coefficient of variation of the hidden layer of the present application.

[0031] Figure 2 The training schematic diagram of the GRBM based on the CD1 algorithm. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0033] The model structure of the deep unsupervised representation learning based on the coefficient of variation of the hidden layer provided by the present application is shown in Figure 1It consists of one layer of GRBM and n layers of covRBM stacked together. The output data can be obtained by training the input data X through one-step GRBM training and n-step covRBM training.

[0034] Step 1 GRBM training:

[0035] The special thing of GRBM (Gaussian-Bernoulli RBM) compared with RBM is that the data of its visible layer is composed of real number nodes, while the hidden layer is still composed of binary nodes. The specific method is to increase the bias value of v 2 . The energy function of GRBM is expressed as:

[0036]

[0037] Where θ i represents the standard deviation of the i-th node.

[0038] Correspondingly, if the visible node v i is known, the conditional probability of the hidden layer node h k is expressed as follows:

[0039]

[0040] Given the hidden node h j , the conditional probability of the visible layer node v k can be expressed as:

[0041]

[0042] Where N(μ, θ 2 ) represents the normal distribution with mean μ and variance θ 2 . With the introduction of GRBM model, the scalability of RMB itself has been greatly improved, and the application scenarios of RBM have also been expanded.

[0043] The training of GRBM is completed through the contrastive divergence algorithm (CD). The CD algorithm seeks to reconstruct the input data in the visible layer to the greatest extent, which has been proved to be the most effective and stable learning algorithm for training RBM. At the same time, it is also an unsupervised and generative learning method. One-step CD (CD1) is a special case of CD algorithm, which is also the simplest and most efficient method. The principle diagram of using CD1 to train GRBM is shown in Figure 2 In each iteration process, CD1 minimizes the error between the input data and the reconstructed data through one-step data process and one-step reconstruction process, so as to learn the model parameters θ={w ij , αi ,β j}. Among them, w ij Indicates v i and h j The connection weight between i represents the i-th visual unit, h j represents the jth hidden unit; α i represents the bias of the i-th visual unit, β j represents the bias of the jth hidden unit.

[0044] In the covRBM model, it is assumed that is the visual layer data of the covRBM model, where v k is the kth row vector of ν, i.e. v k =[v k1 ,v k2 ,…,v kj ,…,v kN ] represents the kth visual feature with N samples, and M represents the number of samples. Similarly, Represents the hidden layer data of the covRBM model, where h k represents the kth hidden feature (h k =[h k1 ,h k2 ,…,h kj ,…,h kN ]).

[0045] A data item v in the dataset v d The execution steps in the covRBM model are as follows:

[0046] (1) Forward process: data v d =[v d1 ,v d2 ,…,v dj ,…,v dN ] is passed into RBM as a visual layer, and Operation to obtain hidden layer

[0047] (2) Data reconstruction: hidden layer data Perform sigmoid (hW + α) operations in sequence to obtain reconstructed data

[0048] (3) The coefficient of variation guides the learning parameter process: Calculate the coefficient of variation value of the hidden layer data And maximize it, that is, calculate the parameter change value Δω through the formula ij , Δβ j and Δα i And update the parameters;

[0049] Repeat steps (1)-(3) above until the loss function of the covRBM model converges, i.e., the training is completed.

[0050] The original data (RD), the standard Gaussian restricted Boltzmann machine (GRBM) and the deep unsupervised representation learning method based on the hidden layer coefficient of variation (DURL) of the present application are trained in sequence on the Microsoft Asia Research Institute Multimedia Dataset (MSRA-MM), and then the training results are subjected to K-means clustering and the precision and recall two evaluation indexes are calculated to verify the performance of the three methods. The performance comparison results of the three are shown in Table 1.

[0051] Table 1, Performance comparison results of RD, GRBM and DURL (%)

[0052]

[0053] As can be seen from the experimental results in Table 1, the deep unsupervised representation learning method based on the hidden layer coefficient of variation of the present application has significant advantages over the original data and the shallow GRBM model on the nine data sets, which reflects the effectiveness and superiority of the method of the present application.

[0054] The above description is only the preferred embodiments of the present application, and does not limit the present application in any form. Although the present application has been disclosed as the above preferred embodiments, it is not intended to limit the present application. Any skilled person in the art can make some changes or modifications to the above disclosed technical contents without departing from the scope of the technical solution of the present application, and any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the present application still belong to the scope of the technical solution of the present application.

Claims

1. A deep unsupervised representation learning method based on hidden layer coefficient of variation, characterized in that, For processing of image data, specifically: input data X is subjected to training 1-step GRBM training and n-step covRBM model training in sequence, i.e. output data is obtained; the covRBM model is a core part of the method, and the covRBM model is composed of a visual layer v, a hidden layer h and a reconstructed visual layer reconstructed hidden layer ; a first target of the covRBM model training is to maximize a log-likelihood function of visual units in the RBM, and a second target is to maximize a coefficient of variation value corresponding to features of the hidden layer; The objective function of the covRBM model is as follows: where θ is the model parameter, is the visible layer data of covRBM model, v i represents is the i-th row vector of X, N denotes the number of samples, represents the hidden layer data of covRBM model, h k represents the k-th hidden feature vector, represents the coefficient of variation value corresponding to the k-th hidden feature, λ ∈ (0, 1) is a proportionality coefficient, used to depict the proportion of the two training targets in the loss function.

2. The deep unsupervised representation learning method based on hidden layer coefficient of variation according to claim 1, wherein, The coefficient of variation value corresponding to the kth hidden feature The calculation formula is as follows: where h k = [h k1 ,h k2 ,…,h kj ,…,h kN ] represents the kth hidden feature with N samples, represents the value obtained by summing all elements in the h k vector, h kj represents the jth sample in the kth hidden feature.

3. The deep unsupervised representation learning method based on hidden layer coefficient of variation according to claim 2, wherein, The first objective maximizes the log-likelihood function of the visible units in the RBM by using the contrastive divergence to approximate the gradient of the log-likelihood.

4. The deep unsupervised representation learning method based on hidden layer coefficient of variation according to claim 2, wherein, The second objective maximizes the coefficient of variation of the hidden layer features, which is solved using gradient descent, i.e., solving the function The partial differential equations for the model parameters θ = {w ij , α i , β j}, where w ij represents the connection weights between v i and h j , v i represents the i-th visible unit, h j represents the j-th hidden unit; α i represents the bias of the i-th visible unit, and β j represents the bias of the j-th hidden unit.

5. The deep unsupervised representation learning method based on hidden layer coefficient of variation according to claim 4, wherein, The update rule of the covRBM model parameters is as follows: where λ ∈ (0, 1) is a proportionality coefficient to characterize the proportion of two training objectives in the loss function; ε represents the learning rate; <~> represents the expected distribution, data represents the forward process, and recon represents the reconstruction process; h kj represents the jth sample in the kth hidden feature; v ki represents the ith sample in the kth visual feature.

6. The deep unsupervised representation learning method based on hidden layer coefficient of variation according to claim 5, wherein, A piece of data v in the dataset v d The execution process in the covRBM model is as follows: (1) Forward process: data v d = [v d1 ,v d2 ,…,v dj ,…,v dN ] as visible layer into RBM, and operation to get hidden layer (2) Data reconstruction: on hidden layer data Sigmoid (hW + a) operation is performed in sequence to obtain reconstruction data (3) Variance coefficient guided learning parameter process: calculate the variance coefficient value of the hidden layer data And maximize it, that is, calculate the change value Δω of the parameters by the formula ij , Δβ j And Δα i And update the parameters; Repeat steps (1)-(3) above until the loss function of the covRBM model converges, i.e., the training is complete.

Images