A self-supervised deep representation method based on contrastive learning

Abstract

The application discloses a self-supervised deep representation method based on contrast learning, comprising the following steps: S1, obtaining a data set V to be trained; S2, training N pieces of data in one batch, and performing data enhancement on each piece of data v; S3, feeding the data v and the enhanced data vs into a restricted Boltzmann machine for forward propagation, and obtaining corresponding data h and data hs in the hidden layer; S4, performing contrast learning to improve the similarity between positive samples; S5, performing back propagation to obtain reconstructed data v' of the data v in the visual layer; S6, minimizing the distance between the data v' and the data v; S7, repeating steps S2-S6 until the reconstruction loss function converges, and realizing the training of the restricted Boltzmann machine; and S8, using the trained restricted Boltzmann machine in S7 to perform deep representation learning on the data set V, and obtaining deep feature representation thereof. The application can overcome the purposeless learning defect of the traditional restricted Boltzmann machine, thereby improving the representation capability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to a self-supervised deep representation method based on contrastive learning. BACKGROUND

[0002] Restricted Boltzmann machine (RBM) is proposed by Geoff Hinton of the University of Toronto, which is an energy-based autoencoder that can perform feature learning. As shown in Figure 1 The restricted Boltzmann machine is a neural network composed of two layers of neurons, the upper layer of neurons forms a visible layer, and the value of the visible layer neuron is represented by a vector v. The lower layer of neurons forms a hidden layer, and the value of the hidden layer neuron is represented by a vector h. The visible layer and the hidden layer are independent of each other in the layer, and full connection is performed between the layers. The matrix W represents the connection weight of the full connection, the vector a represents the bias of the visible layer, and the vector b represents the bias of the hidden layer. The full connection between the layers is bidirectional, and the state of the visible layer acts on the hidden layer, and the state of the hidden layer also acts on the visible layer. The restricted Boltzmann machine is divided into two stages: the encoding stage and the decoding stage.

[0003] Encoding stage: feature extraction process of forward propagation, visible layer features as input, visible layer neurons v and hidden layer neurons h are multiplied by weight W, the result is accumulated and added to hidden layer bias b, and the final result is obtained by activation function sigma to get the value of hidden layer neuron h. This forward process is a coding.

[0004] Decoding stage: reconstruction process of back propagation, the features of the hidden layer become input, and the hidden layer unit h and the visible layer neuron v are multiplied by the weight W, the result is accumulated and added to the visible layer bias a to obtain v', which is an approximation of the visible layer v. This reverse process is a decoding.

[0005] Learning process: as shown in Figure 2 v' is obtained by feature extraction of forward propagation and reconstruction of back propagation, the similarity of v' and v is measured by KL divergence, the maximum likelihood function is used as the loss function, and W is continuously updated through continuous learning. Finally, the loss function converges to the vicinity of a minimum value, so that the distribution of the reconstructed v'(reconstructed feature probability distribution q(x)) and the visible layer v(original data probability distribution p(x)) gradually approaches.

[0006] Through the above learning process, the restricted Boltzmann machine can minimize the energy to make the data more stable to learn representative features. However, the restricted Boltzmann machine adopts a contrast divergence learning method to train the model, which belongs to the unsupervised learning category, that is, the learning process has no label and no other prior knowledge as a reference, and only learns by digging its own characteristics, which lacks interpretability, has poor expression ability and weak learning ability. SUMMARY

[0007] In view of the above problems, the present application aims to provide a self-supervised deep representation method based on contrast learning.

[0008] The technical scheme of the present application is as follows:

[0009] A self-supervised deep representation method based on contrast learning, comprising the following steps:

[0010] S1: obtaining a data set V to be trained; the data set V={v1, v2, …v n} represents a data set composed of n p-dimensional vectors, and v is a p-dimensional vector representing original data;

[0011] S2: training N data in one batch, performing data enhancement on each data v to obtain enhanced data vs; retaining the data v and the enhanced data vs, so that the data quantity changes from N to 2N, and obtaining a data set V'; the data v=[v1, v2, …, v p ], the enhanced data vs=[vs1, vs2, …, vs p ], v and v s are positive samples, and v and the remaining 2N-2 data are negative samples; the data set V'={v1, …, v n , vs1, …, vs n};

[0012] S3: passing each pair of positive sample data v and enhanced data vs in the data set V' into the restricted Boltzmann machine for forward propagation to obtain the corresponding data h and data hs in the hidden layer, and obtaining the feature H of the data set V' in the hidden layer; the data h=[h1, h2, …, h q ], the enhanced data hs=[hs1, hs2, …, hs q ], and the feature H={h1, …, hn, hs1, …, hs n};

[0013] S4: performing contrast learning on the feature H to reduce the distance between positive samples and increase the distance between positive and negative samples;

[0014] S5: back-propagating the data h to obtain reconstructed data v' of the data v in the visible layer; the reconstructed data v' = [v'1, v'2, …, v' p ];

[0015] S6: taking the maximum likelihood function as the reconstruction loss, minimizing the distance between the data v' and the data v;

[0016] S7: repeating steps S2-S6 until the reconstruction loss function of the restricted Boltzmann machine converges, realizing the training of the restricted Boltzmann machine;

[0017] S8: using the trained restricted Boltzmann machine in S7 to perform deep feature learning on the data set V to obtain the deep feature representation of the data set V.

[0018] As a preferred, in step S2, data augmentation is performed by adding Gaussian noise.

[0019] As a preferred, in step S3, when forward propagation is performed, the operation method is:

[0020] h = σ(b + Wv) (1)

[0021] In the formula: σ is a function, x is the dependent variable of the function; b = [b1, b2, …, b q ] represents the hidden layer bias vector; W is a matrix, matrix W = [w 11 … w ij … w pq ], w ij represents the connection weight between the i-th unit of the visible layer and the j-th unit of the hidden layer.

[0022] As a preferred, in step S5, when back-propagation is performed, the operation method is:

[0023] v' = σ(a + hW) (2)

[0024] In the formula: a = [a1, a2, …, a p ] represents the bias vector of the visible layer.

[0025] As a preferred, in step S4, when contrastive learning is performed on the feature H, the loss function used is the NT-Xent loss function.

[0026] As a preferred, the contrastive loss between any pair of samples in the same batch is calculated by the following formula:

[0027]

[0028] In the formula: NT-Xent: l(i,j) is the loss between sample i and sample j under the loss function NT-Xent; sim(h i , h j ) represents the cosine similarity of h i , h j ; h i , h j are hidden layer features; tau is a temperature parameter; k is a variable traversing the feature H, k [1, 2N]; 1 k≠i is an indication factor; h k is all samples in the feature H except h i .

[0029] The average contrastive loss between all samples in the same batch is calculated by the following formula:

[0030]

[0031] In the formula: Loss is the average contrastive loss between all samples in the same batch; l is the loss of a pair of samples in the same batch.

[0032] The beneficial effects of the present application are:

[0033] The present application fuses the contrastive learning of self-supervised learning with the restricted Boltzmann machine, can overcome the shortcomings of the purposeless learning of the traditional restricted Boltzmann machine, increase the explainability, and improve the stability and representation ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0035] Figure 1 It is a structural schematic diagram of the traditional restricted Boltzmann machine;

[0036] Figure 2 It is a "moving route" schematic diagram of the reconstruction feature distribution probability in the learning process of the traditional restricted Boltzmann machine;

[0037] Figure 3 It is a flow schematic diagram of the self-supervised deep representation method based on contrastive learning of the present application. DETAILED DESCRIPTION

[0038] The application will be further described below in conjunction with the drawings and examples. It should be noted that the examples in the present application and the technical features in the examples can be combined with each other without conflict. It should be noted that all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art unless otherwise specified. The present application discloses that the "including" or "containing" and similar words mean that the elements or objects before the words cover the elements or objects listed after the words and their equivalents, and do not exclude other elements or objects.

[0039] As shown in Figure 3 , the present application provides a self-supervised deep representation method based on contrastive learning, comprising the following steps:

[0040] S1: obtaining a data set V to be trained; the data set V = {v1, v2, … vn} represents a data set composed of n p-dimensional vectors, and v is a p-dimensional vector representing original data. n

[0041] S2: training N data in one batch, performing data enhancement on each data v to obtain enhanced data vs; retaining data v and enhanced data vs, so that the data amount changes from N to 2N, and obtaining a data set V'; the data v = [v1, v2, …, vn], the enhanced data vs = [vs1, vs2, …, vsn], v and v are positive samples, and v and the remaining 2N-2 data are negative samples; the data set V' = {v1, …, vn, vs1, …, vsn}. p p s n n

[0042] In one specific embodiment, data enhancement is performed by adding Gaussian noise. It should be noted that in machine learning, the method of data enhancement is prior art, and in addition to the method selected in the present embodiment, other data enhancement methods such as paraphrasing and sampling can also be applied to the present application.

[0043] S3: passing each pair of data v and enhanced data vs as positive samples in the data set V' into a restricted Boltzmann machine for forward propagation to obtain corresponding data h and data hs in the hidden layer, and obtaining the feature H of the data set V' in the hidden layer; the data h = [h1, h2, …, hn], the enhanced data hs = [hs1, hs2, …, hsn]; the feature H = {h1, …, hn, hs1, …, hsn}. q q n

[0044] ​​​​​​​​​In one specific embodiment, when forward propagation is performed, the operation method is:

[0045] h = σ(b + Wv) (1)

[0046] wherein σ is a function, x is a function dependent variable; b = [b1, b2, …, b q ] represents a hidden layer bias vector; W is a matrix, the matrix W = [w 11 … w ij … w pq ], w ij represents the connection weight between the i-th unit of the visible layer and the j-th unit of the hidden layer.

[0047] S4: Perform contrastive learning on the feature H to reduce the distance between positive samples and increase the distance between positive and negative samples.

[0048] In one specific embodiment, the NT-Xent loss function is used to calculate the loss between any pair of samples in the same batch, and the contrastive loss between any pair of samples in the feature H in the same batch is calculated by the following formula:

[0049]

[0050] wherein NT-Xent: l(i, j) is the loss between sample i and sample j under the loss function NT-Xent; sim(h i , h j ) represents the cosine similarity between h i and h j ; h i and h j are both hidden layer features; τ is a temperature parameter (a kind of hyperparameter, used to balance the influence between the numerator and the denominator); k is a variable traversing the feature H, k ∈ [1, 2N]; 1 k≠i is an indicator factor (used to exclude the similarity of the sample and itself); h k is all samples in the feature H except h i ; the denominator as a whole is to calculate the similarity of the remaining 2N-1 samples except itself, which is similar to the normalization operation.

[0051] The average contrastive loss between all samples in the same batch is calculated by the following formula:

[0052]

[0053] wherein Loss is the average contrastive loss between all samples in the same batch; l is the loss of a pair of samples in the same batch (calculated by formula (3)).

[0054] In the embodiment, through the NT-Xent contrast loss, the same kind is attracted and the different kind is repelled, so that the RBM learned features are more closely in the same kind and obviously different in different kinds, thereby improving the representation ability of the RBM. It should be noted that the NT-Xent loss function is only a preferred loss function of the present application, and other loss functions in the prior art can also be applicable to the present application.

[0055] S5: back-propagating the data h to obtain reconstructed data v' of the data v in the visible layer; the reconstructed data v' = [v'1, v'2, …, v' p ] of the data v.

[0056] In a specific embodiment, when back-propagating, the operation method is:

[0057] v'=σ(a+hW) (2)

[0058] In the formula, a = [a1, a2, …, a p ] represents a bias vector of the visible layer.

[0059] S6: taking the maximum likelihood function as a reconstruction loss, minimizing the distance between the reconstructed data v' and the data v.

[0060] S7: repeating steps S2-S6 until the reconstruction loss function of the restricted Boltzmann machine converges, and realizing the training of the restricted Boltzmann machine.

[0061] S8: using the trained restricted Boltzmann machine in S7 to perform deep representation learning on the data set V to obtain a deep feature representation of the data set V. That is, the restricted Boltzmann machine trained by step S7 is used to forward-propagate N pieces of data in the data set V to obtain the final feature set H' = [h1, h2, …, h q ] of the data set V. Each h represents a feature vector and can be used for specific downstream tasks, such as classification tasks, clustering tasks, regression, etc.

[0062] In one specific embodiment, the improved restricted Boltzmann machine (CL-RBM) of the present application and the traditional restricted Boltzmann machine (RBM) are verified using the dataset of Microsoft Asia Research Institute Multimedia (MSRA-MM). The MSRA-MM dataset is trained according to the above steps S1-S7, and the obtained feature representation is clustered using the K-means and spectral clustering clustering algorithms. In addition, in order to further verify the effect of the method, two groups of ablation experiments are additionally performed, i.e., the original data is directly clustered using the K-means and spectral clustering clustering algorithms. By comparing the accuracy of the above comparative experiments, the above experiments are repeated 10 times, and the mean and variance of the clustering accuracy are recorded to verify the feasibility of the method. The experimental results are shown in Table 1:

[0063] Table 1 Comparison of CL-RBM and RBM performance

[0064]

[0065] As can be seen from Table 1, the CL-RBM is trained on 6 different datasets, and the obtained features achieve good results in downstream clustering tasks. The CL-RBM has high clustering accuracy in different clustering algorithms (kmeans, spectral clustering), which indicates that the features learned by the CL-RBM have good generalization ability. The clustering accuracy of the features obtained by the CL-RBM is much higher than that of the features obtained by the traditional RBM and much higher than that of the results obtained by directly clustering the original data, which indicates that the improvement method of the CL-RBM is effective. In summary, the present application can perform better representation learning than the traditional RBM.

[0066] In summary, the present application combines the contrastive learning of self-supervised learning with the restricted Boltzmann machine, which can overcome the shortcomings of the traditional restricted Boltzmann machine without purposeful learning, increase the interpretability, and improve the representation ability of the model. Compared with the prior art, the present application has significant progress.

[0067] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the present application, and any simple modification, equivalent change and modification of the above embodiments based on the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A self-supervised deep representation method based on contrastive learning, characterized in that, The method for learning picture features comprises the following steps: S1: obtaining a data set V to be trained; the data set V = {v1, v2, … vn} indicates a data set composed of n p-dimensional vectors, and v is a p-dimensional vector representing original data; n} indicates a data set composed of n p-dimensional vectors, and v is a p-dimensional vector representing original data; S2: Batch train N data at a time, perform data enhancement on each data v, and obtain the enhanced data vs; retain the data v and the enhanced data vs, so that the amount of data changes from N to 2N, and obtain the data set V'; the data v = [v1, v2, ..., v p ], the enhanced data vs=[vs1,vs2,…,vs p ], v and v s are positive samples of each other, and v and the remaining 2N-2 data are negative samples of each other; the dataset V'={v1,…,v n ,vs1,…,vs n }; S3: forward propagating each pair of data v and enhanced data vs which are positive samples to each other in the data set V' into the restricted Boltzmann machine to obtain corresponding data h and data hs in the hidden layer, and obtain the features H of the data set V' in the hidden layer; the data h = [h1, h2, …, h q ], the enhanced data hs = [hs1, hs2, …, hs q ]; the features H = {h1, …, h n , hs1, …, hs n}. S4: contrast learning is performed on the features H to reduce the distance between positive samples and increase the distance between positive and negative samples; S5: back-propagating the data h to obtain reconstructed data v' of the data v in the visual layer; the reconstructed data v' = [v'1, v'2, …, v' p ] S6: a maximum likelihood function is used as a reconstruction loss to minimize the distance between the data v' and the data v; S7: steps S2-S6 are repeated until the reconstruction loss function of the restricted Boltzmann machine converges, and the training of the restricted Boltzmann machine is completed; S8: the data set V is subjected to deep feature learning by using the trained restricted Boltzmann machine in S7 to obtain the deep feature representation of the data set V.

2. The contrastive learning based self-supervised deep representation method according to claim 1, wherein, In step S2, data augmentation is performed by adding Gaussian noise.

3. The contrastive learning based self-supervised deep representation method according to claim 1, wherein, In step S3, when forward propagation is performed, the operation method is: （1） where σ is a function, x is a function dependent variable; b = [b1, b2,..., b q ] represents the hidden layer bias vector; W is a matrix, matrix W = [w 11 ... w ij ... w pq ], w ij represents the connection weight between the i-th unit of the visible layer and the j-th unit of the hidden layer.

4. The contrastive learning based self-supervised deep representation method according to claim 3, wherein, In step S5, when back propagation is performed, the operation method is: （2） Where: a=[a1,a2,…,a p ] represents the visual layer bias vector.

5. The contrastive learning based self-supervised deep representation method according to any one of claims 1-4, characterized in that, In step S4, when contrast learning is performed on the features H, the loss function used is an NT-Xent loss function.

6. The contrastive learning based self-supervised deep representation method according to claim 5, wherein, The contrast loss between any pair of samples in the same batch is calculated by the following formula: （3） wherein: NT-Xent(i,j) is the loss between sample i and sample j under the loss function NT-Xent; sim(h i ,h j ) represents the cosine similarity between h i and h j ; h i and h j are hidden layer features; τ is a temperature parameter; k is a variable traversing the feature H, k∈[1,2N]; 1 k≠i is an indication factor; h k is all samples in the feature H except h i . The average contrast loss between all samples in the same batch is calculated by the following formula: （4） In the formula: Loss is the average contrast loss between all samples in the same batch; l is the loss of a pair of samples in the same batch.

Images