Artificial neural network self-supervised contrast learning method

Abstract

The invention provides an artificial neural network self-supervised contrast learning method. The method comprises the following steps: forward propagation; in forward propagation, a batch of an input signal is expanded to obtain a plurality of expanded batches, and then the plurality of expanded batches are calculated through an artificial neural network to obtain embedding of each input signal corresponding to each expanded batch; embedding the input signals corresponding to the plurality of expansion batches into an input loss function to obtain network loss; the number of the plurality of expansion batches is greater than 2; the loss function is shown in the specification. According to the method, a multi-variant neural network architecture and a novel loss function are utilized, the classification correctness better than that of conventional self-supervised learning is achieved in the aspect of classification tasks, and universal features in samples can be accurately extracted. When the algorithm is used for pre-training the model, the model can achieve a better effect compared with a conventional unsupervised algorithm.

Description

Technical Field

[0001] The present application relates to an artificial neural network, and in particular to an artificial neural network self-supervised contrast learning method. Background Art

[0002] In classification tasks, unsupervised learning still has a big gap in accuracy compared to supervised learning. Take the classification results of the ResNet18 convolutional neural network on the CIFAR10 image database as an example: the classification accuracy after supervised learning (cross entropy back propagation) training can reach more than 95%. SimCLR and VICREG are both well-known self-supervised contrastive learning algorithms in the industry. The unsupervised learning accuracy published by the authors in the SimCLR paper is 90.6%. The VICREG paper did not publish official data, but it has been verified by practitioners that its accuracy is about 91%. BYOL is a well-known self-supervised learning algorithm using a pseudo-twin network, with an accuracy of 91.3% published in the paper. SeCu and SCAN are well-known self-supervised deep clustering algorithms, with the accuracy published in the paper being 93.0% and 87.6% respectively. It can be seen that there is still a gap in classification accuracy between unsupervised and supervised.

[0003] Self-supervised contrastive learning is an important branch of unsupervised learning, and its architecture usually adopts a Siamese network structure or a pseudo-Siamese network. In the former, two samples are fed into two neural networks with the same structure and shared weights, or it can be described as two samples being fed into the same neural network, while in the latter, the two neural networks are roughly the same but slightly different. Figure 1 Some self-supervised learning architectures are described. It can be seen that most of them use twin structures, and the remaining few use pseudo twin structures. In contrastive learning, a pair of samples with the same label is called a positive pair, and a pair with different labels is called a negative pair. The label here is not necessarily the true label of the category to which the data belongs. Some algorithms treat each sample as a class, that is, each sample has its own unique label. The first step of contrastive learning is to take a batch of N samples, and obtain two deformed batches containing N variants through two parallel data enhancements. Then, the two deformed batches are sent to the neural network to obtain their output vectors. Because the last few layers of the neural network are usually multi-layer perceptrons (MLPs) that map high-dimensional space, the industry usually calls the output vector embedding. The embedding is used to calculate the loss through the loss function, and finally the loss is propagated back to update the weights. The loss function of contrastive learning usually contains two items. Item A measures the similarity between positive pairs, and item B measures the similarity between negative pairs. By adjusting the weights to make the embeddings of positive pairs similar and the embeddings of negative pairs distant, a clustering effect can be achieved.

[0004] Unsupervised contrastive learning can effectively extract the common features of samples, so it is extremely suitable for model pre-training and can be adapted to various scenarios through transfer learning or model fine-tuning. The performance disadvantage of unsupervised learning has affected its development space, and improving the performance of unsupervised learning is a hot research direction in this field today. Summary of the invention

[0005] In view of the above problems, the present application aims to propose an artificial neural network self-supervised contrastive learning method, which can effectively improve the performance of self-supervised learning and make it closer to supervised learning.

[0006] The artificial neural network self-supervised contrastive learning method of the present application includes forward propagation; in the forward propagation, a batch of input signals is expanded to obtain multiple extended batches, and then the multiple extended batches are calculated by the artificial neural network to obtain embeddings of each input signal corresponding to each extended batch; the embeddings of each input signal corresponding to the multiple extended batches are input into the loss function to obtain the network loss; it is characterized by: The number of the multiple extended batches is greater than 2; The loss function is: ; in, ; ; ; in, represents the embedding vector of the nth signal in the pth deformation batch after L2 norm normalization, represents the embedding vector of the mth signal in the pth deformation batch after L2 norm normalization, represents the L2-norm normalized embedding vector of the nth signal in the qth deformation batch; α and β are weighting constants.

[0007] Preferably, the expansion is performed in a reference manner, and the expanded batch includes, in addition to the deformation batch obtained by data enhancement, an untransformed copy of the input signal as the 0th deformation batch.

[0008] Preferably, the expansion is performed in a reference-free manner, and the expansion batch only includes multiple deformation batches obtained by data enhancement.

[0009] Preferably, the loss function can also be replaced by , , , , , Any of the following; in, represents the embedding vector of the nth signal in the 0th deformation batch after L2 norm normalization; represents the embedding vector of the nth signal in the 0th deformation batch after L2 norm normalization; .

[0010] Preferably, the loss function can also be replaced by .

[0011] The present invention utilizes a multi-variant neural network architecture and a novel loss function to achieve better classification accuracy than conventional self-supervised learning in classification tasks, and can accurately extract common features in samples. Using this algorithm to pre-train the model can enable the model to achieve better results than using conventional unsupervised algorithms. The present invention is highly versatile, and any neural network that uses a supervised learning backpropagation algorithm to update network weights or uses other contrastive learning algorithms can be transplanted and used with the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Schematic diagram of the architecture for self-supervised learning; Figure 2 The forward propagation flowchart for multi-variant reference-free twin network comparative learning; Figure 3 Forward propagation flowchart for multi-variant reference Siamese network comparative learning; Figure 4 Schematic diagram of PCLoss loss function; Figure 5 This is a schematic diagram of the embedding classification test process; Figure 6 Schematic diagram of the expression classification test process. DETAILED DESCRIPTION

[0013] The present invention is a self-supervised contrastive learning algorithm, which includes two parts: (1) a multi-variant contrastive learning neural network architecture, which is subdivided into two types: reference type and no-reference type; (2) a multi-variant contrastive learning-specific loss function: PCLoss, which includes a basic version and three variants. The above background mentioned that the principle of contrastive learning is to adjust the weights to make the output vectors of positive pairs tend to be similar and the output vectors of negative pairs tend to be distant. The core idea of ​​the present invention is how to enhance the effect of the former. Because self-supervised contrastive learning uses unlabeled data, except for multimodal data (that is, array data points to the same object), the acquisition of positive pairs depends on data enhancement. Conventional contrastive learning algorithms obtain two variants from each sample as a set of positive sample pairs through two data enhancements. The core idea of ​​the present invention is that the number of data enhancements is not twice, but can be any integer greater than two. The following example is given: Under conventional contrastive learning architectures such as SimCLR, a batch of N samples is augmented with data to obtain 2 variant batches, which are combined into an extended batch containing 2N variants, including N groups of positive pairs and N*(N-1) groups of negative pairs, where the number of positive pairs is much smaller than the number of negative pairs. Under the multi-variant reference-free contrastive learning architecture proposed in the present invention, assuming the number of augmentations is P, P b variant batches are generated, which are combined into an extended batch containing N*P variants, with N*P*(P-1) / 2 groups of positive pairs and N*(N-1)*P groups of negative pairs in theory, but they can be simplified in actual applications to ensure the calculation speed.

[0014] Increasing the number of positive pairs without expanding the batch size has two advantages, which are explained below.

[0015] 1. Increasing the number of positive sample pairs can extract features more effectively. The purpose of contrastive learning is to allow the neural network to extract invariants (category common features) as much as possible. Through random transformation (data enhancement), sample-specific features are randomized (such as the specific color of a pixel in a corner of the image, the specific brightness relationship of several adjacent pixels in a small range). By making the positive pair embeddings similar, the influence of these sample-specific features on the embedding vector is removed, and what remains are the category common features that are beneficial to data classification. This process is random. Taking image classification as an example, under traditional two-variant contrastive learning, data enhancement may include random shearing, rotation, Gaussian blur, color change, etc. Because the transformation is random, it is very likely that the two variants of a picture have not been color-changed. At this time, in addition to the category common features, the common embeddings also include a specific color distribution. The neural network will regard color as part of the common features, and when updating the weights, it will increase the influence of color on the classification results. In the case of multiple variants, due to the random nature, the probability of this happening is greatly reduced, and the common features between variants are more likely to be category common features that are beneficial to classification.

[0016] Second, increasing the number of variants of the same sample can effectively prevent feature disappearance. In some data sets, the object referred to by the label may appear at the very edge of the sample. In this case, random rotation and cropping may completely cut it off. Or the object referred to by the label may be very similar in color to the surrounding area. Random exposure and color change may make it indistinguishable. These situations are all considered feature disappearance, and multiple variants can effectively reduce the impact of feature disappearance on training.

[0017] The present invention includes two neural network architectures: reference type and non-reference type. The above describes the non-reference type. The difference of the reference type is that in addition to the P deformation batches obtained by P times of data enhancement, the expansion batch also includes a group of non-transformed copies of the original data as the 0th deformation batch, which participates in the calculation of the loss function. The original data serves as a reference when calculating the loss function, which can further prevent the disappearance of features. The two architectures have their own advantages. After testing, the non-reference type has better training effect when the data enhancement amplitude is relatively conservative, while the reference type has better effect when the data enhancement is relatively aggressive.

[0018] Existing contrastive learning loss functions such as NT-Xent are only applicable to the case of two variants. In order to adapt to the above-mentioned complex enhancement architecture and maximize its advantages, we invented a new loss function: PCLoss (PrimaryComponent Loss). The loss function is calculated as follows:

[0019] ; in, ; ; ; in, represents the embedding vector (neural network output vector) of the nth image in the deformation batch p after L2 norm normalization, represents the embedding vector (neural network output vector) of the mth image in the deformation batch p after L2 norm normalization, represents the embedding vector (neural network output vector) of the nth image in the deformation batch q after L2 norm normalization; α and β are weighted constants, 〈〉 is the expected value operation, and || is the absolute value operation. L is the loss. The first term of L measures the similarity of the embedding vectors of different variants of the same sample, and the second term measures the dissimilarity of the embedding vectors of different samples. By minimizing L, we can achieve a clustering effect in which different variants of the same sample have similar embeddings and different samples have distant embeddings. The computational complexity is , the first component The computational complexity is , From the first ingredient , The computational complexity is , Ab initio calculation ( , ) complexity is If we use conventional loss functions such as NT-Xent to calculate, it is equivalent to This item usually has a complexity of It can be seen that PCLoss uses the intermediate step of calculating the first component to greatly reduce the difficulty of calculation when P>>1, ensuring the scalability of this architecture.

[0020] The PCLoss loss function has 7 optional transformations: ; ; ; ; ; ; ; The above loss functions have their own advantages in performance and stability, and can be selected according to specific circumstances.

[0021] In order to measure the effect of the present invention, a convolutional neural network (CNN) image classification task was adopted.

[0022] 1. Multivariate non-reference comparative studies Architecture Figure 4 As shown in the figure, the dataset uses CIFAR10, the encoder uses a slightly modified ResNet18 neural network, and the decoder is a two-layer MLP with dimensions of 2048 and 128 respectively. The CIFAR10 dataset contains 50,000 training images and 10,000 test images, divided into ten categories, with image pixels of 32*32 and RGB three channels.

[0023] (1): The training set is divided into batches, with a batch size of 512, and sent to the data enhancement module to obtain the transformed batch I 1 to I P .

[0024] (2): The transformed batch is fed into the encoder (ResNet18) and decoder in turn through the multiplexer, and the embedding A is obtained in turn through the multiplexer 1 To A P .

[0025] (3): Use the PCLoss function to calculate the loss, and use the back-propagation algorithm to calculate the gradient and update the weights.

[0026] (4): Repeat (1) to (3) until the training set is exhausted, which is an epoch.

[0027] (5): Repeat (4) until the number of epochs reaches the preset value (1000 in this test), and the training ends.

[0028] (II): Test accuracy To verify the training effect, the classification accuracy needs to be tested. The architecture is as follows Figure 6 As shown in the figure, the linear classifier is a fully connected layer with an input dimension equal to the embedding vector dimension and an output dimension equal to the number of sample categories (since CIFAR10 is used, the output dimension is 10). When testing the accuracy, the decoder is not used, only the encoder and linear classifier are used.

[0029] (1): The training set is divided into batches with a batch size of 512 (the batch size will not seriously affect the training results), which are then fed into the encoder and linear classifier to obtain the predicted value.

[0030] (2): The predicted value is compared with the actual label, the loss is calculated using cross entropy, and the weights of the linear classifier are updated using the backpropagation method. The weights of the encoder are not updated.

[0031] (3): Repeat (1) and (2) until the training set is exhausted.

[0032] (4): The test set is divided into batches with a batch size of 512 (the batch size has no effect), and is sent to the encoder and linear classifier in turn to obtain the predicted value.

[0033] (5): Compare the predicted value with the actual label and calculate the accuracy.

[0034] (6): Repeat (4) and (5) until the training set is exhausted, and this period ends.

[0035] (7): Repeat (3) and (6) until the number of epochs reaches the preset value (100 in this test), and report the best accuracy.

[0036] According to the above process, when the number of variants P=6, the classification accuracy is 93.2%, which is higher than all the self-supervised learning algorithms mentioned in the background, and much higher than the results of conventional comparative learning such as SimCLR and VICREG. The ResNet18 neural network used this time did not bring out the full potential of the present invention. If a neural network with a higher number of parameters such as ResNet50 is used, the accuracy should be further improved.

[0037] At test time, in addition to using the encoder output vector (i.e., features, see Figure 2 , Figure 3 ) to classify ( Figure 6 ), and can also use the embedding output by the decoder for classification ( Figure 5 ), there is no significant difference in accuracy between the two. After training, when migrating the model, you can either migrate only the encoder or the complete neural network (encoder plus decoder).

[0038] Each patent, patent application, patent application disclosure, and other materials, such as articles, books, instructions, publications, documents, etc., cited in this application is hereby incorporated by reference in its entirety. Except for application history documents that are inconsistent with or conflicting with the content of this application, documents that limit the broadest scope of the claims of this application (currently or later attached to this application) are also excluded. It should be noted that if the description, definition, and / or use of terms in the attached materials of this application are inconsistent or conflicting with the content described in this application, the description, definition, and / or use of terms in this application shall prevail.

[0039] Finally, it should be understood that the embodiments described in this application are only used to illustrate the principles of the embodiments of the present application. Other variations may also fall within the scope of the present application. Therefore, as an example and not a limitation, the alternative configurations of the embodiments of the present application may be considered to be consistent with the teachings of the present application. Accordingly, the embodiments of the present application are not limited to the embodiments explicitly introduced and described in the present application.

Claims

1. An artificial neural network self-supervised contrastive learning method, comprising forward propagation; in the forward propagation, a batch of input signals is expanded to obtain multiple extended batches, and then the multiple extended batches are calculated by the artificial neural network to obtain embeddings of each input signal corresponding to each extended batch; the embeddings of each input signal corresponding to the multiple extended batches are input into a loss function to obtain a network loss; Features: The number of the multiple extended batches is greater than 2; The loss function is: ; in, ; ; ; in, represents the embedding vector of the nth signal in the pth deformation batch after L2 norm normalization, represents the embedding vector of the mth signal in the pth deformation batch after L2 norm normalization, represents the L2-norm normalized embedding vector of the nth signal in the qth deformation batch; α and β are weighting constants.

2. The artificial neural network self-supervised contrastive learning method according to claim 1, Features: The expansion is performed in a reference manner, and the expanded batch includes, in addition to the deformation batch obtained by data enhancement, an untransformed copy of the input signal as the 0th deformation batch.

3. The artificial neural network self-supervised contrastive learning method according to claim 1, Features: The expansion is performed in a reference-free manner, and the expansion batch only includes multiple deformation batches obtained by data enhancement.

4. The artificial neural network self-supervised contrastive learning method according to claim 2, Features: The loss function can also be replaced by 、 、 、 、 、 Any of the following: in, represents the embedding vector of the nth signal in the 0th deformation batch after L2 norm normalization; represents the embedding vector of the nth signal in the 0th deformation batch after L2 norm normalization; .

5. The artificial neural network self-supervised contrastive learning method according to claim 3, Features: The loss function can also be replaced by 。

Images