Artificial neural network self-supervised contrastive learning method

Abstract

The application provides an artificial neural network self-supervised contrast learning method, which comprises forward propagation; in the forward propagation, a batch of input signals is expanded to obtain a plurality of expanded batches, and then the plurality of expanded batches are calculated through an artificial neural network to obtain embeddings of each input signal corresponding to each expanded batch; the embeddings of each input signal corresponding to the plurality of expanded batches are input into a loss function to obtain a network loss; the number of the plurality of expanded batches is greater than 2; and the loss function is L. The application uses a multi-variant neural network architecture and a new loss function, and better classification correctness than conventional self-supervised learning is achieved on a classification task, and general features in samples can be accurately extracted. Using the algorithm for model pre-training can make the model achieve better effect than using a conventional unsupervised algorithm.

Description

Technical Field

[0001] This application relates to artificial neural networks, and more particularly to a self-supervised contrastive learning method for artificial neural networks. Background Technology

[0002] In classification tasks, unsupervised learning still lags significantly behind supervised learning in terms of accuracy. Taking the classification results of the ResNet18 convolutional neural network on the CIFAR10 image database as an example: supervised learning (cross-entropy backpropagation) can achieve a classification accuracy of over 95%. SimCLR and VICREG are well-known self-supervised contrastive learning algorithms. The authors of the SimCLR paper reported an unsupervised learning accuracy of 90.6%, while the VICREG paper did not publish official data, but practitioners have verified that its accuracy is approximately 91%. BYOL is a well-known self-supervised learning algorithm using pseudo-Siamese networks, with a reported accuracy of 91.3%. SeCu and SCAN are well-known self-supervised deep clustering algorithms, with reported accuracy rates of 93.0% and 87.6%, respectively. It is evident that there is still a gap in classification accuracy between unsupervised and supervised learning.

[0003] Self-supervised contrastive learning is an important branch of unsupervised learning. Its architecture typically employs a Siamese network structure or a pseudo-Siamese network. In the former, two samples are fed into two neural networks with identical structures and shared weights, or it can be described as two samples being fed into the same neural network. In the latter, the two neural networks are largely the same, but with slight differences. Figure 1 This paper describes several self-supervised learning architectures, most of which employ a Siamese structure, while a few use a pseudo-Siamese structure. In contrastive learning, a pair of samples with the same label is called a positive pair, and those with different labels are called negative pairs. The labels here are not necessarily the true labels of the data's category. Some algorithms treat each sample as a separate class, meaning each sample has its own unique label. The first step in contrastive learning is to take a batch of N samples and perform two parallel data augmentations to obtain two variant batches, each containing N variants. These two variant batches are then fed into a neural network to obtain their output vectors. Because the last few layers of the neural network are often multilayer perceptrons (MLPs) that resemble high-dimensional space mappings, the output vectors are commonly referred to as embeddings. The embeddings are then used to calculate the loss using a loss function, and finally, backpropagation is used to backpropagate the loss and update the weights. The loss function of contrastive learning usually contains two terms: A measures the similarity between positive pairs and B measures the similarity between negative pairs. By adjusting the weights, positive pair embeddings tend to be similar and negative pair embeddings tend to be distant, thus achieving a clustering effect.

[0004] Unsupervised contrastive learning can effectively extract general features from samples, making it extremely suitable for model pre-training and adapting to various scenarios through transfer learning or model fine-tuning. However, the performance disadvantages of unsupervised learning limit its development potential, making performance improvement a hot research area in this field. Summary of the Invention

[0005] In view of the above problems, this application aims to propose a self-supervised contrastive learning method for artificial neural networks, which can effectively improve the performance of self-supervised learning and make it closer to supervised learning.

[0006] The self-supervised contrastive learning method for artificial neural networks disclosed in this application includes forward propagation; in the forward propagation, a batch of input signals is expanded to obtain multiple expanded batches, and then the multiple expanded batches are processed by an artificial neural network to obtain the embeddings of each input signal corresponding to each expanded batch; the embeddings of each input signal corresponding to the multiple expanded batches are input into a loss function to obtain the network loss; characterized in that: The number of the multiple extended batches is greater than 2; The loss function is: ; in, ; ; ; in, Representing the The embedding vector of the nth signal in a deformed batch after L2 norm normalization. Representing the The embedding vector of the m-th signal in a deformed batch after L2 norm normalization. Representing the The embedding vector of the nth signal in each deformed batch after L2 norm normalization; α and β are weighting constants.

[0007] Preferably, the expansion is performed in a reference manner, and the expansion batch includes, in addition to the deformed batch obtained by data augmentation, an untransformed copy of the input signal as the 0th deformed batch.

[0008] Preferably, the expansion is performed without reference, and the expansion batch contains only multiple deformed batches obtained from data augmentation.

[0009] Preferably, the loss function can also be replaced with , , , , , Any one of them; in, This represents the embedding vector of the nth signal in the 0th deformed batch after L2 norm normalization; This represents the embedding vector of the nth signal in the 0th deformed batch after L2 norm normalization; .

[0010] Preferably, the loss function can also be replaced with .

[0011] This invention utilizes a multivariate neural network architecture and a novel loss function to achieve better classification accuracy than conventional self-supervised learning in classification tasks, accurately extracting common features from samples. Pre-training the model using this algorithm can achieve better results than conventional unsupervised algorithms. This invention has high versatility; it can be ported to any neural network that uses supervised learning backpropagation algorithms for network weight updates or other contrastive learning algorithms. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the architecture for self-supervised learning. Figure 2 A flowchart of the forward propagation process for contrastive learning in multivariable, no-reference twin networks; Figure 3 Forward propagation flowchart for contrastive learning in multivariable reference Siamese networks; Figure 4 This is a schematic diagram of the PCLoss loss function; Figure 5 This is a schematic diagram of the embedded classification testing process; Figure 6 This is a schematic diagram illustrating the classification test process. Detailed Implementation

[0013] This invention is a self-supervised contrastive learning algorithm, which consists of two parts: (1) a multivariate contrastive learning neural network architecture, which is divided into two types: reference and no-reference; (2) a multivariate contrastive learning-specific loss function: PCLoss, which includes a basic version and three variants. The background mentioned above states that the principle of contrastive learning is to adjust the weights to make the output vectors of positive pairs more similar and the output vectors of negative pairs more distant. The core idea of ​​this invention is how to enhance the effect of the former. Self-supervised contrastive learning uses unlabeled data. Except for multimodal data (i.e., array data pointing to the same object), the acquisition of positive pairs depends on data augmentation. Conventional contrastive learning algorithms obtain two variants from each sample as a set of positive sample pairs through two data augmentations. The core idea of ​​this invention is that the number of data augmentations is not two, but can be any integer greater than two. For example: under conventional contrastive learning architectures such as SimCLR, a batch of N samples is augmented to obtain two variant batches, which are combined to form an extended batch containing 2N variants, including N sets of positive pairs and N (N-1) negative pairs, with the number of positive pairs being much smaller than the number of negative pairs. In the multi-variant no-reference contrastive learning architecture proposed in this invention, if the number of enhancements is P, then P b-variant batches are generated, combined to form a set containing N... The extended batch of P variants has N forward pairs. P (P-1) / 2 groups, negative pairs can theoretically reach N (N-1) Group P, but it can be simplified in practical applications to ensure calculation speed.

[0014] Increasing the number of positive sample pairs without expanding the batch size has two advantages, which will be explained below.

[0015] I. Increasing the number of positive sample pairs can extract features more effectively. The purpose of contrastive learning is to allow the neural network to extract as many invariants (common features of the category) as possible. Through random transformations (data augmentation), sample-specific features are randomized (such as the specific color of a pixel in a corner of an image, or the specific brightness relationship of several neighboring pixels within a small area). By making the embeddings of the positive pairs similar, the influence of these sample-specific features on the embedding vector is removed, leaving only the common features of the category that are beneficial to data classification. This process is random. Taking image classification as an example, in traditional two-variant contrastive learning, data augmentation may include random cropping, rotation, Gaussian blur, color changing, etc. Because the transformation is random, two variants of an image may not have undergone color changing. In this case, in addition to the common features of the category, the common embeddings also include a specific color distribution. The neural network will treat color as part of the common features and increase the influence of color on the classification result when updating the weights. In the case of multiple variants, due to the random nature, the probability of this happening is greatly reduced, and the common features between variants are more likely to be the common features of the category that are beneficial to classification.

[0016] Second, increasing the number of variants for the same sample can effectively prevent feature vanishing. In some datasets, the labeled object may appear very close to the edge of the sample; in this case, random rotation and cropping may completely cut it out. Or, the labeled object may be very similar in color to its surroundings; random exposure and color changes may make it indistinguishable. These situations all fall under the category of feature vanishing, and multiple variants can effectively reduce the impact of feature vanishing on training.

[0017] This invention includes two neural network architectures: reference-based and non-reference-based, with the non-reference-based architecture described above. The difference in the reference-based architecture lies in that, in addition to the P deformed batches obtained from P data augmentations, the expanded batch also includes a set of untransformed copies of the original data as the 0th deformed batch, which participates in the loss function calculation. The original data acts as a reference in the loss function calculation, further preventing feature vanishing. Both architectures have their advantages and disadvantages. Testing shows that the non-reference-based architecture performs better when the data augmentation is relatively conservative, while the reference-based architecture performs better when the data augmentation is relatively aggressive.

[0018] Existing contrastive learning loss functions, such as NT-Xent, are only applicable to two variants. To adapt to the complex augmentation architecture described above and maximize its advantages, we have invented a novel loss function: PCLoss (Primary Component Loss). The loss function is calculated as follows: ; in, ; ; ; in, The embedding vector (neural network output vector) of the nth image in the deformed batch p after L2 norm normalization. The embedding vector (neural network output vector) of the m-th image in the deformed batch p after L2 norm normalization. Let represent the embedding vector (neural network output vector) of the nth image in the variant batch q after L2 norm normalization; α and β are weighting constants, 〈〉 is the expectation operation, and || is the absolute value operation. L is the loss, where the first term of L measures the similarity of embedding vectors of different variants of the same sample, and the second term measures the dissimilarity of embedding vectors of different samples. By minimizing L, a clustering effect can be achieved where the embeddings of different variants of the same sample are similar, while the embeddings of different samples are distant. The computational complexity is First component The computational complexity is , From the first component , The computational complexity is , Calculate from scratch ( , The complexity is then O(n). If conventional loss functions such as NT-Xent are used to calculate the equivalent... This term typically has a complexity of O(n). It is evident that PCLoss significantly reduces the computational difficulty when P>>1 by utilizing the intermediate step of calculating the first component, thus ensuring the scalability of this architecture.

[0019] There are 7 transformations of the PCLoss loss function to choose from: ; ; ; ; ; ; ; The loss functions mentioned above each have their own advantages and disadvantages in terms of performance and stability, and can be selected according to specific circumstances.

[0020] To measure the effectiveness of this invention, a convolutional neural network (CNN) image classification task was used.

[0021] (I): Multivariate Non-reference Comparative Studies Architecture such as Figure 4As shown, the dataset used is CIFAR10. The encoder uses a slightly modified ResNet18 neural network, and the decoder is a two-layer MLP with dimensions of 2048 and 128, respectively. The CIFAR10 dataset contains 50,000 training images and 10,000 test images, divided into ten classes, with each image having a resolution of 32 pixels. 32, RGB three-channel.

[0022] (1): The training set is divided into batches with a batch size of 512, and sent to the data augmentation module to obtain transformed batch I. 1 to I P .

[0023] (2): The transform batch is sequentially fed into the encoder (ResNet18) and decoder through a multiplexer, and then sequentially obtained into embedded A through a multiplexer. 1 To A P .

[0024] (3): Calculate the loss using the PCLoss function, and calculate the gradient and update the weights using the backpropagation algorithm.

[0025] (4): Repeat (1) to (3) until the training set is exhausted, as one epoch.

[0026] (5): Repeat (4) until the number of periods reaches the preset value (1000 times in this test), and the training ends.

[0027] (II): Test accuracy To evaluate the training effect, the classification accuracy needs to be tested. The architecture is as follows: Figure 6 As shown, the linear classifier is a fully connected layer with an input dimension equal to the embedding vector dimension and an output dimension equal to the number of sample classes (since CIFAR10 is used, the output dimension is 10). When testing accuracy, a decoder is not used; only the encoder and the linear classifier are employed.

[0028] (1): The training set is divided into batches with a batch size of 512 (the batch size will not significantly affect the training results). The batches are then fed into the encoder and the linear classifier to obtain the predicted values.

[0029] (2): Compare the predicted values ​​with the actual labels, calculate the loss using cross-entropy, update the weights of the linear classifier using backpropagation, and do not update the weights of the encoder.

[0030] (3): Repeat (1) and (2) until the training set is exhausted.

[0031] (4): The test set is divided into batches with a batch size of 512 (the batch size has no effect). The batches are then fed into the encoder and the linear classifier to obtain the predicted values.

[0032] (5): Compare the predicted values ​​with the actual labels and calculate the accuracy.

[0033] (6): Repeat (4) and (5) until the training set is exhausted, at which point a period ends.

[0034] (7): Repeat (3) and (6) until the number of periods reaches the preset value (100 times in this test), and report the best accuracy.

[0035] Following the above procedure, the classification accuracy was 93.2% when the number of variants P=6. This is higher than all the self-supervised learning algorithms mentioned in the background, and far exceeds the performance of conventional contrastive learning algorithms such as SimCLR and VICREG. The ResNet18 neural network used in this study did not fully realize the potential of this invention. Using a neural network with higher parameter counts, such as ResNet50, would likely further improve the accuracy.

[0036] During testing, in addition to using the encoder's output vector (i.e., features, see...) Figure 2 , Figure 3 ) to classify ( Figure 6 Furthermore, the embeddings output by the decoder can be used for classification. Figure 5 There is no significant difference in accuracy between the two. When porting the model after training, you can either port only the encoder or the complete neural network (encoder plus decoder).

[0037] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this application, the entire contents of that material are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this application, as well as documents that limit the broadest scope of the claims in this application (currently or subsequently appended to this application). It should be noted that if there is any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials of this application and the content of this application, the descriptions, definitions, and / or terminology used in this application shall prevail.

[0038] Finally, it should be understood that the embodiments described in this application are merely illustrative of the principles of the embodiments of this application. Other modifications may also fall within the scope of this application. Therefore, alternative configurations of the embodiments of this application are considered as examples and not limitations, and are regarded as consistent with the teachings of this application. Accordingly, the embodiments of this application are not limited to the embodiments explicitly described and illustrated in this application.

Claims

1. A self-supervised contrastive learning method using artificial neural networks for image classification, comprising forward propagation; in the forward propagation, expanding a batch of images as input signals to obtain multiple expanded batches, and then using an artificial neural network to calculate the embeddings of each input signal corresponding to each expanded batch; inputting the embeddings of each input signal corresponding to the multiple expanded batches into a loss function to obtain the network loss; characterized in that: The number of the multiple extended batches is greater than 2; The loss function is: ; in, ; ; ; in, Representing the The embedding vector of the nth signal in a deformed batch after L2 norm normalization. Representing the The embedding vector of the m-th signal in a deformed batch after L2 norm normalization. Representing the The embedding vector of the nth signal in a deformed batch after L2 norm normalization; and These are weighting constants; This is for the expected value calculation; P is the number of modified batches; When the expansion is performed in a reference manner, the expansion batch includes not only the deformed batch obtained from data augmentation, but also an untransformed copy of the input signal as the 0th deformed batch; when the expansion is performed without a reference, the expansion batch only includes multiple deformed batches obtained from data augmentation.

2. The self-supervised contrastive learning method for artificial neural networks according to claim 1, characterized in that: When the extension is performed using a reference formula, the loss function can also be replaced with... 、 、 、 、 、 Any one of them; in, This represents the embedding vector of the nth signal in the 0th deformed batch after L2 norm normalization; This represents the embedding vector of the nth signal in the 0th deformed batch after L2 norm normalization; ; It is a linear rectification function.

3. The self-supervised contrastive learning method for artificial neural networks according to claim 1, characterized in that: When the expansion is performed without a reference, the loss function can also be replaced with ; It is a linear rectification function.

Images