A task-oriented self-supervised learning method based on contrastive Siamese networks

Abstract

The present invention relates to a self-supervised learning method with target task orientation based on a contrastive twin network. The method performs data augmentation on a large amount of unsupervised data, uses a target task encoder to guide the convergence direction of the contrastive learning in the representation space, and uses the EM algorithm for training. After the network converges, a specific downstream task is then connected to the network for fine-tuning, thereby obtaining a network that can solve the target image task. The directed self-supervised learning adopted by the present invention solves the problem of network training when labeled data is insufficient for two-dimensional image data, reduces the computing power consumption of self-supervised contrastive learning, alleviates the difficulty of self-supervised contrastive learning in converging, and improves the accuracy of downstream tasks.

Description

Technical Field

[0001] The present invention relates to the field of self-supervised learning, and in particular to a self-supervised learning method with target task orientation based on a contrastive twin network. Background Art

[0002] With the development of deep learning, it has become an integral part of most intelligent systems. Due to its ability to learn rich patterns from large amounts of data, deep neural networks (DNNs) have become an effective method for most computer vision (CV) tasks, such as image classification, object detection, image segmentation, and action recognition, as well as natural language processing (NLP) tasks, such as sentence classification, language modeling, and machine translation. However, supervised methods for learning features from labeled data have reached a bottleneck due to the enormous cost of manually annotating millions of data samples. This is because most modern (supervised) computer vision systems often obtain some form of mapping representation by finding patterns between data points and their respective annotations in large datasets. Works such as GRAD-CAM have proposed techniques to provide visual explanations for the decisions made by the model, making them more interpretable.

[0003] Traditional supervised learning methods rely heavily on the amount of available labeled training data. Despite the abundance of available data, the lack of annotations has forced researchers to find alternative ways to leverage it. This has been a major driver in the development of self-supervised deep learning methods—those that don't require expensive annotation and learn supervised feature representations.

[0004] Obtaining accurate annotations is a prohibitively expensive undertaking. Even large 2D image datasets like ImageNet have only 1,000 categories. For 2D pixel-level tasks or 3D point clouds, the annotation cost rises exponentially. In machine learning and cognitive science, an artificial neural network (ANN), also known as a neural network (NN) or a neural network-like model, is a mathematical or computational model that mimics the structure and function of biological neural networks (the central nervous system of animals, particularly the brain) and is used to estimate or approximate functions. Neural networks perform computations by connecting large numbers of artificial neurons.

[0005] Self-supervised learning is popular because it eliminates the cost of large-scale data annotation. It can employ custom pseudo-labels as supervision and leverage the learned representations for multiple downstream tasks. Contrastive learning has recently become a staple of self-supervised learning in computer vision, natural language processing (NLP), and other fields. It aims to place data augmented embeddings of the same sample close to each other while pushing samples with different pseudo-labels further apart.

[0006] Generative models gained popularity after the introduction of Generative Self-Supervised Networks (GANs) in 2014. This work later became the basis for many successful architectures such as CycleGAN, StyleGAN, PixelRNN, Text2Image, DiscoGAN

[24] , etc. These methods inspired more researchers to turn to training deep learning models using unlabeled data in a self-supervised setting. As researchers achieved success, people began to realize some of the complex drawbacks of GAN-based generative models. (1) Non-convergence - model parameters fluctuate greatly and are difficult to converge, and (2) when the discriminator is too strong, the generator network cannot create realistic pseudo signals and therefore cannot continue to learn. In addition, proper synchronization between the generator and the discriminator is required to prevent the discriminator from converging and the generator from diverging.

[0007] Unlike generative models, contrastive learning (CL) is a discriminative method that aims to map samples from similar groups closer together and samples from different groups farther apart. Siamese networks have become a common architecture for self-supervised visual representation learning. Contrastive learning networks generally require a large amount of unlabeled data and negative sample pairs to perform well. The effectiveness of representation learning is generally measured by the uniformity and separability of the mapped sample distribution. When the sample distribution is extremely uneven and the loss is very small, it is called collapsing.

[0008] The core idea of ​​contrastive learning is that positive examples attract and negative examples repel. In contrastive learning, Siamese networks are particularly widely used.

[0009] In practical applications, contrastive learning methods benefit from a large number of negative samples. These samples can be stored in memory. In a Siamese network, MoCo maintains a queue of negative samples and converts one branch into a momentum encoder to improve the consistency of the queue. SimCLR directly uses the negative samples present in the current batch and requires a large batch size to work properly.

[0010] In addition, in the actual application of contrastive learning, all algorithms do not incorporate the directionality of downstream tasks in the self-supervision stage, so the convergence is slow during downstream fine-tuning and the task accuracy is limited. Summary of the Invention

[0011] In response to the problems existing in the prior art, the purpose of the present invention is to provide a self-supervised learning method with target task orientation based on a contrastive twin network to improve the accuracy and convergence speed of downstream tasks.

[0012] To achieve the above object, the technical solution adopted by the present invention is:

[0013] A self-supervised learning method with target task orientation based on a contrastive twin network is proposed. The method performs data augmentation on a large amount of unsupervised data, uses a target task encoder to guide the convergence direction of the representation space for the convergence of contrastive learning, and uses the EM algorithm for training. After the network converges, it then connects to a specific downstream task for network fine-tuning to obtain a network that can solve the target image task.

[0014] The method specifically comprises the following steps:

[0015] Step 1: Determine the target task encoder TF for model training and construct a self-supervised network with it, encoder D, predictor P1 and predictor P2;

[0016] The encoder D has two branches in common; the output of the first branch of the encoder D is connected to the predictor P1 and the predictor P2, and the output of the predictor P1 and the output of the second branch of the encoder D are used to calculate the similarity loss , calculate the cosine similarity of the two branches in the representation space; p1 and z2 are output by the encoder D and predictor P1 respectively;

[0017] The output of the predictor P2 and the output of the target task encoder TF are used to calculate the latent space similarity loss , represents the degree of similarity between the encoder’s hidden representation and the teacher network representation in the space;

[0018] The total loss can be obtained based on the similarity loss and the latent space similarity loss in, is the hyperparameter weight factor;

[0019] Step 2: Obtain the unlabeled dataset X from the target task dataset;

[0020] The dataset X consists of a sample set Composed and divided into multiple batches ;

[0021] Step 3: Take the unlabeled dataset X as the input of the self-supervised network. To optimize the target, iteratively optimize on the self-supervised network to obtain the encoder D from X to the target representation space;

[0022] The training process is as follows:

[0023] (1) Network parameter set Initialize ;

[0024] (2) All batches The data is input into the network in turn for iterative training; Input the network and perform data augmentation T1 and T2 to derive two enhanced batches of samples t1 and t2;

[0025] (3) The processed samples t1 and t2 are passed through the self-supervised network to obtain the total loss L. The network parameters are calculated based on the total loss L. Back propagation of the gradient updates the predictor P1 and encoder D;

[0026] Optimization goal Decomposed into the following two alternating sub-problems, where: As network parameters, As a hidden variable, the subscript t is the number of iterations:

[0027] (1)

[0028] (2)

[0029] The above two problems are consistent with the EM algorithm optimization framework. The E step is mainly to solve the expectation of the latent variable, which is mainly undertaken by the predictor P1 in the network, while the M step is to minimize the loss function, which is mainly undertaken by the back propagation process in the network. The specific details of the back propagation encoder parameter update are as follows:

[0030] Problem (1) is the M-step in the EM algorithm, which seeks to minimize the expected loss during network backpropagation by updating the encoder parameters:

[0031]

[0032] , is the weight parameter,

[0033] Question (2) is the E step in the EM algorithm based on the current To calculate the data for The distribution of is solved using predictor P1:

[0034]

[0035] In the actual iterative optimization process, the predictor P1 has no effect at the beginning, so , but as the iteration proceeds, P1 will gradually record the sample distribution data, making Close to expectations;

[0036] Make

[0037]

[0038] The iterative process will be repeated M times according to the set epoch number M, and each process will fully train all batches ;

[0039] After training, the data X is obtained into the encoder D of the representation space;

[0040] Step 4: Connect the fully connected layer of the encoder D obtained in step 3 to the linear layer of the linear network to obtain the neural network f; at the same time, obtain a small amount of labeled data set;

[0041] Step 5: Based on a small amount of labeled data sets, perform gradient descent according to CROSSENTROPYLOSS. After N epochs, a neural network f suitable for the target technical task is obtained.

[0042] To promote optimization goals Convergence, the learning rate lr decays according to the equation:

[0043]

[0044] Attenuation is performed, where i is the current cycle number, .

[0045] The weight parameter Adopt the weight-decreasing algorithm , i= .

[0046] After adopting the above scheme, the present invention performs data augmentation on a large amount of unsupervised data, uses a target task-related encoder to guide the convergence direction of the representation space for contrastive learning, uses the EM optimization algorithm for training, and connects to specific downstream tasks for network fine-tuning after network convergence. The directional self-supervised learning adopted by the present invention solves the problem of network training when there is insufficient labeled data for two-dimensional image data. It reduces the computing power consumption of self-supervised contrastive learning, alleviates the difficulty of self-supervised contrastive learning in converging, and improves the accuracy of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a diagram of the architecture of the self-supervisory network of the present invention;

[0048] Figure 2 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0049] The present invention discloses a self-supervised learning method with target task orientation based on a contrastive twin network. It performs data augmentation and amplification on a large amount of unsupervised data, uses a target task encoder to guide the convergence direction of the representation space for the convergence of contrastive learning, and uses the EM algorithm for training. After the network converges, it then connects to specific downstream tasks for network fine-tuning, thereby obtaining a network that can solve the target image task.

[0050] To better illustrate the solution content of the present invention, the following detailed description is given using the fine-grained classification task of cat breeds as an example.

[0051] Reference Figure 1 and Figure 2 As shown, the self-supervised learning method of the present invention includes the following steps:

[0052] Step 1: Determine the target task encoder TF for model training and construct a self-supervised network with it, encoder D, predictor P1 and predictor P2.

[0053] Among them, the encoder D has two branches shared by each other, and the encoder D adopts a 50-layer residual network and a fully connected layer.

[0054] The output of the first branch of the encoder D is connected to the predictor P1 and the predictor P2. The output of the predictor P1 and the output of the second branch of the encoder D are used to calculate the similarity loss. , calculate the cosine similarity of the two branches in the representation space. p1 and z2 are output by the encoder D and predictor P1 respectively.

[0055] The output of the predictor P2 and the output of the target task encoder TF are used to calculate the latent space similarity loss , represents the degree of similarity between the encoder’s hidden representation and the teacher network representation in the representation space. Since the latent space obeys different metrics, it is necessary to learn the mapping function P2 between the two metric spaces.

[0056] The total loss can be obtained based on the similarity loss and the latent space similarity loss in, is the hyperparameter weight factor.

[0057] The target task encoder TF uses an existing encoder. In this embodiment, the target task encoder TF uses an existing classification encoder. Specifically, the transformer network vit_h_14 in torchvision is used, and its pre-trained weights IMAGENET1K_SWAG_E2E_V1 on the ImageNet1000 dataset are used as the network weights. The last linear layer of the network is deleted, and this network is used as the target task encoder TF.

[0058] Step 2: Obtain the unlabeled dataset X from the target task dataset.

[0059] In this embodiment, the target task dataset is a variety of cat data crawled by a web crawler. The data has the same size of 255×255. The dataset X consists of a three-channel RGB cat image sample set Composition, divided into multiple batches according to the specific operating equipment performance and training method .

[0060] Step 3: Take the unlabeled dataset X as the input of the self-supervised network. To optimize the target, iterative optimization is performed on the self-supervised network to obtain the encoder D from X to the target representation space.

[0061] The training process is as follows:

[0062] (1) Network parameter set Initialize ;

[0063] (2) All batches The data is input into the network in turn for iterative training; Input the network and perform data enhancement T1 and T2 to derive two enhanced batches of samples t1 and t2.

[0064] In this embodiment, two sets of data enhancement The following data augmentation combinations are used: a. Randomly crop the image and randomly retain 20% to 100% of the image data; b. Supersample to 32x32 pixels; c. Randomly change the brightness, contrast, saturation and hue; d. Randomly adjust the grayscale; e. Randomly perform Gaussian blur; f. Randomly flip the image horizontally.

[0065] (3) The processed samples t1 and t2 are Figure 1 The network obtains the total loss L, and the network parameters are calculated based on the total loss L. The gradient of is back-propagated to update the predictor P1 and the encoder D.

[0066] The optimization principle is as follows:

[0067] Optimization goal It is the probability distribution constructed by encoder D to obtain the maximum likelihood An equivalent alternative to , where X is a sampled data set, is the mapping set of X in the latent space, is the parameter of encoder D, encoder D is related to X The distribution relationship is .

[0068] The solution to this problem can be decomposed into the following two alternating sub-problems, where: As network parameters, As a hidden variable, the subscript t is the current iteration number:

[0069] (1)

[0070] (2)

[0071] The above two problems are consistent with the EM algorithm optimization framework. The E step is mainly to solve the expectation of the latent variable, which is mainly undertaken by the predictor P1 in the network, while the M step is to minimize the loss function, which is mainly undertaken by the back propagation process in the network. The specific details of the back propagation encoder parameter update are as follows:

[0072] Problem (1) is the M-step in the EM algorithm, which seeks to minimize the expected loss during network backpropagation by updating the encoder parameters:

[0073]

[0074] , weight parameter Adopt the weight-decreasing algorithm ,i= , here an inverse proportional function is used to reduce the weight, in order to make the sub-problem Ls converge better.

[0075] It is specially noted in each iteration that the gradient calculation does not involve the gradient stop (Stop-grad) part, so the network in the gradient stop part will not be backpropagated.

[0076] The specific iteration formula in the generation is as follows:

[0077]

[0078]

[0079] The learning rate lr is subject to the decay method described below and decays according to the rounds.

[0080] It should be noted that the parameters that need to be modified in the network The network parameters of encoder D and the network parameters of predictor P1 Composition, for the convenience of expression, it is uniformly called network parameters Encoder D and predictor P Together we call it the objective function to be optimized .

[0081] Question (2) is the E step in the EM algorithm based on the current To calculate the data for The distribution of is solved using predictor P1:

[0082]

[0083] Predictor in the network , is to approximate the expectation of spatial sample points to T through a neural network Phase approximation is achieved, thus completing the calculation of step E.

[0084] in, , indicating that the variable is modified Make After minimization, the , its value is , the value in the network is the output of predictor P1.

[0085] Among them, in the tth iteration It is an encoder The output of , through the above design, will be approximately approximated when the network is fed back. , thus reaching the formula

[0086] In the actual iterative optimization process, the predictor P1 has no effect at the beginning, so , but as the iteration proceeds, the predictor P1 will gradually record the sample distribution data, making Close to expectations.

[0087] Make

[0088]

[0089] The iterative process will be repeated M times according to the set epoch number M, and each process will fully train all batches .

[0090] To promote optimization goals Convergence, the learning rate lr will decay according to the equation:

[0091]

[0092] Attenuation is performed, where t is the current cycle number, .

[0093] After the loop is finished, we will get the data X to the encoder D of the representation space.

[0094] It should be noted that the function of predictor P2 is similar to that of P1. The difference is that P2 is calculating the expectation about the target encoder FT.

[0095] Step 4: Connect the encoder D obtained in Step 3 to the linear layer of the linear network to obtain the neural network f. At the same time, obtain a small amount of annotated dataset. The dimensions of the linear network are the same as the annotation types of the annotated dataset.

[0096] In this example, there are 13 cat types in a small annotated dataset, denoted as c1…c13. A fully connected layer is used to interconnect the encoder D, from 2048 dimensions to 13 dimensions. Each dimension is proportional to the likelihood of each cat type during subsequent fine-tuning, thus obtaining the neural network f.

[0097] Step 5: Based on a small amount of labeled data sets, gradient descent is performed according to CROSSENTROPYLOSS. After N (100 in this embodiment) epochs, a neural network f suitable for the target technical task is obtained.

[0098] The neural network f concentrates the knowledge of the parent task, unlabeled target task data and a small amount of annotations in the process, so it will have excellent classification accuracy.

[0099] In summary, the directed self-supervised learning employed in this paper solves the network training problem when insufficient labeled data is available for two-dimensional image data. It also reduces the computational power consumption of self-supervised contrastive learning, alleviates its difficulty in convergence, and improves the accuracy of downstream tasks.

[0100] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0104] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0105] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0106] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0107] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

Claims

1. A task-oriented self-supervised learning method based on a contrastive twin network, characterized by: The method performs data augmentation on a large amount of unsupervised data, uses a target task encoder to guide the convergence direction of the representation space for contrastive learning, and uses the EM algorithm for training. After the network converges, it then fine-tunes the network by introducing a specific downstream task, thereby obtaining a network that can solve the target image task. The method specifically comprises the following steps: Step 1: Determine the target task encoder TF for model training and construct a self-supervised network with it, encoder D, predictor P1 and predictor P2; The encoder D has two branches in common; the output of the first branch of the encoder D is connected to the predictor P1 and the predictor P2, and the output of the predictor P1 and the output of the second branch of the encoder D are used to calculate the similarity loss , calculate the cosine similarity of the two branches in the representation space; p1 and z2 are output by the encoder D and predictor P1 respectively; The output of the predictor P2 and the output of the target task encoder TF are used to calculate the latent space similarity loss , represents the degree of similarity between the encoder’s hidden representation and the teacher network representation in the space; The total loss can be obtained based on the similarity loss and the latent space similarity loss ,in, is the hyperparameter weight factor; Step 2: Obtain the unlabeled dataset X from the target task dataset; The dataset X consists of a sample set Composed and divided into multiple batches ; Step 3: Take the unlabeled dataset X as the input of the self-supervised network. To optimize the target, iteratively optimize on the self-supervised network to obtain the encoder D from X to the target representation space; The training process is as follows: (1) Network parameter set Initialize ; (2) All batches The data is input into the network in turn for iterative training; Input the network and perform data augmentation T1 and T2 to derive two enhanced batches of samples t1 and t2; (3) The processed samples t1 and t2 are passed through the self-supervised network to obtain the total loss L. The network parameters are calculated based on the total loss L. Back propagation of the gradient updates the predictor P1 and encoder D; Optimization goal Decomposed into the following two alternating sub-problems, where: As network parameters, As a hidden variable, the subscript t is the number of iterations: （1） （2） The above two problems are consistent with the EM algorithm optimization framework. The E step is mainly to solve the expectation of the latent variable, which is mainly undertaken by the predictor P1 in the network, while the M step is to minimize the loss function, which is mainly undertaken by the back propagation process in the network. The specific details of the back propagation encoder parameter update are as follows: Problem (1) is the M-step in the EM algorithm, which seeks to minimize the expected loss during network backpropagation by updating the encoder parameters: , is the weight parameter, , is the learning rate; Question (2) is the E step in the EM algorithm based on the current To calculate the data for The distribution of is solved using predictor P1: In the actual iterative optimization process, the predictor P1 has no effect at the beginning, so , but as the iteration proceeds, the prediction period P1 will gradually record the sample distribution data, making Close to expectations; Make The iterative process will be repeated M times according to the set epoch number M, and each process will fully train all batches ; After training, the data X is obtained into the encoder D of the representation space; Step 4: Connect the fully connected layer of the encoder D obtained in step 3 to the linear layer of the linear network to obtain the neural network f; at the same time, obtain a small amount of labeled data set; Step 5: Based on a small amount of labeled data sets, perform gradient descent according to CROSSENTROPYLOSS. After N epochs, a neural network f suitable for the target technical task is obtained.

2. The self-supervised learning method with target task orientation based on contrastive twin networks according to claim 1, characterized in that: To promote optimization goals Convergence, the learning rate lr decays according to the equation: Attenuation is performed, where i is the current cycle number, .

3. The self-supervised learning method with target task orientation based on contrastive twin networks according to claim 1, characterized in that: The weight parameter Adopt the weight-decreasing algorithm 。

Images