A feature diversification-based self-supervised image contrastive learning method and system

Abstract

The application discloses a kind of self-supervised image contrast learning method and system based on feature diversification, including model training and test, when model training is carried out, the high-dimensional feature vector of enhanced sample is extracted, i.e. The similarity probability of the low-dimensional feature vector of a normalized image is calculated, and then the probability that two normalized low-dimensional feature vectors belong to the same picture is calculated. The contrast loss is calculated from the probability that two normalized low-dimensional feature vectors belong to the same picture, and the simple distance loss is calculated from the low-dimensional feature vector of a normalized image. A high-rank regularization term is added to the contrast loss or simple distance loss, serving as the final loss function. The application can effectively avoid the model collapse problem faced by the joint embedding architecture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a self-supervised image contrast learning method based on feature diversification. BACKGROUND

[0002] In recent years, supervised learning has made rapid progress. However, to apply supervised learning in deep neural networks, sufficient labeled data is required, and the labeled data requires a large amount of manual labor, which is both time-consuming and expensive. Therefore, the current main bottleneck of supervised learning is label generation and annotation. In order to fundamentally solve the bottleneck problem of supervised learning, self-supervised methods for obtaining labels through generative learning, predictive learning, image contrast learning and other ways have emerged.

[0003] The best self-supervised model at present is achieved by a joint embedding architecture. The joint embedding architecture is divided into upper and lower branches, and the two branches are trained for different perspectives of the same image to obtain similar feature embeddings. The main challenge of the joint embedding architecture is to prevent the two branches from ignoring the input and producing the same and constant output vector, which leads to collapse, that is, model collapse.

[0004] The existing methods for solving the problem of model collapse mainly include: image contrast learning-based methods and information maximization-based methods. The image contrast learning-based method takes the current input as the positive sample and other samples as the negative sample, and uses the contrast loss function to pull the positive sample and push the negative sample; but the image contrast learning-based method requires large batch size, momentum encoder and other redundant and complex parts. Therefore, in recent years, information maximization-based methods have attracted much attention. The information maximization method maximizes the information of the embedding vector by decorrelating each pair of variables of the embedding vector to prevent model collapse. This method only uses a common distance loss function MSE Loss, and adds variance and covariance regularization terms on the basis of the loss function. SUMMARY

[0005] Therefore, the present application provides a self-supervised image contrast learning method based on feature diversification, which effectively avoids model collapse.

[0006] The present application achieves the above technical purpose by the following technical means.

[0007] A self-supervised image contrast learning method based on feature diversification:

[0008] (1) cropping and scaling a batch of N images in a classical data set, and then dividing them into enhanced samples X i and X j , and sending them into the upper and lower branches of the joint embedding architecture, respectively;

[0009] (2) Training of the joint embedding architecture

[0010] extracting a high-dimensional feature vector h of the enhanced sample i , h j ; based on the high-dimensional feature vector h i , h j , extracting a corresponding low-dimensional feature vector z i , z j , and normalizing the low-dimensional feature vector;

[0011] calculating the similarity probability of the normalized low-dimensional feature vector of an image, and then calculating the probability that two normalized low-dimensional feature vectors belong to the same image;

[0012] calculating the contrast loss from the probability that two normalized low-dimensional feature vectors belong to the same image, and calculating the simple distance loss from the normalized low-dimensional feature vector of an image; adding a high-rank regularization term to the contrast loss or simple distance loss as the final loss function;

[0013] (3) The trained joint embedding architecture is tested to determine the top1 and top5 accuracy, and the model parameters at the highest accuracy are used as the model parameters for the downstream task.

[0014] Further, the extracting of the corresponding low-dimensional feature vector z i , z j is specifically:

[0015] z i =g(h i )=W (2) σ(W (1) h i )

[0016] z j =g(h j )=W (2) σ(W (1) h j )

[0017] wherein g(·) represents a projection head structure for extracting a low-dimensional feature vector, W (1) , W (2) represent a dimension reduction weight matrix, and σ represents a ReLU nonlinear activation function.

[0018] Further, the similarity probability of the normalized low-dimensional feature vector of an image is:

[0019]

[0020] wherein τ sis an adjustable temperature parameter when calculating the similarity probability, and is a normalized low-dimensional feature vector.

[0021] Further, the probability that the two normalized low-dimensional feature vectors belong to the same picture is:

[0022]

[0023] Further, the contrast loss is:

[0024]

[0025] Further, the simple distance loss is:

[0026]

[0027] Further, the high-rank regularization term is -‖Z i,j ‖ * , and the high-rank kernel norm where θ is the network parameter, ||·|| * is the sum of the singular values of the matrix, denotes the forward propagation process.

[0028] Further, the test includes KNN test and linear classification test.

[0029] Further, the high-dimensional feature vector of the enhanced sample is extracted by a skeleton network ResNet50, and the low-dimensional feature vector is extracted by a projection head composed of a double-layer fully connected network.

[0030] A self-supervised image contrast learning system based on feature diversification, comprising:

[0031] A data enhancement module for cropping and scaling images in a classic dataset and dividing them into enhanced samples;

[0032] A feature extraction module for extracting high-dimensional feature vectors of the enhanced samples;

[0033] A feature dimension reduction module for extracting low-dimensional feature vectors corresponding to the high-dimensional feature vectors and normalizing the low-dimensional feature vectors;

[0034] A loss function design module for adding a high-rank regularization term to the contrast loss or simple distance loss as the final loss function;

[0035] A model test module for testing the trained joint embedding architecture to determine the top1 and top5 accuracy.

[0036] The beneficial effects of the present application are:

[0037] (1) The present application focuses on making the last output features of the joint embedding architecture recover the low-dimensional feature vectors processed by the projection head when the loss function acts, so that the low-dimensional feature vectors are more accurate, the diversity of feature representation is increased, and the possibility of feature vector converging to the same value is reduced, effectively avoiding model collapse.

[0038] (2) The present application combines a high-rank regularization term to improve the effect of self-supervised contrast method using contrast loss alone; the low-dimensional feature vectors obtained by the projection head may be damaged or deviate from the ideal model, and the high-rank regularization term can recover the low-dimensional feature vectors and separate similar samples, so that the method extracts better feature embedding based on the features extracted by the skeleton network and the projection head.

[0039] (3) The present application first introduces a high-rank regularization term in the existing self-supervised learning field as a way to solve model collapse, and uses the loss function plus the high-rank regularization term, even when using a simple distance loss, still can get an observable correct rate, this simple and efficient method can provide a novel idea for the development of self-supervised learning in the future. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a training flowchart of the joint embedding architecture. DETAILED DESCRIPTION

[0041] The present application will be further described below in combination with the drawings and specific embodiments, but the protection scope of the present application is not limited thereto.

[0042] In order to realize more efficient and more accurate self-supervised learning task, the present application provides a self-supervised image contrast learning method based on feature diversification, which includes five steps: data enhancement, extracting features of enhanced samples, extracting low-dimensional features of sample features, adding a high-rank regularization term to the main loss term, and testing the model. The present application first performs data enhancement on the input sample, then extracts the features of the enhanced sample through the ResNet skeleton network, then sends the features extracted by the skeleton network into the projection head for dimension reduction, then modifies the loss function, adds a high-rank regularization term to the ordinary function (i.e. non-contrast loss function) to obtain the final loss function, completes the model training, and finally tests the model through KNN (K nearest neighbor method) and linear classification. The specific steps are as follows:

[0043] Step (1), data enhancement

[0044] One of the classic data sets of CIFAR10, CIFAR100, STL10, imagnet-100, imagenet-1K, etc. is adopted, and two enhanced samples are obtained by using basic operations (scaling, cropping) on the basis of the original classic data set, which are respectively sent into the upper and lower branches of the joint embedding architecture; the data enhancement process is as shown in the data enhancement part of Figure 1 The data enhancement process is as follows:

[0045] Step (1.1), given a batch of one of the classic data sets, the batch contains N images X∈R N×W×H×3 , where R represents a real number, W is the width of the image, and H is the height of the image;

[0046] Step (1.2), randomly crop N images, then adjust the size back to the original size, and perform random color distortion and random Gaussian blur, and then divide them into two enhanced samples X i , X j :

[0047] X i = t i (X) (1)

[0048] X j = t j (X) (2)

[0049] Where t i is the basic operation performed on the enhanced sample sent into the upper branch, t j is the basic operation performed on the enhanced sample sent into the lower branch, where two enhanced samples of one image are represented as x i , x j , and x i ∈X i , x j ∈X j ; the basic operation can also be rotated and flipped according to actual needs.

[0050] Step (2), extracting features of enhanced samples

[0051] Taking ResNet50 as the skeleton network, the features of the enhanced samples are extracted through the convolutional layers in the network, and the process of extracting the features is as shown in the feature extraction part of Figure 1 The specific process of extracting the features of the enhanced samples through the skeleton network ResNet50 can be calculated by the following formula:

[0052] h i = f(x i ) = ResNet50(x i ) (3)

[0053] h j =f(x j )=ResNet50(x j ) (4)

[0054] wherein f(·) is an encoder;

[0055] The present application allows various selections of network architecture without any restrictions, such as ResNet18, ResNet32, etc. network model, ResNet50 is used as the skeleton network in the present application, and the output h i , h j is obtained through five stages (Stage 0, Stage 1, …) of ResNet50. In this network, features are extracted by convolutional layers and image size is reduced and computational load is reduced by pooling layers.

[0056] Step (3), extracting low-dimensional features of enhanced sample features and normalizing

[0057] The high-dimensional feature vector output by the skeleton network is mapped to a low-dimensional space by a projection head composed of a double-layer MLP (fully connected network), to obtain a low-dimensional feature vector. The process of extracting low-dimensional feature vectors z i , z j from high-dimensional feature vectors h i , h j using the projection head can be calculated by equations (5) and (6) as shown in the feature dimension reduction part of the following equation: Figure 1

[0058] Step (3.1), the high-dimensional feature vector h i , h i is mapped to a low-dimensional feature space by the two-layer internal structure FC->BN->ReLU->FC of the MLP layer of the projection head, to obtain the mapped low-dimensional feature vector z i :

[0059] z i =h(h i )=W (2) σ(W (1) h i ) (5)

[0060] z j =g(h j )=W (2) σ(W (1) h j ) (6)

[0061] wherein g(·) represents the projection head structure for extracting low-dimensional feature vectors, W (1) ​, W (2) denotes the weight matrix of dimension reduction (i.e. two-layer MLP), and σ denotes a ReLU nonlinear activation function;

[0062] Step (3.2), respectively calculating ‖h i ‖2, ‖h j ‖2, ‖z i ‖2and ‖z j ‖2for L2 normalization of features when the joint embedding architecture outputs:

[0063]

[0064]

[0065]

[0066]

[0067] where n h denotes the dimension of vector h i , h j , n z denotes the dimension of vector z i , z j , ‖h i ‖2denotes the L2 norm of h i , ‖h j ‖2denotes the L2 norm of h j , ‖z i ‖2denotes the L2 norm of z i , and ‖z j ‖2denotes the L2 norm of z j ;

[0068] Step (3.3), L2 normalizing high-dimensional feature vectors h i , h j and low-dimensional feature vectors z i , z j , i.e. dividing each feature vector by its L2 norm, as follows:

[0069]

[0070]

[0071]

[0072]

[0073] Step (4), adding a high-rank regularization term

[0074] respectively, as the main loss term, and on this basis, add the high-rank regularization term to obtain the final loss function, and the normalized low-dimensional feature vector is input into the final loss function; The specific process of constructing the loss function is as follows, as shown in the part of designing the loss in Figure 1

[0075] Step (4.1), the similarity probability of the normalized low-dimensional feature vector of an image is calculated using cosine similarity:

[0076]

[0077] Where τ s is an adjustable temperature parameter when calculating the similarity probability;

[0078] Step (4.2), compare and the enhanced samples of all images in the batch, and use softmax to calculate the probability that the two normalized low-dimensional feature vectors belong to the same picture:

[0079]

[0080] Step (4.3), taking InfoNCE as a typical contrastive loss, the negative logarithm of is taken to calculate the contrastive loss based on formula (16):

[0081]

[0082] Taking MSE as a representative to calculate the simple distance loss:

[0083]

[0084] Where, is the calculation formula of the two-norm;

[0085] Step (4.4), calculate the high-rank regularization term from a global perspective, and the high-rank kernel norm is as follows:

[0086]

[0087] Where θ is the network parameter, and ||·|| * is the sum of singular values of a matrix, represents the forward propagation process, that is, after data enhancement, skeleton network and projection head, the negative high-rank kernel norm is taken as the high-rank regularization term, that is, -‖Z i,j ‖ * ;

[0088] Step (4.5), calculate the final loss function, add the high-rank regularization term to the contrastive loss to obtain the final loss function, as shown below:​

[0089] L = L(i,j) Inf o NCE -‖Z i,j ‖ * (20)

[0090] The final loss function is obtained by adding a high-rank regularization term to the simple distance loss as follows:

[0091] L = L(i,j) MSE -‖Z i,j ‖ * (21)

[0092] Step (4.6), back propagation is performed to reduce the contrast loss or simple distance loss, and the Adam optimizer is used to find the optimal learning rate to update the joint embedding architecture parameters.

[0093] Through steps (2)-(4), the training of the model (joint embedding architecture) is completed.

[0094] Step (5), test the model (i.e., joint embedding architecture) by KNN

[0095] Test the trained model by KNN clustering, and save the joint embedding architecture parameters with the highest accuracy in each epoch to the pth file. The specific process of testing the model by KNN is as follows:

[0096] Step (5.1), store the features of the training set in the feature bank, and then calculate the cosine similarity between the features in the feature bank and the features feature (i.e., normalized high-dimensional feature vector) obtained by training the test set to obtain a similarity matrix (similar matrix):

[0097]

[0098] where: τ m is an adjustable temperature coefficient when calculating the similarity matrix;

[0099] Step (5.2), sort the similarity matrix, and take the top k most similar test set samples by the topk algorithm. The labels of the k similar samples are obtained as the candidate label set of the test samples, and the similarity weight (sim_weight) is also obtained:

[0100] sim_weight = topk(sim_matrix) (23)

[0101] Step (5.3), multiply the label and similarity weight of the similar sample to obtain the predicted score, sort the score, calculate the top1 and top5 accuracy respectively, save the model parameters when the top1 (or top5) accuracy is the highest to the model file pth, and use the parameters in the model file as the model parameters of the downstream task KNN clustering, and predict the image label when performing the downstream task KNN clustering.

[0102] If the downstream task is linear classification, after the KNN test model, a linear classification test model is also needed.

[0103] Step (6), test the model by linear classification

[0104] Read the pth file to obtain the model parameters, remove the projection head, and test the linear classification. The specific process of the linear classification test model is as follows:

[0105] Step (6.1), read the pth file, and directly add a fully connected layer after data augmentation and ResNet50 skeleton network in the fixed parameter model to obtain the output label probability, as shown in equations (24) and (25). out is h i the label probability vector:

[0106] out=fc(h i ) (24)

[0107] out=fc(h j ) (25)

[0108] Where: fc(·) is a fully connected layer;

[0109] Step (6.2), calculate the cross entropy loss (Cross Entropy Loss) of the label probability out output by the last fully connected layer and the label target in the test set, to obtain the predicted probability of the label, as shown in equation (26):

[0110]

[0111] Step (6.3), sort the predicted probability, calculate the top1 and top5 accuracy respectively, save the model parameters when the top1 (or top5) accuracy is the highest to the model file pth, and use the parameters in the model file as the model parameters of the downstream task linear classification to predict the label.

[0112] A self-supervised image contrast learning system based on feature diversification, comprising a data enhancement module, a feature extraction module, a feature dimension reduction module, a loss function design module and a model test module;

[0113] a data augmentation module configured to crop and scale N images in a batch of images in a classical dataset to obtain augmented samples;

[0114] a feature extraction module configured to extract high-dimensional feature vectors of the augmented samples;

[0115] a feature dimension reduction module configured to extract low-dimensional feature vectors corresponding to the high-dimensional feature vectors and normalize the low-dimensional feature vectors;

[0116] a loss function design module configured to add a high-rank regularization term to a contrastive loss or a simple distance loss to obtain a final loss function;

[0117] a model test module configured to test the trained joint embedding architecture to determine top1 and top5 accuracies.

[0118] The self-supervised image contrast learning system based on feature diversification can be implemented in the form of a computer program, which can run on a computer device, which can be a terminal or a server. The server can be a standalone server or a server cluster; the terminal can be a tablet computer, a desktop computer, a notebook computer, a mobile phone, a wearable device, or a personal digital assistant, etc.

[0119] The computer device comprises a network interface, a memory and a processor connected by a system bus. The memory can include a memory storage and a non-volatile storage medium; the non-volatile storage medium can store a computer program and an operating system. The computer program comprises program instructions which, when executed, enable the processor to perform any feature-diversified self-supervised image contrast learning method. The processor can provide control and computing power to ensure the smooth operation of the entire computer device. The memory provides a running environment for the computer program in the non-volatile storage medium, and when the computer program is executed by the processor, the processor can execute any feature-diversified self-supervised image contrast learning method. The network interface can perform network communication, such as sending assigned tasks, etc.

[0120] It should be appreciated that a processor can be a Digital Signal Processor (DSP), a Central Processing Unit (CPU), other general purpose processors, a Field-Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), or other programmable logic device, discrete hardware component, transistor logic, discrete gate, or the like. The general purpose processor can be a microprocessor or any conventional processor, or the like.

[0121] The embodiments of the present application also provide a computer readable and writable storage medium storing a computer program, the computer program comprising program instructions. The processor executes the program instructions to implement the feature diversification based self-supervised image contrast learning method of the present application. The computer readable and writable storage medium can be an internal storage unit of the computer device, such as a hard disk of a memory of the computer device. The computer readable and writable storage medium can also be an external storage device of the computer device, such as a Smart Media Card (SMC), a plug-in hard disk, a Flash Card, a Secure Digital (SD), or the like.

[0122] The embodiments are preferred embodiments of the present application, but the present application is not limited to the above embodiments. Any obvious improvement, replacement or modification made by those skilled in the art without departing from the essential content of the present application shall fall within the protection scope of the present application.

Claims

1. A self-supervised image contrastive learning method based on feature diversification, characterized in that: (1) Crop and scale N images in a batch from the classical dataset, then split into augmented samples X i and X j , respectively, into the upper and lower branches of the joint embedding architecture; (2) Training of the joint embedding architecture extracting a high-dimensional feature vector h of the enhanced sample i , h j ; based on the high-dimensional feature vector h i , h j , extracting a corresponding low-dimensional feature vector z i , z j , and normalizing the low-dimensional feature vector The similarity probability of the normalized low-dimensional feature vector of an image is calculated, and then the probability that two normalized low-dimensional feature vectors belong to the same picture is calculated; The contrastive loss is calculated from the probability that two normalized low-dimensional feature vectors belong to the same picture, and the simple distance loss is calculated from the normalized low-dimensional feature vector of an image; a high-rank regularization term is added to the contrastive loss or simple distance loss as the final loss function; (3) The joint embedding architecture after training is tested to determine the top1 and top5 accuracy, and the model parameters at the highest accuracy are used as the model parameters for the downstream task.

2. The feature diversification based self-supervised image contrastive learning method of claim 1, wherein, The extracted corresponding low-dimensional feature vector z i , z j , specifically: z i = g(h i ) = W (2) σ(W (1) h i ) z j = g(h j ) = W (2) σ(W (1) h j ) where g(·) denotes the projection head structure to extract the low-dimensional feature vector, W (1) , W (2) denotes the weight matrix for dimension reduction, and σ denotes a ReLU nonlinear activation function.

3. The feature diversification based self-supervised image contrastive learning method of claim 1, wherein, The similarity probability of the normalized low-dimensional feature vector of an image is: where: τ s is an adjustable temperature parameter when computing the similarity probability, and is the normalized low-dimensional feature vector.

4. The self-supervised image contrastive learning method based on feature diversification of claim 3, wherein, The probability that two normalized low-dimensional feature vectors belong to the same picture is:

5. The self-supervised image contrastive learning method based on feature diversification of claim 4, wherein, The contrastive loss is:

6. The self-supervised image contrastive learning method based on feature diversification according to claim 3, characterized in that, The simple distance loss is:

7. The self-supervised image contrastive learning method based on feature diversification according to claim 3, characterized in that, The high-rank regularization term is -‖Zi ,j ‖ * and high-rank kernel norm where θ is the network parameter, ||·|| is the matrix singular value, * is the sum of the matrix singular values, denotes the forward propagation process.

8. The feature diversification based self-supervised image contrastive learning method of claim 1, wherein, The test includes KNN test and linear classification test.

9. The feature diversification based self-supervised image contrastive learning method of claim 1, wherein, The high-dimensional feature vector of the enhanced sample is extracted by the skeleton network ResNet50, and the low-dimensional feature vector is extracted by a projection head composed of a double-layer fully connected network.

10. A system implementing the self-supervised image contrastive learning method based on feature diversification of any one of claims 1-9, characterized in that, It includes: A data augmentation module for cropping and scaling images in a classic dataset and dividing them into enhanced samples; A feature extraction module for extracting high-dimensional feature vectors of the enhanced samples; A feature dimension reduction module for extracting low-dimensional feature vectors corresponding to the high-dimensional feature vectors and normalizing the low-dimensional feature vectors; A loss function design module adds a high-rank regularization term to the contrastive loss or simple distance loss as the final loss function; A model testing module for testing the trained joint embedding architecture to determine the top1 and top5 accuracy.

Images