An asymmetric mask based contrastive learning method

Abstract

The application discloses a method for image classification of semantic information of extracted images based on a contrast learning method of an asymmetric mask, mainly comprising the following steps: performing random cutting twice on an obtained original image and respectively performing different data enhancement to construct a positive sample pair; adopting an asymmetric mask strategy to sample image blocks, reducing the sampling rate of the overlapping area, and constructing a positive sample pair with large appearance difference; inputting the sampled image blocks into a pair of parameter-shared encoders, wherein one side of the encoder is updated using gradient descent, and the other side of the encoder is momentum updated to extract image representation; the image representation is respectively input into a mapping module and a prediction module, a contrast loss function is used to minimize the difference between the positive samples and maximize the difference between the negative samples; and in the training process, an adaptive cutting gradient strategy is used to stabilize the training process. The application can better capture image representation information, thereby improving the accuracy of image classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image classification, and relates to a contrast learning method based on asymmetric mask. BACKGROUND

[0002] The rapid progress of deep learning has made it a fundamental method for many modern artificial intelligence techniques. Traditional supervised training methods rely heavily on labeled data sets. This is where self-supervised methods play an important role in advancing deep learning, as they do not require expensive labeled data, i.e. do not need to provide supervised data when learning image representations. In recent years, the field of self-supervised / unsupervised learning has made great breakthroughs. Based on the difference of proxy tasks, the current popular branches include contrastive learning (CL) and masked image modeling (MIM).

[0003] For the contrastive learning task, the model needs to be trained to distinguish different views of the same image from other images, which is called instance discrimination. In order to learn effective image representations, asymmetric design plays a crucial role in contrastive learning, which introduces a series of changes in low-level features, but does not change the semantic information of the image. The most important asymmetric design is a series of data augmentation techniques applied to positive sample pairs, such as color transformation, grayscale transformation, and exposure operations, which significantly change the color of the positive sample, so this requires the model based on contrastive learning to capture color-independent representations in the image. Then, the random cropping operation introduces changes in image size and scale, which further eliminates the model's dependence on the size and scale of the image. Therefore, the model is trained to use semantic features to recognize objects rather than superficial surface features. In addition, asymmetric design is also introduced into the network architecture, such as the prediction module and momentum encoder, which change the representation of the positive sample pair. Therefore, reasonable asymmetric design is very important for the performance of contrastive learning. However, due to the overlap of image regions between positive sample pairs, there is still a lot of appearance similarity between positive sample pairs in existing contrastive learning methods, which reduces the model's ability to learn good representations.

[0004] Unlike contrastive learning, MIM tasks follow the idea of Masked Language Modeling (MLM) in Natural Language Processing (NLP), where partially masked data is fed into the model to predict the non-visible part of the data in an autoencoding manner. Due to the severe spatial redundancy of images, the masked images in MIM tasks can still effectively preserve the semantics of the original image, which achieves very good performance in self-supervised learning. However, under similar semantics, the original pixels or their image patches have large fluctuations in appearance, resulting in non-unique prediction targets for reconstructing the non-visible region from the randomly masked image, especially when the masking ratio is large. Existing MIM methods try to map the highly masked image to a fixed target, which inevitably introduces a large fitting error even if the predicted target is a reasonable target for the given input. We call this the non-unique target problem, which essentially limits the flexibility of MIM models.

[0005] To this end, some methods use feature prediction-based methods to alleviate the above-mentioned problems. Although these methods achieve good results, these methods still heavily rely on unstable targets for reconstructing pixels or image patches. Therefore, by contrast, the method we propose models the masked image by learning the instance similarity between different views sampled by an asymmetric sampling strategy, which provides a more flexible and stable target for self-supervised learning. SUMMARY

[0006] The technical problem solved by the present application is to provide a contrastive learning method based on asymmetric masking to improve the accuracy of image classification, which addresses the shortcomings of the prior art.

[0007] To achieve the above-mentioned purpose, the technical solution of the present application is as follows:

[0008] A contrastive learning method based on asymmetric masking, comprising the following steps:

[0009] Step 1, performing two random crops, random horizontal and vertical flips, and color changes on the obtained original image, etc. data augmentation to construct positive sample pairs.

[0010] Step 2, using an asymmetric masking strategy to sample image blocks, reducing the sampling rate of overlapping regions between positive samples, and constructing positive sample pairs with large appearance differences.

[0011] Step 3, respectively placing the positive sample pairs into the initial parameter shared encoder to extract image representation information. One side encoder uses gradient descent update, and the other side encoder uses momentum update.

[0012] Step 4, passing the representation obtained by the encoder through the mapping module and the prediction module, respectively, and using the contrastive loss function to minimize the difference between positive samples while maximizing the difference between negative samples.

[0013] Step 5, using adaptive clipping gradient strategy to smooth the gradient and stabilize the training process in the process of training.

[0014] Step 6, fine-tuning the model obtained by training on image classification tasks, and classifying on test dataset images.

[0015] Further, in step 1, the process of twice asymmetric data augmentation on the original image is: first, a rectangular region is cropped from the original image at a random area ratio and a random width-height ratio. Save the position and size data of the cropped region:

[0016] Box=[i,j,h,w]

[0017] Where Box represents the rectangular frame of the cropped region; i represents the left upper corner horizontal coordinate of the cropped rectangular frame, j represents the left upper vertical coordinate of the cropped rectangular frame, h represents the height of the cropped rectangular frame, and w represents the width of the cropped rectangular frame.

[0018] After cropping the image twice, the rectangular frames of the two views of the image are obtained, which are Box1 and Box2 respectively. Then, the cropped image is restored to the specified model input size using bilinear interpolation, so that the model learns the representation regardless of the image size and area. Then, the image is randomly flipped horizontally and vertically.

[0019] The torchvision transform toolkit is used for data augmentation on the image, which is a commonly used Python toolbox in the field of computer vision;

[0020] Further, in order to make the model learn the representation information regardless of color, color transformation, grayscale transformation and exposure operation are performed on the image.

[0021] Further, in step 2, due to the serious redundancy of image space, in order to increase the difficulty of the proxy task, so that the model learns deeper semantic information rather than shallow representation information. The view Figure 1 Sparse sampling is used to realize the mask operation. The mask block size is consistent with the image block size, and the total number of mask blocks is the mask ratio multiplied by the total number of image blocks.

[0022] In order to further reduce the appearance similarity between the two views of the positive samples, an asymmetric mask strategy is used to reduce the sampling rate of the image blocks in the overlapping area and increase the sampling rate of the image blocks in the non-overlapping area, so that the total number of sampled image blocks remains unchanged. Specifically, first, randomly sample the image blocks in view Figure 1 According to the sampled image blocks in view Figure 1 , the sampling rate of the image blocks in view Figure 2The image blocks to be sampled. The sampling method used is to determine the sampling rate of the image block according to the overlap ratio of the image block.

[0023] The calculation method of the overlap ratio is as follows:

[0024]

[0025] Where P i (i = 1, 2) represents the image block in view i. S(·) represents the area of the given image block. In order to reduce the sampling probability of the overlapping image block, in view Figure 2 A selective image block sampling method is used, in which the sampling probability density p sel is calculated by the following formula:

[0026] p sel = (γ + 1) · s1· (1-r overlap ) γ

[0027] Where s1 refers to the sampling probability of the image block in view Figure 1 The sampling probability of the image block in the overlapping area x1∩x2 is s1, which is the same as the probability in x1. Therefore, the sampling probability density needs to satisfy the following equation:

[0028] Since the sparse sampling strategy uniformly samples image blocks from the first view x1, the probability of sampling image blocks in the overlapping area x1∩x2 is also s1, which is the same as the probability in x1. Therefore, the sampling probability density needs to satisfy the following equation:

[0029] This ensures that the total probability satisfies the ratio (x1∩x2) of the proposed sampling strategy.

[0030] Further, in step 3, a visual Transformer is used to extract the representation information of the image. All image blocks are subjected to one-dimensional linear projection E(·) to obtain the corresponding Token embedding vector. In order to make the image not lose the position information, all Token embedding vectors are added with learnable position encoding E posIn addition to all the existing tokens, a class Token is added to obtain the global information of the image. Then, the information interaction between tokens is realized through the encoder layer of the Transformer. The structure of each layer of the encoder in the transformer structure is the same. The adopted transformer structure is composed of a series of transformer blocks with the same structure. Each transformer block is composed of a multi-head attention mechanism and a multi-layer perception mechanism. The input is normalized by using Layer Norm before each transformer block, and residual connection is used after each transformer block.

[0031] The visual Figure 1 and visual Figure 2 Token sequences are input into two encoders with the same initialization parameters, one of which uses normal gradient descent update, and the other uses momentum to update the parameters. The momentum update formula is as follows:

[0032] P t =m·P t-1 +(1-m)·P t

[0033] Where P t is the model parameter after normal gradient descent update at step t, P t-1 is the model parameter of the momentum encoder at step (t-1), and m is the momentum update coefficient.

[0034] Further, in step 4, the image representations class Token of visual Figure 1 and visual Figure 2 are mapped through the mapping module and the prediction module. Then, the contrastive loss function is used to minimize the difference between positive samples and maximize the difference between negative samples. The contrastive loss function formula is as follows:

[0035]

[0036]

[0037]

[0038] Where τ and N represent the temperature parameter and the batch size, respectively, q (i) and z (i) represent the representation q and z of the i-th sample in the data batch, respectively, and sg(·) represents the gradient clipping operation. q represents the feature after the mapping module, and z represents the feature after the mapping module and the prediction module.

[0039] Further, in step 5, in order to solve the unstable optimization problem of the visual Transformer-based contrast learning method, stabilize the training process, and set an adaptive threshold value for step t according to the exponential moving average of the gradient The formula is as follows:

[0040]

[0041] Where m∈[0, 1) is the momentum coefficient, g t t represents the gradient of the model parameters in step t. When , the gradient is scaled by the norm of the threshold :

[0042]

[0043] By setting ∈ to improve the numerical stability of gradient scaling, the amplitude of the gradient is adjusted to a reasonable range, thereby improving the training stability of contrast learning.

[0044] Further, in step 6, the pre-trained Transformer model is fine-tuned on the labeled data set. The parameters of the pre-trained model are loaded into the standard Transformer model, and the mapping layer and prediction layer parameters of the pre-trained model are discarded. In order to realize the classification task, the complete picture is cropped into an image block sequence form and input into the model, and the Token representation sequence is obtained through the Transformer model coding. The global representation class Token is taken and input into the classifier module composed of a single-layer linear network, and the cross-entropy loss function between it and the one-hot label of the labeled data is calculated to guide the model to optimize the parameters in the correct direction of classification.

[0045] Advantages

[0046] The model constructed by the application is a contrast learning neural network based on asymmetric mask, mainly including twice random cropping of the obtained original image and constructing a positive sample pair by respectively performing different data augmentation; an asymmetric mask strategy is used to sample the image blocks, the sampling rate of the overlapping area image blocks is reduced, and a positive sample pair with large appearance difference is constructed; the sampled image blocks are input into a pair of encoders with the same initialization parameters to obtain image representations, one side of the encoder uses normal gradient descent update, and the other side of the encoder uses momentum to update parameters; the image representations are respectively input into a mapping module and a prediction module, and a contrast loss function is used to minimize the difference between the positive samples and maximize the difference between the negative samples; meanwhile, an adaptive cropping gradient strategy is used to stabilize the training process. Experiments on ImageNet and CIFAR data sets show that the application can improve the accuracy of image classification and reduce the memory occupation. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a general schematic diagram of the method of the application.

[0048] Figure 2 is a schematic diagram of the application for calculating the overlapping area of the image blocks. DETAILED DESCRIPTION

[0049] As Figure 1 shown, the asymmetric mask based contrast learning method proposed by the application mainly includes the following steps:

[0050] Step 1: Select the public image classification data sets ImageNet and CIFAR. Divide the data set into training set and test set, use the training set data for pre-training, and use the test set data for fine-tuning. The pictures in the training set are subjected to two times of asymmetric data augmentation. First, a rectangular region is cropped from the original image at a random area ratio and a random width-height ratio. The area ratio of the cropped region is between 0.15 times and 1.0 times of the original image, and the width-height ratio of the cropped region is 3 / 4 to 4 / 3. The position and width-height data of the cropped region are saved.

[0051] Subsequently, the cropped image is restored to the specified model input size using bilinear interpolation, wherein the input size of the model on the ImageNet data set is 224x224, and the input size of the model on the CIFAR data set is 32x32. Then, the image is subjected to random horizontal and vertical flipping, and the probability p of horizontal and vertical flipping is 0.5. Further, the image is subjected to color transformation, grayscale and exposure operation, and finally the original data is subjected to normalization operation.

[0052] Step 2: Sample the image patches with asymmetric sampling strategy. The size of image patches is 16x16 on ImageNet dataset and 2x2 on CIFAR dataset. The sampling probability of view Figure 1 and view Figure 2 is set to 0.25, i.e., s1 = s2 = 0.25. For view Figure 1 , uniform sampling strategy is used, i.e., the sampling probability of each image patch is equal. Specifically, shuffle an array with the same length as the number of image patches, starting from 0, with a step of 1, and take the first 25% of the data in the array as the index of the image patches to be sampled. Calculate the overlap ratio of the image patches in view Figure 1 and all image patches in view Figure 2 :

[0053]

[0054] where P i (i = 1, 2) represents the sampled image patches in view i. S(·) represents the area of the given image patch. For view Figure 2 , selective sampling strategy is used, and the sampling probability density p sel is calculated by the following formula:

[0055] p sel = (γ + 1) · s1 · (1 - r overlap ) γ

[0056] where s1 refers to the sampling probability of the image patches in view Figure 1 , and γ is a hyperparameter that adjusts the sampling sensitivity, set to 3 to increase the difference between image patches.

[0057] Step 3: Input all the sampled image patches in view Figure 1 and view Figure 2 to the Transformer encoder with the same initialization parameters. All image patches are subjected to one-dimensional linear projection E(·), which is specifically using convolution operation to project all features to the specified dimension to obtain the corresponding Token embedding vector. The convolution kernel and step are both the size of the image patch, i.e., 16 on ImageNet dataset and 2 on CIFAR dataset. Taking the base version of the Transformer encoder as an example, the input channel of the convolution is 3 and the output channel is 768. All Token embedding vectors are added with learnable position encoding E pos , which has the same dimension as the embedding dimension, also 768. In addition to all Tokens, a class Token embedding is also added to obtain the global information of the image.

[0058] Then, we use MHSA with h attention heads to realize the information interaction between tokens:

[0059]

[0060] where 1≤l≤L, denotes the output of the (l-1)th layer of Transformer, and Norm denotes the batch normalization operation. After that, the method uses a two-layer FFN to perform nonlinear conversion on , and uses ReLU as the activation function between linear layers. The calculation formula is as shown below:

[0061]

[0062] where FFN is defined as FFN(x) = W2(ReLU(W1x + b1) + b2). The result obtained by taking the last layer of the Transformer is used as the final image representation. We use MHSA with h = 12 attention heads and a Transformer encoder with L = 12 layers to obtain the image representation.

[0063] Step 4, build the mapping and prediction modules, and calculate the contrastive loss.

[0064] Both the mapping module and the prediction module use 3-layer linear layers, and use the ReLU function as the activation function between linear layers, and use BatchNorm to perform feature normalization between linear layers. The contrastive loss function is used to minimize the difference between positive samples and maximize the difference between negative samples. The formula of the contrastive loss function is as follows:

[0065]

[0066]

[0067]

[0068] where τ and N represent the temperature parameter and batch size respectively, q (i) and z (i) represent the representation g and z of the i-th sample in the data batch, and sg(·) represents the gradient clipping operation. q represents the feature after the mapping module, and z represents the feature after the mapping module and the prediction module.

[0069] The gradient clipping strategy is used to stabilize the training process. According to the exponential moving average of the gradient , an adaptive threshold is set for step t, and the formula is as follows:

[0070]

[0071] where m ∈ [0, 1) is the momentum coefficient, g t denotes the gradient of the model parameters in step t. When the gradient is scaled by the norm of the threshold :

[0072]

[0073] For the ImageNet-1K dataset, we use the AdamW optimizer with a batch size of 4096, a learning rate of 1.28 x 10 -3 , a momentum update coefficient of 0.9, and a weight decay of 0.1. The learning rate is warmed up for 20 epochs, and then the remaining 780 epochs follow a cosine learning rate decay adjustment. To further stabilize training, adaptive gradient clipping operations are performed on each Transformer block, with m = 0.4 and a = 1.05. For image patch sampling, we set g = 3 to increase the appearance difference between positive sample pairs.

[0074] For the CIFAR dataset, we use the AdamW optimizer with a batch size of 512, a learning rate of 1 x 10 -3 , a momentum of 0.9, and a weight decay of 0.05. The model is trained for 1600 epochs, with the first 20 epochs used for learning rate warm-up. In addition, no gradient clipping operations are performed on the CIFAR model.

[0075] Step 5, fine-tune the above pre-trained Transformer model on the labeled dataset. Load the parameters of the pre-trained model into a standard Transformer model, discard the mapping layer and prediction layer parameters of the pre-trained model, and input the entire picture in the form of an image block sequence into the model to obtain a Token representation sequence after encoding by the Transformer model. Take the global representation class Token and input it into a classifier module composed of a single-layer linear network to calculate the cross-entropy loss function with the one-hot label of the labeled data, guiding the model to optimize the parameters in the correct direction of classification. The formula of the cross-entropy loss function is as follows:

[0076]

[0077] where q represents all learnable parameters of the model, N is the number of samples, y i is the true label of the sample, The prediction results of the model. All pre-trained models are fine-tuned for 100 epochs, and the epoch with the highest fine-tuning result is selected as the fine-tuning classification result.

Claims

1. A contrastive learning method based on asymmetric masks, characterized in that, Includes the following steps: Step 1: Perform two random cropping, random horizontal and vertical flipping, and color change data augmentation on the obtained original image to obtain the cropping box, construct positive sample pairs, and obtain View 1 and View 2 of the image respectively; Step 2: Uniformly sample the image blocks after dividing View 1, where the mask block size is the same as the image block size, and the total number of mask blocks is the mask ratio multiplied by the total number of image blocks. To further reduce the appearance similarity of positive sample pairs, an asymmetric masking strategy is used to reduce the sampling rate in overlapping areas and increase the sampling rate in non-overlapping areas. First, randomly sample the image blocks in View 1, and then determine the image blocks to be sampled in View 2 based on the already sampled image blocks in View 1. The sampling method used is to determine the sampling rate between View 2 and View 1 based on the overlap ratio of the image blocks between View 2 and View 1. The overlap ratio is calculated as follows: Among them, P i (i = 1, 2) represents the sampled image patch in view i; S(·) represents the area of ​​the given image patch; to reduce the sampling probability of overlapping image patches, a selective image sampling method is used in view 2, with a sampling probability density p sel Calculated using the following formula: p sel =(γ+1)·s1·(1-r overlap ) γ Where s1 refers to the sampling probability of the image patch in view 1, and γ is a hyperparameter for adjusting the sampling sensitivity; the larger γ is, the less likely the image patch is to be sampled; this selective sampling and sparse sampling method forms an asymmetric sampling strategy; since the sparse sampling strategy samples image patches uniformly from the first view x1, the overlapping region x1∩x2 The probability of sampling an image patch in x1 is also s1, the same as the probability in x1; therefore, the sampling probability density needs to satisfy the following equation: By using the above asymmetric masking strategy to sample image blocks, the sampling rate of overlapping areas between positive samples is reduced, thus constructing positive sample pairs with significant appearance differences. Step 3: Place the positive sample pairs into the encoder with shared initial parameters and extract the image representation information. One encoder is updated using gradient descent, while the other encoder is updated using momentum. Step 4: The image representations are processed through the mapping module and the prediction module respectively. The contrastive loss function is used to minimize the difference between positive samples and maximize the difference between negative samples. Step 5: During training, use an adaptive gradient pruning strategy to smooth the gradient and stabilize the training process. Step 6: Fine-tune the trained model on the image classification task and classify the images in the test dataset.

2. The contrastive learning method based on asymmetric masks according to claim 1, characterized in that, In step 1, the process of performing two asymmetric data augmentations on the original image is as follows: First, a rectangular region is cropped from the original image with random area and width-to-height ratios; the coordinates of the top left corner and the length and width data of the cropped region are saved as a Box. Box = [i,j,h,w] After cropping the image twice, two rectangular boxes, Box1 and Box2, are obtained for the image. Then, bilinear interpolation is used to restore the cropped image to the specified model input size, allowing the model to learn representations independent of image size and region. Subsequently, the image is randomly flipped horizontally and vertically. Further, color transformation, grayscale change, and exposure operations are performed on the image. Finally, the original data is normalized.

3. The contrastive learning method based on asymmetric masks according to claim 1, characterized in that, In step 3, a visual Transformer is used to extract the image representation information; all image patches are subjected to a one-dimensional linear projection E(·) to obtain the corresponding token embedding vectors; in order to prevent the image from losing positional information, a learnable positional code E is added to all token embedding vectors. pos In addition to all tokens, a class token embedding is added to obtain global information about the image. Then, information exchange between tokens is achieved through the Transformer encoder layer. The adopted Transformer structure consists of a series of identical Transformer blocks. Each Transformer block is composed of a multi-head attention mechanism and a multi-layer perception mechanism. LayerNorm is used to normalize the input before each Transformer block, and residual connections are used after each Transformer block. The token sequences of View 1 and View 2 are input into two encoders with the same initialization parameters. One encoder updates using normal gradient descent, while the other encoder updates parameters using momentum. The momentum update formula is shown below: P t =m·P t-1 +(1-m)·P t Where P t Let P be the parameters of the model updated by normal gradient descent at step t. t-1 Here are the model parameters of the momentum encoder in step (t-1), and m is the momentum update coefficient.

4. The contrastive learning method based on asymmetric masks according to claim 1, characterized in that, In step 4, the image representation class tokens of View 1 and View 2 are feature-mapped through a mapping module and a prediction module, where the mapping module and prediction module are implemented using a multilayer perceptron. Then, a contrastive loss function is used to minimize the difference between positive samples while maximizing the difference between negative samples. Finally, the Transformer model is pre-trained using the following contrastive loss function: Where τ and N represent the temperature parameter and batch size, respectively, q (i) and z (i) Let q and z represent the representations of the i-th sample in the data batch, respectively, and sg(·) denotes the gradient truncation operation; q represents the feature obtained after the mapping module, and z represents the feature obtained after the mapping module and the prediction module.

5. The contrastive learning method based on asymmetric masks according to claim 1, characterized in that, In step 5, a gradient pruning strategy is used to stabilize the training process, based on the gradient. The exponential moving average is used to set an adaptive threshold in step t, as shown in the following formula: Where m∈[0,1) is the momentum coefficient, g t This represents the gradient of the model parameters in step t; when At that time, gradient By threshold Norm scaling: By setting ∈, the numerical stability of gradient scaling is improved, and the magnitude of the gradient is adjusted to a reasonable range, thereby improving the training stability of contrastive learning.

6. The contrastive learning method based on asymmetric masks according to claim 1, characterized in that, In step 6, the pre-trained Transformer model is fine-tuned on the labeled dataset. The parameters of the pre-trained model are loaded into the standard Transformer model, discarding the mapping and prediction layer parameters. To achieve the classification task, the complete image is cropped into a sequence of image patches and input into the model. These patches are then encoded by the Transformer model to obtain a token representation sequence. The global representation class token is extracted and input into a classifier module composed of a single-layer linear network. The cross-entropy loss function between the token and the one-hot label of the labeled data is calculated to guide the model towards correct classification through parameter optimization. The formula for the cross-entropy loss function is as follows: In the formula, θ represents all learnable parameters of the model, N is the number of samples, and y i These are the actual labels of the samples. This represents the model's prediction results.

Images