A vision mamba-based implicit contrastive learning method

Abstract

The application discloses an implicit contrast learning method based on Vision Mamba, and realizes implicit contrast learning by guiding gradient to stop on a self-supervised learning network architecture, and better performance indexes are obtained on an image classification task; for an input image, the method obtains a positive sample pair of views of the image through data enhancement, obtains embedding representation of the views by using linear projection, obtains position information of the image in space by adding position embedding and a class label, understands global structure and local features of the image, and obtains a feature vector of the image by processing in two directions of front and back through a one-dimensional convolution layer and an S6 layer of a state space model Mamba. Gradient is guided to stop in the self-supervised learning network architecture, the feature vectors of the positive sample pair are controlled to approach in a single direction in a feature space, the actual effect that negative samples are far away is achieved, implicit contrast learning is realized, and the model performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image self-supervised learning, and particularly relates to an implicit contrastive learning method based on Vision Mamba. BACKGROUND

[0002] Contrastive learning is a self-supervised learning method in the field of machine learning and deep learning, and its core goal is to drive the model to learn good features or representations by comparing the similarities or differences between data samples. Specifically, contrastive learning ensures that similar data samples as positive sample pairs are close to each other in the feature space, and dissimilar data samples as negative sample pairs are far away from each other, so as to achieve better learning of features. Contrastive learning has many applications in computer vision and natural language processing, and is commonly used in computer vision for image classification, object detection, image segmentation, and prediction of downstream tasks. Specifically, contrastive learning can be applied to medical image analysis and diagnosis, face recognition and expression analysis, video action segmentation and prediction, etc. Contrastive learning is still an active research field, and its theory and application are still developing and improving, with broad research prospects and research value.

[0003] In the field of computer vision, contrastive learning methods extract feature representations of images through encoders. Convolutional Neural Networks (CNN) are commonly used encoders in contrastive learning models. Due to their local receptive fields and weight sharing characteristics, CNNs can capture local features well, and have relatively low parameter quantities and high computational efficiency. They have proven their strong feature extraction capabilities in large-scale image recognition tasks, but are weak in capturing long-range dependencies. Vision Transformer is another type of visual encoder that performs well in processing sequential data and can capture long-range dependencies. Due to its self-attention mechanism, Vision Transformer can process all input elements in parallel, which can provide more global context information when processing images, but usually requires more computational resources and memory. Vision Mamba is the latest visual encoder that can capture and understand global and local information in images, encode positions, and thus improve the model's understanding of visual data. It has become a new foundation architecture in the image field. The present application attempts to apply Vision Mamba to the contrastive learning model, replacing the original visual encoder in the model, to enhance the model's feature extraction capabilities and improve the model's performance.

[0004] One of the core contents of contrastive learning is to construct positive and negative sample pairs and learn feature representations through symmetric networks. Strict speaking, only the self-supervised learning method with positive and negative sample pairs is called a contrastive learning method, and the SimSiam and BYOL models have only positive sample pairs without negative sample pairs, so they can only be regarded as self-supervised learning methods with symmetric structure similar to contrastive learning. Due to the presence of the one-sided gradient stop structure in the SimSiam model network, in previous studies, the positions of the source encoder and the target encoder need to be exchanged when calculating the loss function, and the average value of two loss function calculations is taken, so the previous conclusion that the feature representations of the positive sample pairs in the SimSiam model are close to each other in the feature space. We found that when one loss function calculation is performed through the one-sided gradient stop structure in the network, the feature representations of the positive sample pairs are one-sided and unidirectional in the feature space. Therefore, based on this finding, by guiding the gradient descent to stably control the unidirectional approach between the feature representations, the SimSiam model with only positive sample pairs can also achieve the effect of negative sample repulsion, realizing an implicit contrastive learning method. SUMMARY

[0005] The purpose of the present application is to propose an implicit contrastive learning method based on Vision Mamba, which uses a Vision Mamba encoder to extract feature representations of images on the basis of a traditional SimSiam model, and controls the unidirectional approach of the feature representations of the positive sample pairs in the feature space through a gradient stop guiding method, realizes the effect of negative sample repulsion, and realizes an implicit contrastive learning method.

[0006] The technical solution of the present application is an implicit learning method based on Vision Mamba, comprising the following steps:

[0007] Step 1, given images x1 and x2, construct positive sample pairs of the images through data augmentation, i.e. views x 11 , x 12 and x 21 , x 22 ;

[0008] Step 2, divide each view into image blocks, obtain embedding vectors, add position embedding and class labels, and compose an initial input sequence T0;

[0009] Step 3, process the input sequence T l-1 of the Vision Mamba encoder lth layer into a main line vector T a and a branch line vector T b , and the main line vector T a passes through a one-dimensional convolution layer and the S6 layer of the Mamba model to obtain a forward state space vector y forwardand the backward state space vector y backward ;

[0010] Step 4, branch vector T b gate state space vector y forward and y backward , the first residual connection is performed, and the sum is added to the input sequence T l-1 the second residual connection is performed, and the output sequence T l of the l-th layer encoder is obtained, and then the final output sequence T L of the Vision Mamba encoder as a whole is obtained.

[0011] Step 5, processing the final output sequence T L to obtain the feature vector z, and the four views of the images x1, x2 correspond to the feature vectors z 11 , z 12 , z 21 , and z 22 , the nearest feature vectors z 11 , z 21 in the feature space are determined, the positive sample pair z 11 , z 12 is guided to approach z 21 , z 22 in a single direction by guiding the gradient to stop, and the negative sample z 11 , z 21 is realized to move away.

[0012] Step 6, the cosine similarity between z 11 and z 21 is calculated as a loss function, and the parameters of the self-supervised learning network framework are optimized according to the loss function to realize implicit contrastive learning.

[0013] Preferably, in step 1, the positive sample pair of the given images x1, x2 is constructed by data augmentation, that is, the views x 11 , x 12 and x 21 , x 22 , and the SimSiam self-supervised learning network architecture is constructed for the positive sample pair.

[0014] Preferably, in step 2, each view is divided into image blocks, and the embedding vector is obtained through a linear projection layer, and then the position embedding and the class label are added to form the initial input sequence T0, which is input into the Vision Mamba encoder.

[0015] Preferably, in step 3, the input sequence T l-1 of the l-th layer of the Vision Mamba encoder is divided into the main line vector T a and the branch vector T b , the main line vector Ta The forward vector T is obtained by reprocessing af and the backward vector T ab , respectively, through a one-dimensional convolution layer and the S6 layer of the state space model Mamba, updates the hidden layer parameters in S6 according to the input, and obtains the state space vector y forward and y backward .

[0016] Preferably, step 4 uses the branch vector T b processed in step 3 to gate the state space vector y forward and y backward , performs the first residual connection between the two state space vectors, adds them, and performs the second residual connection with the input sequence T l-1 , to obtain the output sequence T l of the lth layer encoder. After passing through the L-layer encoder, the final output sequence T L of the entire VisionMamba encoder is obtained.

[0017] Preferably, step 5 further processes the final output sequence T L output in step 4, through a normalization layer and a linear projection layer, to obtain the feature vector z. The two views x 11 , x 12 of the image x1 correspond to the feature vectors z 11 , z 12 , and the two views of another image x2 correspond to the feature vectors z 21 , z 22 , and the feature vectors z 11 , z 21 that are closest to each other in the feature space are determined according to the Euclidean distance. By guiding the gradient stop, z 11 is controlled to move in one direction towards z 12 , and z 21 is controlled to move in one direction towards z 22 , so as to realize the moving away of z 11 from z 21 , and further realize the moving away of the feature vectors z1 and z2, that is, by guiding the moving close of the positive sample pair, the moving away of the negative sample is realized.

[0018] Preferably, step 6 calculates the cosine similarity between z 11 and z 21 according to the result of guiding the gradient stop in step 5, takes it as the loss function of the model, and performs gradient backpropagation according to the loss function to optimize the parameters of the self-supervised learning network framework. Since the loss function of the self-supervised learning framework only reflects the one-way moving close of the positive sample pair, but actually realizes the moving away of the negative sample, an implicit contrastive learning method is realized.

[0019] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the program.

[0020] A computer readable storage medium having stored thereon a computer program, the program implementing the steps of the above method when executed by a processor.

[0021] A computer program product comprising a computer program, the computer program implementing the steps of the above method when executed by a processor.

[0022] Compared with the prior art, the present application has the following advantages: (1) the present application processes images through a Vision Mamba encoder, effectively encodes spatial information through position embedding, and uses a bidirectional S6 layer to capture context information of images from the front and rear directions, thereby achieving a more comprehensive understanding of the context of the images and obtaining feature vectors with higher efficiency and performance; (2) the present application realizes guided gradient stopping based on a traditional self-supervised learning network framework Simsiam, controls the feature vectors of positive sample pairs to move in a single direction in the feature space, and achieves the actual effect of moving away from negative samples, thereby realizing an implicit contrast learning method and improving the performance of the model.

[0023] The present application will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The present application is a flowchart.

[0025] Figure 2 The present application is a whole network framework diagram.

[0026] Figure 3 The present application is a framework diagram of the Vision Mamba module.

[0027] Figure 4 The present application is a framework diagram of the Vision Mamba encoder module.

[0028] Figure 5 The present application is a framework diagram of the S6 layer.

[0029] Figure 6 The present application is a schematic diagram of the guided gradient stopping module. DETAILED DESCRIPTION

[0030] As Figure 1 , Figure 2As shown, an implicit contrast learning method based on Vision Mamba and guided gradient stop guides the gradient stop on the self-supervised learning network architecture to realize implicit contrast learning. Given images x1, x2, the positive sample pair of the image is constructed through data enhancement, that is, views x 11 、 12 x 21 、 22 x l-1 ; each view is divided into image blocks to obtain embedding vectors, and position embedding and class labels are added to form an initial input sequence T0; the input sequence T a of the first layer of the Vision Mamba encoder is processed, which is divided into main line vector T b and branch line vector T a , the main line vector T forward passes through a one-dimensional convolution layer and the S6 layer of Mamba to obtain the forward state space vector y backward and the backward state space vector y b ; the branch line vector T forward gates the state space vectors y backward and y l-1 , and the first residual connection is performed, and the sum is added to the input sequence T l for the second residual connection to obtain the output sequence T L of the first layer of the encoder, and then the final output sequence T L of the entire Vision Mamba encoder is obtained; the feature vector z is obtained by processing the final output sequence T 11 , and the four views of the images x1, x2 correspond to the feature vectors z 12 , z 21 , z 22 , and the nearest feature vectors z 11 , z 21 in the feature space are determined, and through guided gradient stop, the positive sample pair z 11 , z 12 is guided to move in a single direction, and the negative sample z 21 , z 22 is guided to move away; the cosine similarity between z 11 and z 21 is calculated as a loss function, and the parameters of the self-supervised learning network framework are optimized according to the loss function to realize implicit contrast learning. The specific steps are as follows:

[0031] Step 1, as shown in Figure 2 , for given images x1, x2, the positive sample pair is constructed through data enhancement, that is, views x 11 、 21 x 11 、 12 x21 , x 22 , and construct the SimSiam self-supervised learning network architecture for positive sample pairs. Data augmentation mainly adopts random cropping and random horizontal flipping in geometric enhancement, color jittering and random grayscale in color enhancement, Gaussian blur in blur enhancement, and one standardization.

[0032] Step 2, as shown in Figure 3 , Figure 3 shows the network architecture of the VisionMamba module, which is an expanded explanation of the same module in Figure 2 . Each view is divided into J identical image blocks of size p, where each block is denoted by , and the overall image block sequence is denoted by x p . The image block is input into a linear projection layer to obtain an embedding vector t p , which is added to a position embedding E pos and a class label t cls to form an initial input sequence T0, which is input into the Vision Mamba encoder. The position embedding can utilize the spatial information in the sequence data, and the class label provides a vector that can aggregate the entire image information for classification or other downstream tasks in the last layer of the model.

[0033] Step 3, as shown in Figure 4 , Figure 4 shows the network architecture of the VisionMamba encoder module, which is an expanded explanation of the same module in Figure 3 . The VisionMamba encoder module has a structure of L = 24 layers. For any l-th layer, the input sequence T l-1 is divided into a main line vector T a and a branch line vector T b . The main line vector T a passes through a linear projection layer to obtain a forward vector T af and a backward vector T ab in the opposite direction. T af and T ab pass through a forward one-dimensional convolution layer and a backward one-dimensional convolution layer, respectively, to capture local features and patterns in the sequence data, obtaining a forward convolution vector T af0 and a backward convolution vector T ab0 . T af0 and T ab0 pass through an activation function layer to introduce nonlinearity, obtaining a forward activation vector x forward and a backward activation vector x backward . x forward and x backwardThe state space vector y is obtained by selective scanning through the S6 layer of Mamba, retaining important context information and filtering out irrelevant content forward and y backward . The network architecture of the S6 layer is shown in Figure 5 , which is an expanded description of the two modules with the same name in Figure 4 . S6 is a state space model, and each hidden state h k is calculated based on the current input x k and the previous hidden state h k-1 , and the output y k is calculated based on the hidden state h k . The state equation and the output equation are defined as:

[0034]

[0035] where C s is a parameter matrix, A s , B s , and Δ s are also parameter matrices, is a discrete parameter matrix obtained using A s , B s , and Δ s according to the zero-order hold technique, and is represented as follows:

[0036]

[0037] where I is the identity matrix.

[0038] Step 4, using the branch vector T b processed in step 3, after passing through the activation function layer, the state space vector y forward and y backward are gated, and the two state space vectors are first connected by residual connection. The sum is passed through a linear projection layer, and then connected by residual connection with the input sequence T l-1 for the second time, to obtain the output sequence T l of the l-th layer encoder. The initial input sequence T0 passes through the L-layer Vision Mamba encoder module to obtain the final output sequence T L .

[0039] Step 5, as shown in Figure 2 and Figure 3 , the final output sequence T L output by step 4 is passed through a normalization layer and a linear projection layer to obtain a feature vector z. The two views x 11 , x 12 of the image x1 correspond to the feature vectors z 11 , z 12The two views of the other image x2 correspond to the feature vector z. 21 z 22 The positional relationships of the aforementioned eigenvectors in the feature space are as follows: Figure 6 As shown in (a), by calculating the Euclidean distance between feature vectors, we find the feature vectors from different views that are closest to each other in the feature space, such as... Figure 6 As shown in (b) above, the Euclidean distance d is calculated using the following formula:

[0040]

[0041] Where i, j, k, l are the indices of the eigenvectors, z ij z kl Let d represent a vector from different images, ||·||2 is the L2 norm, and d ij,kl d represents the distance between two vectors from different images. m This represents the minimum Euclidean distance. Assume d m =d 11,21 Then the closest eigenvectors in the feature space are z. 11 z 21 .like Figure 2 As shown, select z 11 z 21 The Vision Mamba encoder on the network side serves as the source encoder, z 12 z 22 The VisionMamba encoder on the network side serves as the target encoder; the feature vectors enter the guided gradient stopping module, where z 11 z 21 Inputting the linear prediction layer yields the prediction vector p. 11 p 21 ; Calculate p 11 With z 12 p 21 With z 22 loss function L 11,12 and L 21,22 The calculation formula is as follows:

[0042]

[0043] Where sg(·) represents the gradient stopping operator, and D(·) represents the cosine similarity. The parameters of the source encoder are updated via gradient backpropagation, while the target encoder's gradient is stopped and its parameters are not updated via gradient backpropagation. After one-sided gradient backpropagation and parameter update, z 11 z 21 It will approach z in one direction in the feature space. 12 z 22 ,like Figure 6(c) in the formula (I) is shown. By calculating the Euclidean distance, it is ensured that the nearest feature vectors are far away from each other, the corresponding side gradient is updated, the other side gradient is stopped, and the gradient is stably guided to stop. By guiding the gradient to stop stably, z 11 and z 21 In the feature space, the far away is shown in (d) in the formula (I). Figure 6 In the feature space, the far away is shown in (d) in the formula (I).

[0044] Step 6, according to the result of guiding the gradient to stop in step 5, calculating z 11 and z 21 The cosine similarity between z

[0045]

[0046] Since the loss function of the self-supervised learning network framework only reflects the proximity of positive samples (z 11 close to z 12 , z 21 close to z 22 ), but actually realizes the effect of negative sample far away (z 11 far away from z 21 ), that is, the negative sample is not reflected in the self-supervised learning loss function, but the same effect of negative sample far away is realized. Therefore, this method realizes implicit contrastive learning.

[0047] Table 1 Comparison of linear evaluation results of image classification of different methods

[0048]

[0049] Table 1 is the comparison of the effects of the implicit contrastive learning method in the present application and other contrastive learning methods or self-supervised learning methods with symmetric structure on the image classification task. The method of the present application is recorded as Vim+GSG. The data set used is ImageNet, the training round is epoch=100, and the evaluation index is linear evaluation accuracy, which is a common index in image classification tasks. The symbol "↑" indicates that the higher the value, the higher the classification accuracy, and the better the accuracy of the method. It can be found that the method of the present application has the highest ranking on this index, which fully proves that the present application has better effects on image classification problems compared with existing contrastive learning methods.

[0050] The present application captures the context information of the image by using the Vision Mamba encoder, obtains more efficient feature vectors, and guides the gradient to stop in the self-supervised learning network framework, realizes implicit contrastive learning, and improves the model performance.

[0051] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An implicit contrastive learning method based on Vision Mamba, characterized in that, Includes the following steps: Step 1: Given images x1 and x2, construct positive sample pairs of the images through data augmentation, i.e., views x 11 x 12 and x 21 x 22 ; Step 2: Divide each view into image blocks, obtain the embedding vector, add the position embedding and category label, and form the initial input sequence T0; Step 3: Process the input sequence T of the l-th layer of the Vision Mamba encoder. l-1 The main vector T a and branch vector T b Principal vector T a After passing through a one-dimensional convolutional layer and the S6 layer of the Mamba model, the forward state space vector y is obtained. forward and the backward state space vector y backward ; Step 4, through the branch vector T b Gated state space vector y forward and y backward Perform the first residual connection, sum the results, and then combine them with the input sequence T. l-1 Perform a second residual connection to obtain the output sequence T of the l-th layer encoder. l This leads to the final output sequence T of the Vision Mamba encoder. L ; Step 5: Process the final output sequence T L The feature vector z is obtained, and the four views of images x1 and x2 correspond to the feature vector z. 11 z 12 z 21 z 22 Determine the closest eigenvector z in the feature space. 11 z 21 By guiding gradient stopping, positive samples are guided towards z. 11 z 12 With z 21 z 22 Unidirectional approach to achieve negative sample z 11 z 21 To get away from; Step 6: Calculate z 11 With z 21 The cosine similarity between the two networks is used as the loss function. Based on the loss function, the parameters of the self-supervised learning network framework are optimized to achieve implicit contrastive learning.

2. The implicit contrastive learning method based on Vision Mamba according to claim 1, characterized in that, Step 2 involves segmenting each view into image blocks, obtaining embedding vectors, adding location embeddings and category labels to form the initial input sequence T0, and inputting it into the Vision Mamba encoder.

3. The implicit contrastive learning method based on Vision Mamba according to claim 1, characterized in that, Step 3: Input sequence T of layer l of the Vision Mamba encoder l-1 The main vector T a and branch vector T b Principal vector T a Further processing yields the forward vector T af and backward vector T ab The state space vector y is obtained by processing the data through a one-dimensional convolutional layer and an S6 layer of Mamba. forward and y backward .

4. The implicit contrastive learning method based on Vision Mamba according to claim 1, characterized in that, Step 4 uses the branch vector T obtained in Step 3. b For the state space vector y forward and y backward Gating is performed by first concatenating the residuals of the two state space vectors, and their sum is then combined with the input sequence T. l-1 Perform a second residual connection to obtain the output sequence T of the l-th layer encoder. l This leads to the final output sequence T of the Vision Mamba encoder. L .

5. The implicit contrastive learning method based on Vision Mamba according to claim 1, characterized in that, Step 5: The final output sequence T output in step 4. L After processing, the feature vector z and two views x of image x1 are obtained. 11 x 12 Corresponding eigenvector z 11 z 12 The two views of the other image x2 correspond to the feature vector z. 21 z 22 The closest eigenvector z in the feature space is determined based on the Euclidean distance. 11 z 21 By guiding gradient stopping, z 11 One direction approaching z 12 , z 21 One direction approaching z 22 Thus achieving z 11 With z 21 The distancing is achieved by guiding positive sample pairs closer together to move negative samples away.

6. The implicit contrastive learning method based on Vision Mamba according to claim 1, characterized in that, Step 6: Based on the results of gradient stopping guided in Step 5, calculate z. 11 With z 21 The cosine similarity between the samples is used as the loss function; the parameters of the self-supervised learning network framework are optimized based on the loss function; since the loss function of this self-supervised learning network framework only reflects the proximity of positive sample pairs, but actually achieves the effect of negative samples moving away, implicit contrastive learning is realized.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the implicit contrastive learning method based on Vision Mamba as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the implicit contrastive learning method based on Vision Mamba as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the implicit contrastive learning method based on Vision Mamba as described in any one of claims 1-6.

Images