Image self-supervised learning method and system fusing contrast learning and feature mask modeling

Abstract

The invention discloses an image self-supervised learning method and system fusing contrast learning and feature mask modeling. An original image which is low in cost and does not need to be labeled is used as training data, a double-branch self-supervised learning network is constructed, one branch is a comparison learning branch based on a SimCLR framework, the other branch is a mask self-encoder branch based on a mask feature model, and the whole network comprises an image preprocessing module, an image feature extraction module and a shared weight encoder. And carrying out self-supervised training by combining the two branches with a loss function specially designed for the two branches, and finally obtaining a pre-training model capable of fully extracting image features. According to the method, the advantages of comparative learning and feature mask modeling are combined, information is effectively extracted from unlabeled image data, and image understanding is enhanced. The method has the main advantages that the robustness is high, the method is particularly suitable for limited data sets in the remote sensing and medical fields, the model precision is improved, and the data requirement and labeling cost are reduced.

Description

Technical Field

[0001] The present invention designs an image self-supervised learning method that integrates contrastive learning and feature mask modeling. This method combines contrastive learning and mask self-encoding, two self-supervised learning methods, to effectively learn the robust feature representation of the image, reduce the model's dependence on the data set in downstream tasks, and improve the accuracy of image classification, segmentation, target recognition and other tasks in small sample conditions. This method has broad application prospects in remote sensing image interpretation, medical image processing, and autonomous driving. Background Art

[0002] In deep learning model training tasks, the backbone network is usually used for feature extraction first, and then the extracted features are sent to downstream tasks such as image classification, object detection, and image segmentation. In this process, due to the large number of parameters and high training difficulty of the backbone network, it is usually necessary to load pre-trained parameters. These pre-trained parameters are usually obtained by training on large-scale data sets (such as ImageNet, etc.), have strong feature expression capabilities, and can greatly reduce the difficulty of training network models. However, when we are in a new field and face new tasks without sufficient labeled data, we need to obtain the pre-trained parameters of the backbone network through self-supervised learning and other methods to improve the feature extraction ability of the backbone network.

[0003] Self-supervised learning (SSL) is a very popular research direction in the field of deep learning. Unlike traditional supervised learning, self-supervised learning does not require true value label data, and its supervision information comes from the data itself or its representation characteristics. By designing auxiliary tasks, self-supervised learning can improve the feature extraction ability of the model. Among them, contrastive learning and feature mask modeling are two particularly effective self-supervised learning strategies.

[0004] Contrastive Learning is an important self-supervised learning method that shows great potential in the field of image processing. Its basic idea is to maximize the similarity between positive samples and minimize the similarity between negative samples through a carefully designed model structure and contrastive loss function. This method effectively utilizes the potential structure in unlabeled data and promotes the development of representation learning, especially in a variety of computer vision tasks such as image recognition, classification, segmentation, and multimodal matching. A typical contrastive learning framework is SimCLR, which achieves effective feature learning by maximizing the consistency between different enhanced views of the same instance while minimizing the consistency between different instance views. The success of SimCLR shows that even when trained on large-scale unlabeled datasets, high-quality representations can be learned through simple data augmentation strategies.

[0005] Feature mask modeling is another powerful self-supervised learning strategy, which learns the relationship between global and local features by predicting the masked parts. In the field of natural language processing (NLP), self-supervised learning methods represented by masked autoencoders in BERT have achieved great success. In the field of computer vision, the idea of ​​masked autoencoders is also applicable. BEiT, as the first model to bring mask prediction mode into the field of computer vision, also adopts the ViT architecture. BEiT mainly consists of two stages: the first stage is to establish the mapping relationship between the original image and the word vector matrix, and map the blocks divided by the original image to the corresponding word vectors by training the discrete variational autoencoder (DVAE); the second stage is to randomly mask a certain proportion of the image blocks, and use the word vectors corresponding to the masked image blocks as supervision data for training. Later, the image mask-based autoencoder algorithm (MAE) proposed by He Kaiming's team has achieved great success in the field of computer vision. The characteristics of this autoencoder algorithm include three aspects: first, it adopts the Vision Transformer (ViT) architecture to solve the problems of mask tokens and position embedding; second, it reduces information redundancy by randomly masking most of the image blocks, resulting in a challenging self-supervision task, so that the trained model goes beyond the overall understanding of low-level image statistics; third, it designs an asymmetric encoder-decoder design, and only performs encoder operations on visible image blocks (blocks without masks), which significantly reduces time and memory consumption. The core of this method lies in how to choose the masking strategy and how to design the decoder to restore the missing information. The advantage of the masked autoencoder is that it can learn a more complete and contextual feature representation, which is very important for understanding the overall structure of the image.

[0006] Although contrastive learning and feature mask modeling have each demonstrated powerful feature learning capabilities under unsupervised or weakly supervised conditions, their combination in the field of image analysis is still in its infancy and there is a huge space for exploration. To this end, the present invention designs a novel self-supervised learning method that combines contrastive learning and feature mask modeling strategies, aiming to utilize their respective advantages to improve the effect of image feature learning. Summary of the invention

[0007] In view of the problems existing in the field of deep learning image processing, such as the lack of large-scale annotated data sets and the difficulty of model training, the present invention proposes an image self-supervised learning method integrating contrast learning and feature mask modeling, which specifically includes the following steps:

[0008] Step 1, construct and train a high-resolution image self-supervised learning network that integrates contrastive learning and feature mask modeling. The self-supervised learning network includes two branches: one is a contrastive learning branch based on the SimCLR architecture, namely the CLR branch, and the other is a masked autoencoder branch based on the mask feature model, namely the MFM branch. The specific processing process is: first, the input image is preprocessed, and the image after data enhancement is output, and the corresponding data set is constructed according to the masked autoencoder and contrastive learning branches. Then, multi-scale high-order features are extracted from the image of the data set through the convolutional feature extraction module, and the high-order feature map is flattened and sent to the CLR branch. The high-order feature map is randomly masked and sent to the MFM branch. The two sequences obtained by the flattening operation and the random masking are further encoded using a shared weight encoder to obtain new features with global context information of the entire image in the CLR branch and the MFM branch, respectively. Finally, a set of feature vectors are obtained through a CLR decoder, and the final predicted image is obtained through an MFM decoder.

[0009] During the training process, different loss functions are used to train the CLR branch and the MFM branch;

[0010] Step 2: construct a semantic segmentation network based on the self-supervised learning network. The backbone network of the semantic segmentation network includes the convolutional feature extraction module and the shared encoder in the self-supervised learning network in step 1. The parameters obtained by training the self-supervised learning network are used, and a decoding network is added on this basis. Image segmentation can be achieved using the constructed semantic segmentation network.

[0011] Furthermore, preprocessing includes data enhancement and positive and negative sample production, among which image data enhancement includes three methods: color jittering, random cropping, and random flipping; positive and negative sample production is as follows: for each batch of input images, the number is set to N, and two new sets of image data are obtained by using data enhancement methods. Then, one set of images is used as a benchmark, and all images in the other group are teamed up to construct positive and negative samples. Each image has one positive sample and N-1 negative samples.

[0012] Furthermore, the convolutional feature extraction module includes a convolutional layer with a convolution kernel of 7×7 and the first three bottleneck units of Resnet50 in sequence. The bottleneck unit is the basic structural block of Resnet50, and each structural block contains three convolutional structures. The convolution kernel size used in the first convolutional structure is 1×1, and the convolution step size is 1, followed by a BN layer and a Relu layer; the second convolutional structure is the same as the first structure, except that the convolution kernel size is 3×3; the third convolutional structure is also the same as the first convolution structure, with a convolution kernel size of 1×1; the above three convolutional structures plus residual connections form a bottleneck unit; the convolutional feature extraction module has a total of three bottleneck units. After each bottleneck unit, the height and width of the output feature map become half of the input of the unit. After a 7×7 convolutional layer and three bottleneck units, the side length of the feature map output by the convolutional feature extraction module becomes 1 / 16 of the original image.

[0013] Furthermore, the output feature of the convolutional feature extraction module is denoted as f. For the contrastive learning branch, the two-dimensional feature map is flattened into a one-dimensional sequence to obtain a new one-dimensional feature f1. For the MFM branch, the two-dimensional feature map is first masked, and then the unmasked part is flattened into a one-dimensional sequence to obtain a new feature f2.

[0014] Furthermore, the shared weight encoder is composed of several stacked Transformer units, each of which is composed of two LayerNorm layers, a multi-head attention mechanism and an MLP layer, and the feature size of the input and output of each Transformer unit is the same; the feature encoding process of the Transformer feature encoding part is to first process the sequences f1 and f2 output by the previous part, add position encoding to the feature vectors in the sequence, and record the positions of these feature vectors in the sequence. After that, the two sequences with position encoding added are respectively input into a series of Transformer units connected in series to obtain a deep encoding sequence; the specific formula of the multi-head attention mechanism is:

[0015]

[0016] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0017] head i =Attention(QW i Q ,KW i K,VW i V )

[0018] Among them, Q represents the input at the current moment, K represents the information in memory, V represents the information to be paid attention to, and d k represents the key of each head and the dimension of the query vector. The softmax function is a function that converts multiple values ​​into probability distributions. Concat is a matrix concatenation operation. i Q , W i K , W i V is the transformation matrix, which projects the vector into a different subspace;

[0019] The calculation process of each Transformer unit is as follows:

[0020] z=MLP(norm(MultiHead(norm(f))))

[0021] Where f represents the input feature sequence, i.e. f1 or f2, and z represents the output feature sequence; MLP is a neural network composed of multiple fully connected layers, which is used for nonlinear transformation, norm is standardization processing, and MultiHead is a multi-head attention network.

[0022] Furthermore, the CLR branch input is a set of image pairs, assuming that the features output by the shared weight encoder of the image pair are z t and z c The CLR decoder first reshapes it from a 2D sequence of size HW×C into a 3D feature map of size H×W×C through a reshape operation, and adjusts the number of channels through a 1×1 convolution operation. Then, the feature map is averaged and pooled avgPool and fully connected fc operations are performed to map the global feature vector to a lower dimensional space. The features are further processed and compressed to extract higher-level semantic information, namely, the feature vector (F t , F c ), where the CLR decoder process is expressed by the following formula:

[0023] F = fc(AveragePooling(Conv 1×1 (reshape(z))))

[0024] Where z is a set of feature sequences output by the shared weight encoder, reshape is the operation of reshaping the feature map, AveragePooling represents the global average pooling of the feature map, and F is a set of feature vectors obtained after processing by the CLR decoder.

[0025] Furthermore, the input of the MFM decoder is the unmasked features processed by the shared weight encoder, and the final predicted image is obtained through three serial operations: mask recovery, Transformer decoding block, and convolution upsampling;

[0026] The mask recovery part restores the original order of the sequence of the previous deep coding, and replaces the masked vacant positions with learnable vectors. The decoded result of this learnable vector is the prediction result of the original mask position. The Transformer decoding block will initially process the padded sequence. This part consists of 6 series-connected Transformer units. Convolution upsampling reshapes the sequence output by the Transformer decoding block into a rectangle, and further uses 4 convolution upsampling units to decode it to restore the original image. Each convolution upsampling unit consists of 1 3×3 convolution and 1 two-fold interpolation upsampling. After three convolution upsampling, it is restored to the original size. The final image prediction head converts the number of channels to 3 through convolution and outputs the original color image.

[0027] The MFM decoder is represented by the following formula:

[0028] u1=Upsample(Conv 3×3 (reshape(TR(z)))

[0029] u2=Upsample(Conv 3×3 (u1))

[0030] u3=Upsample(Conv 3×3 (u2))

[0031] I pre =Conv 1×1 (Upsample(Conv 3×3 (u3)))

[0032] Here the input z is the sequence encoded by the shared weight encoder, TR represents the Transformer unit, Conv 3×3 is a 3×3 convolution, Upsample is an upsampling operation, I pre It is the output of the MFM decoder, i.e. the reconstructed image.

[0033] Furthermore, the CLR branch adopts the infoNCE Loss loss function, and the MFM branch adopts the L1Loss loss function;

[0034] The infoNCE Loss calculation formula is as follows:

[0035]

[0036] In the above formula, N is the batch size, z i and z j are the feature vectors of two different views of the same sample, which are positive samples of each other; sim(z i ,z j ) is z i and z j The similarity between them is calculated using the dot product; τ is the temperature parameter used to control the sharpness of the similarity distribution; 1 [k≠i] is an indicator function, indicating that k is not equal to i; z k Yes i The corresponding negative samples;

[0037] The L1Loss calculation formula is as follows:

[0038]

[0039] Among them, f(x i ) is the predicted image obtained by MFM decoding, x i is the original input image, y i is the original image as the true value; N′={1,1,1,…,1} is a one-dimensional matrix with all 1s, M is a mask matrix, which is a one-dimensional matrix with element values ​​of 0 or 1, where 0 represents the masked block and 1 represents the unmasked block; the elements of N'-M that are 1 correspond to the masked blocks in the feature map, and sum represents the sum of the elements in N'-M, that is, the number of masked blocks;

[0040] During back propagation, the above two loss functions are weighted, and the specific calculation is as follows:

[0041] loss=0.1×InfoNCELoss(z1′,z1″)+L1loss(f(x),y)

[0042] Where z1' and z1" are the feature vectors decoded by the CLR branch, x is the input original image, y is the original image as the true value, and f(x) is the predicted image decoded by the MFM branch.

[0043] Furthermore, the decoding network in step 2 uses 4 convolutional upsampling units for decoding, and adds a skip connection between the convolutional feature extraction module and the convolutional upsampling unit to fuse high-resolution features from different scales of the encoder.

[0044] The present invention also provides an image self-supervised learning system integrating contrast learning and feature mask modeling, comprising:

[0045] one or more processors;

[0046] A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement an image self-supervised learning method that integrates contrast learning and feature mask modeling as described in the above scheme.

[0047] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0048] The present invention designs an image self-supervised learning method that integrates contrastive learning and feature mask modeling. Using low-cost, unlabeled original images as training data, a two-branch self-supervised learning network is constructed, which includes a contrastive learning branch (CLR branch) based on the SimCLR architecture and a masked autoencoder branch (MFM branch) based on a masked feature model. The CLR branch uses the contrastive learning principle to learn robust feature representation by maximizing the similarity between similar samples and the difference between different samples; while the MFM branch predicts the randomly masked pixel area through the masked autoencoder, thereby learning a more comprehensive feature representation. The two branches share the same encoder to ensure that the learned features can reflect the distinguishability of contrastive learning and maintain the integrity of the masked autoencoder. At the same time, a comprehensive loss function is designed to balance the objectives of the two branches and guide network learning. By combining the advantages of contrastive learning and feature mask modeling, the present invention effectively extracts information from unlabeled image data and enhances image understanding. The main advantages are: strong robustness, particularly suitable for limited data sets in remote sensing and medical fields, improving model accuracy, and reducing data requirements and annotation costs.

[0049] The proposed method has been experimentally verified on a remote sensing image building extraction dataset and its performance has been compared with the existing mainstream self-supervised learning methods. The experimental results show that the method performs well in a variety of downstream tasks, effectively reducing the dependence on true value label data, and proving the effectiveness of the fusion contrast learning and feature mask modeling strategy in improving the learning effect of remote sensing image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is the overall flow chart of the image self-supervised learning network that integrates contrastive learning and feature mask modeling.

[0051] Figure 2 It is a schematic diagram of image preprocessing of the present invention.

[0052] Figure 3 It is a schematic diagram of the feature mask modeling branch (MFM) decoder structure.

[0053] Figure 4 This is a schematic diagram of the downstream task - semantic segmentation network.

[0054] Figure 5 This is a comparison chart of the downstream task-semantic segmentation visualization results. DETAILED DESCRIPTION

[0055] The technical solution of the present invention is further specifically described below by taking remote sensing images as an example and combining with the accompanying drawings.

[0056] like Figure 1 As shown, the present invention provides an image self-supervised learning method integrating contrast learning and feature mask modeling, comprising the following steps:

[0057] Step 1: Construct a high-resolution remote sensing image self-supervised learning network that integrates contrastive learning and feature mask modeling. The network mainly consists of two branches: one is the contrastive learning branch based on the SimCLR architecture (referred to as the CLR branch), and the other is the masked autoencoder branch based on the mask feature model (referred to as the MFM branch). The entire network includes an image preprocessing module, an image feature extraction module, and a shared weight encoder. Through the two branches of CLR and MFM, and combined with the loss function specially designed for these two branches, self-supervised training is performed to finally obtain a pre-trained model that can fully extract image features.

[0058] Step 2: Verify the effect of high-level image features. A semantic segmentation network is constructed based on the high-resolution remote sensing image self-supervised learning network that integrates contrast learning and feature mask modeling. The backbone network uses the parameters obtained by self-supervised learning in step 1, and a decoder is added on this basis. In order to verify the effect of self-supervised learning, only the decoder parameters are fine-tuned during the training process.

[0059] The specific implementation of step 1 includes the following sub-steps:

[0060] Step 1.1, image preprocessing. The input of image preprocessing is the original image, and the output is the image after data enhancement. The corresponding data set is constructed according to the specific requirements of the two branches of mask autoencoder and contrastive learning. Among them, the mask autoencoder only needs to process a single image, while the contrastive learning branch needs to convert the original image into different preprocessed images as input to calculate the positive and negative sample losses. Therefore, image preprocessing involves two processes: data enhancement and positive and negative sample production. In terms of image data enhancement, three methods, color jittering, random cropping, and random flipping, are selected. In terms of positive and negative sample production, for each batch of input images (the number is set to N), the data enhancement method is used to obtain two sets of new image data. Then, one group of images is used as the benchmark, and all the images of the other group are teamed to construct positive and negative samples. Each image has one positive sample and N-1 negative samples. In order to keep the positive and negative sample ratio the same, only one input comparison branch is randomly selected from the N-1 negative samples.

[0061] In this patented method, contrastive learning can be regarded as an auxiliary task of the MFM branch. By calculating the loss of positive and negative samples of remote sensing images, the feature extraction capability of model parameters is enhanced, which is expected to achieve better results in downstream tasks.

[0062] Step 1.2, convolutional feature extraction. The input of the convolutional feature extraction module is the image after data enhancement, and the output is the multi-scale high-order features extracted from the original image. The convolutional feature extraction module in the present invention adopts the classic Resnet50 network to extract high-dimensional, low-size deep features from the input shallow features (pixels, texture information). The convolutional feature extraction part includes a convolution layer with a convolution kernel of 7×7 and the first three bottlenecks of Resnet50 in sequence. Bottleneck is the basic structural block of Resnet50. Each structural block contains three convolutional structures. The convolution kernel size used in the first convolution is 1×1, and the convolution step size is 1, followed by the BN layer and the Relu layer; the second convolution layer has the same structure as the first, except that the convolution kernel size is 3×3; the third convolution layer structure is also the same as the first structure, and the convolution kernel size is 1×1. The above three convolution kernels plus the residual connection of resnet form a bottleneck. The convolutional network feature extraction part has three bottleneck units. After each bottleneck unit, the height and width of the output feature map become half of the unit input. After a 7×7 convolution layer and three bottleneck units, the side length of the output feature map of the convolutional network feature extraction part becomes 1 / 16 of the original image. The feature calculation process of the convolutional network feature extraction part is as follows:

[0063] F(x)=ResNet50(x)

[0064] Where x is the input original image, ResNet50 is the convolution operation based on ResNet50, and F(x) is the extracted image feature.

[0065] Step 1.3, feature map processing. Feature map processing includes two parts: flattening operation and random masking. It mainly processes the feature map output by the convolutional feature extraction module to make it suitable for the two task branches of contrastive learning and MFM. First, the output feature of the convolutional feature extraction module is recorded as f. For the contrastive learning branch, the 14×14 two-dimensional feature map is flattened into a one-dimensional sequence (length 196) to obtain a new one-dimensional feature f1. For the MFM branch, the 14×14 two-dimensional feature map is first masked, and then the unmasked part is flattened into a one-dimensional sequence to obtain a new feature f2. The masking operation divides the 14×14 feature map into 196 blocks, and generates 196×mask_ratio non-repeating indexes according to the randomly generated mask ratio mask_ratio (between 25% and 80%), and masks the corresponding feature map blocks according to the index. After the feature map is masked, the unmasked blocks are flattened into a sequence of length 196-(196×mask_ratio) and dimension 768.

[0066] Step 1.4, shared weight encoding. The shared weight encoder is responsible for further encoding the two sequences f1 and f2 output by the feature processing module, and obtaining new features with global context information of the entire image through the excellent self-attention mechanism of Transformer. The shared weight strategy is to reduce model parameters and training difficulty. In terms of network structure, the shared weight encoder consists of 12 stacked Transformer blocks. Each Transformer unit consists of two LayerNorm layers, a multi-head attention mechanism, and an MLP layer, and the input and output feature sizes of each Transformer unit are the same. The feature encoding process of the Transformer feature encoding part is to first process the sequences f1 and f2 output by the previous part, add position encoding to the feature vectors in the sequence, and record the positions of these feature vectors in the sequence. After that, the two sequences with position encoding added are respectively input into a series of Transformer units in series to obtain a deep encoding sequence. Since the Transformer unit does not change the input and output, the output results of the two sequences are sequences with lengths of 196 and 196-(196×mask_ratio) and dimensions of 768. The specific formula of the multi-head attention mechanism here is:

[0067]

[0068] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0069] head i =Attention(QW iQ ,KW i K ,VW i V )

[0070] Among them, Q represents the input at the current moment, K represents the information in memory, V represents the information to be paid attention to, and d k represents the key of each head and the dimension of the query vector. The softmax function is a function that converts multiple values ​​into probability distributions. Concat is a matrix concatenation operation. i Q , W i K , W i V is a transformation matrix that projects the vector into a different subspace.

[0071] The calculation process of each Transformer unit is as follows:

[0072] z=MLP(norm(MultiHead(norm(f))))

[0073] Where f represents the input feature sequence (i.e. f1 or f2), z represents the output feature sequence, and the two have the same size. MLP is a neural network composed of multiple fully connected layers, used for nonlinear transformation, norm is standardization, and MultiHead is a multi-head attention network.

[0074] Step 1.5, CLR decoder. From the image preprocessing in step 1.1, we know that the input of the CLR branch is a set of preprocessed image pairs. Assume that the features output by the shared weight encoder of the image pair are z t and z c， The CLR decoder first reshapes it from a 2D sequence of size HW×C into a 3D feature map of size H×W×C through a reshape operation, and adjusts the number of channels through a 1×1 convolution operation. Then, the feature map is subjected to operations such as average pooling (avgPool) and fully connected (fully connected, referred to as fc), and the global feature vector is mapped to a lower dimensional space (128 in this patent), and the features are further processed and compressed to extract higher-level semantic information, namely the feature vector (Ft, Fc), and the infoNCEloss loss function is used for similarity calculation. The CLR decoder process can be expressed by the following formula:

[0075] F = fc(AveragePooling(Conv 1×1 (reshape(z))))

[0076] Where z is a set of feature sequences output by the shared weight encoder, reshape is the operation of reshaping the feature map, AveragePooling represents the global average pooling of the feature map, and F is a set of feature vectors obtained after processing by the CLR decoder.

[0077] Step 1.6, MFM decoder. The input of the MFM decoder is the unmasked features processed by the shared weight encoder. The final predicted image is obtained through three serial operations: mask recovery, Transformer decoding block, and convolution upsampling.

[0078] The mask recovery part restores the original order of the sequence encoded in the previous deep layer, and replaces the masked vacant positions with learnable vectors. The decoded result of this learnable vector is the prediction result of the original mask position. The Transformer decoding block will initially process the padded sequence. This part consists of 6 series-connected Transformer units. The convolution upsampling decoding block reshapes the sequence output by the Transformer decoding block into a rectangle (reshape), and further uses 4 convolution upsampling units to decode and restore the original image. Each convolution upsampling unit consists of 1 3×3 convolution and 1 two-fold interpolation upsampling. Through three convolution upsampling, the feature map changes from 14×14 to the final original size of 224×224. The final image prediction head changes the number of channels to 3 through convolution and outputs the original color image. The MFM decoder can be expressed by the following formula:

[0079] u1=Upsample(Conv 3×3 (reshape(TR(z)))

[0080] u2=Upsample(Conv 3×3 (u1))

[0081] u3=Upsample(Conv 3×3 (u2))

[0082] I pre =Conv 1×1 (Upsample(Conv 3×3 (u3)))

[0083] Here, the input z is the sequence encoded in step 1.4, TR represents the Transformer unit, and the TR calculation process is shown in step 1.4. Upsample is the upsampling operation, and I pre is the output of the MFM decoder, i.e., the reconstructed image. When calculating the loss function, the loss between the reconstructed image P and the original image is calculated using the L1 loss function.

[0084] The specific implementation of step 2 includes the following sub-steps:

[0085] Step 2.1, semantic segmentation network construction. The input of the semantic segmentation network is the image of the building extraction dataset, and the output is the building segmentation image, which aims to verify the effect of the image self-supervised learning method that integrates contrast learning and feature mask modeling. The network structure retains the backbone network of convolutional feature extraction and shared encoder in step 1, and introduces a new decoding network and jump connection operation in the decoder part to form a "U"-shaped structure. Specifically, the decoding network part uses 4 convolutional upsampling units for decoding. Skip connections are added between the convolutional feature extraction and convolutional upsampling modules to fuse high-resolution features from different scales of the encoder, thereby reducing the loss of spatial information caused by the downsampling process, thereby improving the performance of semantic segmentation, a pixel-intensive prediction task.

[0086] Step 2.2, pre-trained model loading. After the semantic segmentation network is constructed, the weights of the convolutional network feature extraction and Transformer feature encoding parts use the parameters trained by the shared encoder in step 1. The parameters of this part of the network are frozen in the subsequent model training stage and do not participate in fine-tuning training.

[0087] Step 2.3, semantic segmentation dataset construction. The present invention uses a building extraction dataset to verify the invention content. The dataset consists of original remote sensing images and building pixel-level labels. Before training, it is divided into a training set, a validation set, and a test set according to a suitable ratio.

[0088] Step 2.4, decoder network training. Use the training data set as the training data of the network model, and iteratively train the network until the model converges. Select the model with the highest accuracy through the validation set, and calculate the accuracy of the model on the test set to objectively evaluate the quality of high-order image features obtained by the image self-supervised learning network that integrates contrast learning and feature mask modeling.

[0089] The implementation of the present invention is divided into two stages.

[0090] In the first stage, the image self-supervised network is constructed and trained by integrating contrastive learning and feature mask modeling. Figure 1 The self-supervised network of the present invention is demonstrated.

[0091] During the training process, the training samples are scaled to a size of 224×224, and data enhancement methods such as random cropping and scaling, random horizontal flipping, and color jitter are used to expand the training samples, as shown in the attached figure. Figure 2As shown. The processed data is used to iteratively train the image self-supervised network that integrates contrastive learning and feature mask modeling and calculate the loss function. The overall loss function of the present invention consists of two parts, CLR branch loss and MFM branch loss. The CLR branch uses the mainstream infoNCE Loss in contrastive learning, and the MFM branch uses the L1loss function.

[0092] The infoNCE Loss calculation formula is shown below.

[0093]

[0094] In the above formula, N is the batch size, z i and z j are the feature vectors of two different views of the same sample, which are positive samples of each other. sim(z i ,z j ) is z i and z j The similarity between them is calculated using the dot product. τ is a temperature parameter that controls the sharpness of the similarity distribution. [k≠i] is an indicator function, indicating that k is not equal to i. k Yes i Corresponding negative samples. Suppose we have a batch of data, each data point has two views (for example, two versions obtained by different data augmentation methods), for each sample z i , the positive sample is z i ′, negative samples are except z i The input can be expressed as: z1,z1′,z2,z2′,…,z N ,z N ′. For z1,z1′, the similarity is sim(z1,z1′). For z1 and z2, the similarity is sim(z1,z2). And so on, calculate the similarity between all feature vectors.

[0095] The L1Loss calculation formula is as follows:

[0096]

[0097] Among them, f(x i ) is the predicted image obtained by MFM decoding in step 1, x i is the original input image, y iis the original image as the true value. N′={1,1,1,…,1} is a one-dimensional matrix with all 1s, M is a mask matrix, which is a one-dimensional matrix with element values ​​of 0 or 1, where 0 represents a masked block and 1 represents an unmasked block. The elements of 1 in N'-M correspond to the masked blocks in the feature map, and sum represents the sum of the elements in N'-M (the number of masked blocks).

[0098] During back propagation, since the branch feature loss values ​​have orders of magnitude difference, the two branch losses are weighted when calculating the total loss to balance the impact of the two branches on the gradient update. The specific calculation is as follows.

[0099] loss=0.1×InfoNCELoss(z1′,z1″)+L1loss(f(x),y)

[0100] Where z1′ and z1″ are the feature vectors obtained by CLR decoding in step 1, x is the input original image, y is the original image as the true value, and f(x) is the predicted image obtained by MFM decoding in step 1.

[0101] After training is completed, only the network parameters of the convolutional feature extraction module and the shared encoder of the pre-trained model are saved, and these parameters are loaded in the second-stage downstream tasks.

[0102] In the second stage, the image self-supervised network quality verification integrating contrastive learning and feature mask modeling is performed. According to step 2.1, a semantic segmentation network is constructed. Figure 4 The self-supervised network of the present invention is demonstrated. After that, the pre-trained model of stage one is loaded, and the convolutional feature extraction and shared encoder network parameters are fixed. During training, the fixed parameters are not updated, and only the decoder parameters are updated. In the implementation of this patent, we conducted comparative experiments on the WHU and Massachusetts building extraction datasets to verify the effect of high-order features generated by the image self-supervised network that integrates contrastive learning and feature mask modeling. Semantic segmentation samples of buildings with different training amounts were randomly selected at a ratio of 5%, 10%, 20%, 40%, 60% and 100% to detect the intersection over union (IoU) indicator of the algorithm of this patent on small sample downstream tasks as statistical accuracy. Table 1 compares the semantic segmentation results of the method proposed in this patent with the current mainstream self-supervised learning algorithms BEiT, SimMIM, MAE, and CAE on WHU datasets with different sample sizes. Appendix Figure 5 The semantic segmentation visualization results are shown.

[0103] Table 1 Comparison of semantic segmentation accuracy of the proposed method and other methods on the WHU dataset

[0104]

[0105] It can be seen from Table 1 that the accuracy of the method of the present invention is better than that of several other fully supervised and self-supervised methods. The image self-supervised learning method proposed in the present invention combines contrastive learning and feature mask modeling, and performs better in the remote sensing image semantic segmentation task.

[0106] Table 2 shows the performance of the patented method on the Massachusetts dataset, which is characterized by a small amount of data.

[0107] Table 2 Comparison of semantic segmentation accuracy of the proposed method and other methods on the Massachusetts dataset

[0108]

[0109]

[0110] It can be seen from the data in Table 2 that the patented method has achieved significant performance improvement on this data set. Based on the data in Tables 1 and 2, we can make the following inferences: First, this method is easier to converge when trained on a data set with a smaller amount of data, such as Massachusetts, and enables the model to quickly obtain better feature extraction capabilities when pre-training (performing pre-tasks) on a data set with a smaller amount of data, and some parameters of the encoder have excellent generalization performance in downstream tasks. Second, the image clarity of the Massachusetts data set is low, and the similarity between images is high. By comparing the loss calculation and back propagation of the CLR branch of the learning CLR, the model's ability to understand the distribution of targets such as buildings is improved, and the ability to extract global context information is enhanced, so the generalization ability of downstream tasks is greatly improved. For small data sets with a small number of samples, compared with existing methods, the method of the present invention has better robustness and can obtain more accurate semantic segmentation results. It is worth noting that in actual engineering applications, due to the limitation of manual annotation costs, small data sets with small annotations are the norm. Therefore, the method of the present invention has high engineering practical value.

[0111] On the other hand, an embodiment of the present invention further provides an image self-supervised learning system integrating contrastive learning and feature mask modeling, comprising:

[0112] one or more processors;

[0113] A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement an image self-supervised learning method that integrates contrast learning and feature mask modeling as described in the above scheme.

[0114] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A self-supervised image learning method integrating contrastive learning and feature mask modeling, characterized in that: The steps include: Step 1, construct and train a high-resolution image self-supervised learning network that integrates contrastive learning and feature mask modeling. The self-supervised learning network includes two branches: one is a contrastive learning branch based on the SimCLR architecture, namely the CLR branch, and the other is a masked autoencoder branch based on the mask feature model, namely the MFM branch. The specific processing process is: first, the input image is preprocessed, and the image after data enhancement is output, and the corresponding data set is constructed according to the masked autoencoder and contrastive learning branches. Then, multi-scale high-order features are extracted from the image of the data set through the convolutional feature extraction module, and the high-order feature map is flattened and sent to the CLR branch. The high-order feature map is randomly masked and sent to the MFM branch. The two sequences obtained by the flattening operation and the random masking are further encoded using a shared weight encoder to obtain new features with global context information of the entire image in the CLR branch and the MFM branch, respectively. Finally, a set of feature vectors are obtained through a CLR decoder, and the final predicted image is obtained through an MFM decoder. During the training process, different loss functions are used to train the CLR branch and the MFM branch; Step 2: construct a semantic segmentation network based on the self-supervised learning network. The backbone network of the semantic segmentation network includes the convolutional feature extraction module and the shared encoder in the self-supervised learning network in step 1. The parameters obtained by training the self-supervised learning network are used, and a decoding network is added on this basis. Image segmentation can be achieved using the constructed semantic segmentation network.

2. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: Preprocessing includes data enhancement and positive and negative sample production. Image data enhancement includes three methods: color jittering, random cropping, and random flipping. The positive and negative sample production is as follows: for each batch of input images, the number is set to N, and two new sets of image data are obtained by using data enhancement methods. Then, one set of images is used as a benchmark and teamed with all images in the other set to construct positive and negative samples. Each image has one positive sample and N-1 negative samples.

3. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: The convolutional feature extraction module includes a convolutional layer with a convolution kernel of 7×7 and the first three bottleneck units of Resnet50 in sequence. The bottleneck unit is the basic structural block of Resnet50. Each structural block contains three convolutional structures. The convolution kernel size used in the first convolutional structure is 1×1, and the convolution step size is 1, followed by a BN layer and a Relu layer; the second convolutional structure is the same as the first structure, except that the convolution kernel size is 3×3; the third convolutional structure is also the same as the first convolution structure, with a convolution kernel size of 1×1; the above three convolutional structures plus residual connections form a bottleneck unit; the convolutional feature extraction module has a total of three bottleneck units. After each bottleneck unit, the height and width of the output feature map become half of the unit input. After a 7×7 convolutional layer and three bottleneck units, the side length of the feature map output by the convolutional feature extraction module becomes 1 / 16 of the original image.

4. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: The output feature of the convolutional feature extraction module is denoted as f. For the contrastive learning branch, the two-dimensional feature map is flattened into a one-dimensional sequence to obtain a new one-dimensional feature f1. For the MFM branch, the two-dimensional feature map is first masked, and then the unmasked part is flattened into a one-dimensional sequence to obtain a new feature f2.

5. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: The shared weight encoder consists of several stacked Transformer units, each of which consists of two LayerNorm layers, a multi-head attention mechanism, and an MLP layer, and the feature size of each Transformer unit input and output is the same; the feature encoding process of the Transformer feature encoding part is to first process the sequences f1 and f2 output by the previous part, add position encoding to the feature vectors in the sequence, and record the positions of these feature vectors in the sequence. After that, the two sequences with position encoding are respectively input into a series of series-connected Transformer units to obtain a deep encoding sequence; the specific formula of the multi-head attention mechanism is: MultiHead(Q,K,V)=Concat(head1,...,head h ) head i =Attention(QW i Q ,KW i K ,VW i V ) Among them, Q represents the input at the current moment, K represents the information in memory, V represents the information to be paid attention to, and d k represents the key of each head and the dimension of the query vector. The softmax function is a function that converts multiple values ​​into probability distributions. Concat is a matrix concatenation operation. i Q , W i K , W i V is the transformation matrix, which projects the vector into a different subspace; The calculation process of each Transformer unit is as follows: z=MLP(norm(MultiHead(norm(f)))) Where f represents the input feature sequence, i.e. f1 or f2, and z represents the output feature sequence; MLP is a neural network composed of multiple fully connected layers, which is used for nonlinear transformation, norm is standardization processing, and MultiHead is a multi-head attention network.

6. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: The input of the CLR branch is a set of image pairs, assuming that the features output by the shared weight encoder of the image pair are z t and z c The CLR decoder first reshapes it from a 2D sequence of size HW×C into a 3D feature map of size H×W×C through a reshape operation, and adjusts the number of channels through a 1×1 convolution operation. Then, the feature map is averaged and pooled avgPool and fully connected fc operations are performed to map the global feature vector to a lower dimensional space. The features are further processed and compressed to extract higher-level semantic information, namely, the feature vector (F t , F c ), where the CLR decoder process is expressed by the following formula: F=fc(AveragePooling(Conv 1×1 (reshape(z)))) Where z is a set of feature sequences output by the shared weight encoder, reshape is the operation of reshaping the feature map, AveragePooling represents the global average pooling of the feature map, and F is a set of feature vectors obtained after processing by the CLR decoder.

7. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: The input of the MFM decoder is the unmasked features processed by the shared weight encoder. The final predicted image is obtained through three serial operations: mask recovery, Transformer decoding block, and convolution upsampling. The mask recovery part restores the original order of the sequence of the previous deep coding, and replaces the masked vacant positions with learnable vectors. The decoded result of this learnable vector is the prediction result of the original mask position. The Transformer decoding block will initially process the padded sequence. This part consists of 6 series-connected Transformer units. Convolution upsampling reshapes the sequence output by the Transformer decoding block into a rectangle, and further uses 4 convolution upsampling units to decode it to restore the original image. Each convolution upsampling unit consists of 1 3×3 convolution and 1 two-fold interpolation upsampling. After three convolution upsampling, it is restored to the original size. The final image prediction head converts the number of channels to 3 through convolution and outputs the original color image. The MFM decoder is represented by the following formula: u1=Upsample(Conv 3×3 (reshape(TR(z))) u2=Upsample(Conv 3×3 (u1)) u3=Upsample(Conv 3×3 (u2)) I pre =Conv 1×1 (Upsample(Conv 3×3 (u3))) Here the input z is the sequence encoded by the shared weight encoder, TR represents the Transformer unit, Conv 3×3 is a 3×3 convolution, Upsample is an upsampling operation, I pre It is the output of the MFM decoder, i.e. the reconstructed image.

8. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: The CLR branch uses the infoNCE Loss loss function, and the MFM branch uses the L1Loss loss function; The infoNCE Loss calculation formula is as follows: In the above formula, N is the batch size, z i and z j are the feature vectors of two different views of the same sample, which are positive samples of each other; sim(z i ,z j ) is z i and z j The similarity between them is calculated using the dot product; τ is the temperature parameter used to control the sharpness of the similarity distribution; 1 [k≠i] is an indicator function, indicating that k is not equal to i; z k Yes i The corresponding negative samples; The L1Loss calculation formula is as follows: Among them, f(x i ) is the predicted image obtained by MFM decoding, x i is the original input image, y i is the original image as the true value; N′={1,1,1,…,1} is a one-dimensional matrix with all 1s, M is a mask matrix, which is a one-dimensional matrix with element values ​​of 0 or 1, where 0 represents the masked block and 1 represents the unmasked block; the elements of N'-M that are 1 correspond to the masked blocks in the feature map, and sum represents the sum of the elements in N'-M, that is, the number of masked blocks; During back propagation, the above two loss functions are weighted, and the specific calculation is as follows: loss=0.1×InfoNCELoss(z1′,z1″)+L1loss(f(x),y) Where z1' and z1" are the feature vectors decoded by the CLR branch, x is the input original image, y is the original image as the true value, and f(x) is the predicted image decoded by the MFM branch.

9. The image self-supervised learning method integrating contrastive learning and feature mask modeling as claimed in claim 1, characterized in that: The decoding network in step 2 uses 4 convolutional upsampling units for decoding, and adds a skip connection between the convolutional feature extraction module and the convolutional upsampling unit to fuse high-resolution features from different scales of the encoder.

10. An image self-supervised learning system integrating contrastive learning and feature mask modeling, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement an image self-supervised learning method that integrates contrast learning and feature mask modeling as described in any one of claims 1 to 9.

Images