Semi-supervised remote sensing image classification method and device based on comparative learning

Abstract

The invention provides a semi-supervised remote sensing image classification method and device based on comparative learning. The method comprises the following steps: constructing and importing image training data; performing self-supervised training on the classification model by using image training data; performing supervised training on the classification model; and acquiring to-be-classified image data, and determining a classification result of the to-be-classified image data by using the configured classification model. The method can be used for realizing the utilization of the label-free data and reducing the data labeling cost; and the performance on unbalanced data and small samples is improved, and the robustness of image classification is improved.

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to a semi-supervised remote sensing image classification method based on contrastive learning. Background Art

[0002] Remote sensing images are images of the earth's surface taken by remote sensing satellites in orbit. Remote sensing images often contain rivers, roads, buildings, etc. Multiple fields rely on these images as information sources, which can be used for analysis and statistics. However, since a large amount of information implicit in the original image is disordered, it is not suitable for specific research tasks. Therefore, in order to facilitate further use, the image must be classified, which is the main purpose of image classification. The solution to this problem is similar to the general image classification algorithm, but it is different due to the characteristics of remote sensing images. Computers view images as data matrices with different dimensions, so classification algorithms identify images by features under these dimensions. In order to classify images, classification methods first extract features of each image and then learn the correlation between features and labels. Classification methods classify images by establishing connections between features and labels and predicting the most likely category of input data. Common methods for image classification models are feature-based adaptation and classifier-based adaptation.

[0003] Basic classification methods do not perform well on semi-supervised or self-supervised tasks, so making use of unlabeled data requires improvements to the high-level architecture of neural networks. Contrastive learning is one of the most widely used self-supervised methods in classification tasks. This method considers data pairs with similar features as positive pairs and data pairs with dissimilar features as negative pairs. The basic idea of ​​contrastive learning is to maximize the similarity of positive pairs and minimize the similarity of negative pairs. Through this optimization, neural networks can learn to distinguish the features of each class without data labels.

[0004] However, the current deep learning remote sensing image classification methods still have certain defects. Most remote sensing image classification methods have high requirements for supervision information and cannot use unlabeled data, which increases the labor cost of data annotation. At the same time, the classification accuracy of existing image classification methods is also significantly affected by the quality of the dataset. The imbalance between classes and the imbalance within classes of the training dataset will cause the algorithm to perform poorly on difficult classification tasks. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to reduce the dependence on supervisory information and the cost of manual annotation, and improve the classification effect on unbalanced data sets with poor robustness to changes in data quality; in view of this, the present invention provides a semi-supervised remote sensing image classification method and device based on contrastive learning.

[0006] The technical solution adopted by the present invention is a semi-supervised remote sensing image classification method based on contrastive learning, comprising:

[0007] Step S1, constructing and importing image training data;

[0008] Step S2, using the image training data to perform self-supervised training on the classification model;

[0009] Step S3, performing supervised training on the classification model;

[0010] Step S4, obtaining the image data to be classified, and using the configured classification model to determine the classification result of the image data to be classified.

[0011] In one embodiment, step S2 includes:

[0012] Using the first data enhancement technology and the second data enhancement technology respectively on the first image training data to obtain corresponding positive samples, and obtaining negative samples from the second image training data after conversion;

[0013] Extracting feature vectors from the processed positive samples using an encoder neural network, that is, when the image training data is N, the feature vectors are 2N, and each feature vector can form a pair of positive sample pairs with a vector derived from the same input, and form 2N-2 pairs of negative sample pairs with vectors derived from other inputs, for a total of 2N-1 pairs of sample pairs;

[0014] Determining a global contrastive learning loss by the cosine similarity of each pair of samples;

[0015] By constructing a projection head, an alignment value of a pair of input local features is determined;

[0016] Based on the alignment value, local feature similarity and local contrastive learning loss are determined.

[0017] In one embodiment, step S2 further comprises:

[0018] For every two of the image training data, generating corresponding interpolation samples;

[0019] An interpolation loss is constructed by minimizing the distance between the interpolation sample and the corresponding two image training data, so as to make the feature representation of the interpolation sample close to the linear combination of the corresponding two image training data.

[0020] In one embodiment, step S2 further comprises:

[0021] MLP with ReLU was used as the classifier, which was optimized using cross entropy loss.

[0022] In one embodiment, step S2 further comprises:

[0023] The classification model achieves self-supervised learning by minimizing the self-supervised loss;

[0024] Among them, the self-supervision loss is specifically:

[0025]

[0026] α is the parameter for balancing global features, β is the parameter for balancing local features, is the interpolation loss.

[0027] In one embodiment, step S3 is specifically implemented by the following formula:

[0028]

[0029] L s To monitor the loss, L C is the cross entropy loss, is the supervised interpolation loss.

[0030] Another aspect of the present invention further provides a semi-supervised remote sensing image classification device based on contrastive learning, comprising:

[0031] A construction unit configured to construct and import image training data;

[0032] A self-supervision unit configured to perform self-supervision training on a classification model using the image training data;

[0033] A supervision unit, configured to perform supervised training on the classification model;

[0034] The classification unit is configured to obtain the image data to be classified, and determine the classification result of the image data to be classified using the configured classification model.

[0035] Another aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the semi-supervised remote sensing image classification method based on contrastive learning as described in any one of the above items.

[0036] Another aspect of the present invention further provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the semi-supervised remote sensing image classification method based on contrastive learning as described in any one of the above items are implemented.

[0037] Compared with the prior art, the present invention has at least the following advantages:

[0038] The present invention can be used to realize the utilization of unlabeled data and reduce the cost of data annotation; as well as to improve the performance on unbalanced data and small samples, thereby improving the robustness of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of a semi-supervised remote sensing image classification method based on contrastive learning according to an embodiment of the present invention;

[0040] Figure 2 A logical diagram of a classification model according to an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a user usage flow according to an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the composition of a semi-supervised remote sensing image classification device based on contrastive learning according to an embodiment of the present invention;

[0043] Figure 5 The figure is a schematic diagram of the composition of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the present invention is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0045] Unless otherwise defined, all terms (including technical terms and scientific terms) used in this article have the same meaning as commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that terms (such as terms defined in commonly used dictionaries) should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in this article.

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] The present invention provides a semi-supervised remote sensing image classification method based on contrastive learning, such as Figure 1 As shown, including:

[0048] Step S1, constructing and importing image training data;

[0049] Step S2, using the image training data to perform self-supervised training on the classification model;

[0050] Step S3, performing supervised training on the classification model;

[0051] Step S4, obtaining the image data to be classified, and using the configured classification model to determine the classification result of the image data to be classified.

[0052] refer to Figure 2 as well as Figure 3 , the method provided in this embodiment will be described in detail below.

[0053] Specifically, the present invention proposes a semi-supervised deep learning method based on a contrastive learning model, wherein the neural network model of the method consists of three parts: a contrastive learning module, an interpolation hybrid module, and a linear classifier. The training process generally has two stages:

[0054] In the first stage, the model is trained through self-supervision through contrastive learning module and hybrid module to obtain better feature expression;

[0055] In the second stage, the linear classifier and the hybrid module cooperate to train the labeled data in order to achieve better classification accuracy in the second stage.

[0056] Specifically, the contrastive learning part above uses unlabeled data to learn global and local features, the interpolation hybrid module participates in both unsupervised and supervised learning processes, and the classifier only participates in the supervised learning process. Figure 2 As shown:

[0057] The contrastive learning module assists the neural network in learning the global and local features of the input image. Data augmentation techniques are used on the input data to create fake images for negative pairs. In one training process, two different data augmentation techniques q are applied to the input image x. 1 and q 2 , get two enhanced data and As a positive sample, and transformed from another input y and After data enhancement, the encoder neural network f(·) is used to extract the negative samples from the processed data. The features extracted are as follows:

[0058]

[0059] where h i is the output of the network, s i is the intermediate state output of the feature extraction process, so s i Contains input data The projection head g(·) transforms h i becomes the global eigenvector z i , as shown below:

[0060] zi =θ(h i ) (2)

[0061] After the above process, an input image x is transformed into two feature vectors z i and z j Assuming that the amount of input data for each training process is N, a total of 2N global feature vectors are obtained. Therefore, each feature vector can form a positive sample pair with a vector from the same input, and form 2N-2 pairs of negative sample pairs with vectors from other inputs, for a total of 2N-1 pairs. Therefore, each batch has 2N*(2N-2) pairs of vectors. By calculating the cosine similarity of each pair, the global contrastive learning loss L G The definitions are as follows:

[0062]

[0063] Where 1[k≠i] is an indicator function that outputs 1 when k≠i and outputs 0 when k=i. τ represents the temperature parameter that affects the learning rate. G As decreases, the similarity between positive sample pairs increases, and the similarity between negative sample pairs decreases. The encoder network thus learns the characteristics of each category through unlabeled data.

[0064] In addition to the global loss, the present invention also introduces a local contrast loss function in the module for fine-grained feature learning. For this purpose, three projection heads p are constructed. v , p q and p k The intermediate local feature s calculated in formula (1) i As input, these headers output a vector of values ​​v i , query vector q i and key vector k i For a pair of input local features s i and j , alignment value v i|j and v j|i The definitions are as follows:

[0065]

[0066] Where d represents the dimension of the feature. We get v i|j and v j|i After that, the local feature similarity sim is calculated by each position (a, b) in the feature size HW l (s i ,s j ), as shown below:

[0067]

[0068] Local contrastive learning loss L based on local feature similarity L The definitions are as follows:

[0069]

[0070] In the formula, τ 2 is another temperature parameter of the local contrast loss function, 1[k≠i] is an indicator function, and its usage is similar to formula (2). By optimizing the loss function L L , the encoder network can better represent the local features of remote sensing images.

[0071] In addition to contrastive learning, this embodiment also introduces an interpolation learning module into the model to obtain better feature extraction and faster convergence speed. This module consists of two parts: sample generation and feature shrinkage. For each input image, the interpolation sample is generated as follows:

[0072]

[0073] where x i and x j are two input images, For x i and x j The generated interpolated samples, λ is a linearly associated parameter. Compared with the original input, it is more difficult for the neural network to learn the feature representation of the interpolated samples.

[0074] By minimizing and x i ,x j The distance between them is used to reduce the feature distribution area of ​​each category in the feature space. The Euclidean distance function D(·,·) is used to measure the distance in the feature space. The interpolation loss L I The purpose is to make The feature representation is closer to x i With x j In this process, the area of ​​the input images of the same category in the feature representation space is reduced, thereby obtaining better feature extraction performance and convergence speed. I The definitions are as follows:

[0075]

[0076] The purpose of σ is to prevent random noise caused by overfitting.

[0077] In addition to the encoder network and the learning module, a classifier suitable for remote sensing image classification tasks is also required. The present invention uses an MLP (multi-layer perceptron) with ReLU containing two linear layers as a classifier, which takes the feature represented by f(·) as the input of the predicted class and outputs the probability that the sample belongs to each category. The cross entropy loss L is introducedc To optimize the accuracy of the classifier:

[0078]

[0079] Where C(·) refers to the classifier, x i is the input data, y i is x i The correct category.

[0080] As mentioned above, the model training process has two stages: self-supervision stage and supervision stage. In the first stage, the model minimizes the self-supervision loss L ss To achieve self-supervised learning, L ss The definition of is as follows:

[0081]

[0082] Among them, α and β are parameters that balance global features and local features. Represents the interpolation loss in the self-supervised stage to distinguish it from the supervised interpolation loss

[0083] After the self-supervised training is completed, the model is supervised using the labeled dataset. In this stage, we optimize the supervised loss L s To improve the performance of the model:

[0084]

[0085] Therefore, the use process of the present invention can be divided into training and actual use. The user first configures the model to training mode and imports the training data set for training. After the training is completed, the trained model parameters are loaded and the data is input to obtain the classification results. The specific process is as follows Figure 3 shown.

[0086] Compared with the prior art, this embodiment has at least the following effects:

[0087] 1) The present invention can be used to realize the utilization of unlabeled data and reduce the cost of data annotation; as well as improve the performance on unbalanced data and small samples, and improve the robustness of image classification;

[0088] 2) The contrastive learning module of the neural network model proposed in the present invention can learn the category features of unlabeled data and achieve a clustering-like effect on the encoder network in the unsupervised training stage. In this process, the encoder network can map input data of different categories to different regions in the feature space, that is, pre-classification of input data in an unsupervised information environment is realized. However, due to the lack of category label data, the class clusters in the feature space cannot be matched with the actual categories. Subsequently, label data is introduced in the supervised learning stage to complete the complete mapping from input data to output categories. In this process, the present invention can utilize a large amount of unlabeled data, reduce data annotation costs, and expand the amount of training samples. At the same time, the method combining global and local contrastive learning also enhances the classification accuracy in an environment with unbalanced data within a class.

[0089] 3) The interpolation mixing module of the present invention is committed to maintaining the same linear relationship between the interpolated feature vector and the original value. In this process, the feature space gradually becomes flat, and the distribution of each category in the feature space gradually becomes concentrated, which accelerates the convergence speed of the model, improves the performance on unbalanced data between classes and small samples, and improves the robustness of image classification.

[0090] The second embodiment of the present invention corresponds to the first embodiment, and can be referred to Figure 4 This embodiment introduces a semi-supervised remote sensing image classification device based on contrastive learning, including the following components:

[0091] A construction unit configured to construct and import image training data;

[0092] A self-supervision unit configured to perform self-supervision training on a classification model using the image training data;

[0093] A supervision unit, configured to perform supervised training on the classification model;

[0094] The classification unit is configured to obtain the image data to be classified, and determine the classification result of the image data to be classified using the configured classification model.

[0095] In this embodiment, the self-supervision unit is further configured as follows:

[0096] Using the first data enhancement technology and the second data enhancement technology respectively on the first image training data to obtain corresponding positive samples, and obtaining negative samples from the second image training data after conversion;

[0097] Extracting feature vectors from the processed positive samples using an encoder neural network, that is, when the image training data is N, the feature vectors are 2N, and each feature vector can form a pair of positive sample pairs with a vector derived from the same input, and form 2N-2 pairs of negative sample pairs with vectors derived from other inputs, for a total of 2N-1 pairs of sample pairs;

[0098] Determining a global contrastive learning loss by the cosine similarity of each pair of samples;

[0099] By constructing a projection head, an alignment value of a pair of input local features is determined;

[0100] Based on the alignment value, local feature similarity and local contrastive learning loss are determined.

[0101] In this embodiment, the self-supervision unit is further configured as follows:

[0102] For every two of the image training data, generating corresponding interpolation samples;

[0103] An interpolation loss is constructed by minimizing the distance between the interpolation sample and the corresponding two image training data, so as to make the feature representation of the interpolation sample close to the linear combination of the corresponding two image training data.

[0104] In this embodiment, the self-supervision unit is further configured as follows:

[0105] MLP with ReLU was used as the classifier, which was optimized using cross entropy loss.

[0106] In this embodiment, the self-supervision unit is further configured as follows:

[0107] The classification model achieves self-supervised learning by minimizing the self-supervised loss;

[0108] Among them, the self-supervision loss is specifically:

[0109]

[0110] α is the parameter for balancing global features, β is the parameter for balancing local features, is the interpolation loss.

[0111] In this embodiment, the supervision unit is further configured as follows:

[0112]

[0113] L s To monitor the loss, L C is the cross entropy loss, is the supervised interpolation loss.

[0114] A third embodiment of the present invention is an electronic device, such as Figure 5 As shown, it can be understood as a physical device, including a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the following operations are performed:

[0115] Step S1, constructing and importing image training data;

[0116] Step S2, using the image training data to perform self-supervised training on the classification model;

[0117] Step S3, performing supervised training on the classification model;

[0118] Step S4, obtaining the image data to be classified, and using the configured classification model to determine the classification result of the image data to be classified.

[0119] The fourth embodiment of the present invention, the process of the semi-supervised remote sensing image classification method based on contrastive learning in this embodiment is the same as that of the first, second or third embodiment, the difference is that, in engineering implementation, this embodiment can be implemented by means of software plus a necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the method of the present invention can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a device to execute the method described in the embodiment of the present invention.

[0120] Through the description of the specific implementation methods, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the predetermined purpose should be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention.

Claims

1. A semi-supervised remote sensing image classification method based on contrastive learning, characterized in that: include: Step S1, constructing and importing image training data; Step S2, using the image training data to perform self-supervised training on the classification model; Step S3, performing supervised training on the classification model; Step S4, obtaining the image data to be classified, and using the configured classification model to determine the classification result of the image data to be classified.

2. The semi-supervised remote sensing image classification method based on contrastive learning according to claim 1, characterized in that: The step S2 comprises: Using the first data enhancement technology and the second data enhancement technology respectively on the first image training data to obtain corresponding positive samples, and obtaining negative samples from the second image training data after conversion; Extracting feature vectors from the processed positive samples using an encoder neural network, that is, when the image training data is N, the feature vectors are 2N, and each feature vector can form a pair of positive sample pairs with a vector derived from the same input, and form 2N-2 pairs of negative sample pairs with vectors derived from other inputs, for a total of 2N-1 pairs of sample pairs; Determining a global contrastive learning loss by the cosine similarity of each pair of samples; By constructing a projection head, an alignment value of a pair of input local features is determined; Based on the alignment value, local feature similarity and local contrastive learning loss are determined.

3. The semi-supervised remote sensing image classification method based on contrastive learning according to claim 2, characterized in that: The step S2 further comprises: For every two of the image training data, generating corresponding interpolation samples; An interpolation loss is constructed by minimizing the distance between the interpolation sample and the corresponding two image training data, so as to make the feature representation of the interpolation sample close to the linear combination of the corresponding two image training data.

4. The semi-supervised remote sensing image classification method based on contrastive learning according to claim 3, characterized in that: The step S2 further comprises: MLP with ReLU was used as the classifier, which was optimized using cross entropy loss.

5. The semi-supervised remote sensing image classification method based on contrastive learning according to claim 4, characterized in that: The step S2 further comprises: The classification model achieves self-supervised learning by minimizing the self-supervised loss; Among them, the self-supervision loss is specifically: α is the parameter for balancing global features, β is the parameter for balancing local features, is the interpolation loss.

6. The semi-supervised remote sensing image classification method based on contrastive learning according to claim 5, characterized in that: The step S3 is specifically implemented by the following formula: L s To monitor the loss, L C is the cross entropy loss, is the supervised interpolation loss.

7. A semi-supervised remote sensing image classification device based on contrastive learning, characterized in that: include: A construction unit configured to construct and import image training data; A self-supervision unit configured to perform self-supervision training on a classification model using the image training data; A supervision unit, configured to perform supervised training on the classification model; The classification unit is configured to obtain the image data to be classified, and determine the classification result of the image data to be classified using the configured classification model.

8. An electronic device, characterized in that: The electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the semi-supervised remote sensing image classification method based on contrastive learning as described in any one of claims 1 to 6 are implemented.

9. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the semi-supervised remote sensing image classification method based on contrastive learning as claimed in any one of claims 1 to 6.

Images