A remote sensing image representation method based on contrast learning

Abstract

This invention provides a remote sensing image representation method based on contrastive learning, belonging to the field of remote sensing image processing. First, the remote sensing image is preprocessed by unifying its coordinates and cropping it into sub-images. Second, positive and negative samples are constructed, and similarity is calculated for each sub-image. Based on the similarity information, positive and negative sample pairs are constructed, and image enhancement is performed on the positive and negative samples. Then, a contrastive learning network is constructed, and the parameters of the contrastive learning network are updated by minimizing the loss function through an optimization algorithm, ultimately obtaining the contrastive learning network. Finally, untrained remote sensing images are used as input, and the feature extraction network in the contrastive learning network extracts representational features as the representation of the remote sensing image. This method can not only represent forests, water bodies, and desert features in remote sensing images differently, but also differentiate the representation of the same features with a long spatial distribution, and distinguish the representation of the same feature over a large imaging time span.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing, and more specifically relates to a remote sensing image representation method based on contrastive learning. Background Technology

[0002] Word embedding technology, as a language representation technique, has achieved great success in the field of natural language processing, playing a crucial role in tasks such as automatic question answering and automatic writing. Drawing inspiration from language representation, image representation methods have made significant progress, and effective image representation can greatly improve the performance of tasks such as image retrieval, image compression, and content recognition. However, supervised learning-based image representation methods require substantial manual labeling. Furthermore, images differ from language and text; the vocabulary in language and text is finite, while image content is infinite, making image content labeling more costly in terms of both manpower and time. Therefore, to save on labor costs and time, and to utilize large amounts of unlabeled image data to expand the application of image representation in remote sensing, there is an urgent need for an unsupervised remote sensing image representation method. Summary of the Invention

[0003] The purpose of this invention is to utilize a large amount of unlabeled image data, while saving labor and time costs, to expand the application of image representation in the field of remote sensing. A remote sensing image representation method based on contrastive learning is proposed. This method can not only represent forests, water bodies, and desert features in remote sensing images differently, but also differentiate the representation of the same features with a long spatial distribution, and further distinguish the representation of the same feature over a large imaging time span.

[0004] The technical solution adopted in this invention is as follows:

[0005] A remote sensing image representation method based on contrastive learning includes the following steps:

[0006] Step 1: Perform coordinate unification on the remote sensing images and crop the remote sensing images with a width exceeding the threshold to a remote sensing sub-image with a set width.

[0007] Step 2: Calculate the image similarity of the remote sensing sub-images obtained in Step 1, and construct positive and negative sample pairs based on the imaging time, imaging location and image similarity information of the remote sensing sub-images. Then, perform image enhancement on the positive and negative samples.

[0008] Step 3: Construct a contrastive learning network. The input is a pair of positive and negative samples. In the contrastive learning network, a feature extraction network is first used to extract the representation features of the positive and negative sample pairs. Then, the representation features are nonlinearly transformed to obtain the loss function to calculate the features. The loss is calculated based on the loss function. The parameters of the contrastive learning network are updated by minimizing the loss function through an optimization algorithm, and finally the contrastive learning network is obtained.

[0009] Step 4: Take the newly acquired remote sensing image that was not used in the training as input, and extract the representation features through the feature extraction network in the contrastive learning network, which will be used as the representation of the remote sensing image.

[0010] Furthermore, the coordinate unification operation in step 1 involves converting the coordinate system of the remote sensing image to the WGS84 coordinate system and cropping it in units of latitude and longitude.

[0011] Furthermore, the image similarity calculation in step 2 includes histogram feature extraction and histogram feature Euclidean distance calculation. The ratio of positive to negative samples in the positive-negative sample pair is 1:1. The specific process of constructing the positive-negative sample pair is as follows:

[0012] right Similarity calculation was performed on the remote sensing sub-graphs to obtain... The system uses several similarity metrics to normalize the similarity metrics. Two remote sensing sub-images with similarity metrics less than a set value are selected as positive and negative sample candidate pairs. Among the positive and negative sample candidate pairs, two remote sensing sub-images with an imaging time difference greater than a set duration or an imaging location distance greater than a set kilometer are selected as positive and negative sample pairs. Data augmentation operations are then performed on the remote sensing sub-images in the positive and negative sample pairs, including random noise addition, random cropping, random rotation, and random cloud / fog addition.

[0013] Furthermore, the input to the contrastive learning network described in step 3 is the data-enhanced positive and negative sample pairs. The positive and negative samples are respectively input into two weight-sharing feature extraction networks. The feature extraction network adopts the CSPDarknet53 network, the feature dimension is set to 2048, and a two-layer fully connected network is used as a non-linear transformation layer to perform non-linear transformation on the feature representation. The loss calculation feature is set to 256, and the loss function is designed as follows:

[0014]

[0015] in To compare the loss function of the learning network, To match the loss, For mismatch loss, To match the loss coefficients, The mismatch loss coefficient is defined as follows: Matching loss includes the degree of matching between positive samples and between negative samples; mismatch loss includes the degree of matching between positive samples and between negative samples and between positive samples. and The value is between 0 and 1, and + =1.

[0016] Furthermore, the remote sensing images of the new images that were not trained in step 4 are input into the feature extraction network after being cropped from the remote sensing sub-images, and the representation feature dimension of the remote sensing images is consistent with the representation feature dimension described in step 3.

[0017] The advantages of this invention over the prior art are:

[0018] (1) The self-supervised contrastive learning method is adopted, which saves the manual and time costs of data annotation in the image representation process;

[0019] (2) By constructing positive and negative samples, the problem of severe imbalance between positive and negative samples during the comparative learning process is avoided;

[0020] (3) By introducing spatial and temporal dimension information, it is possible to differentiate remote sensing images of different land features, different spatial distributions, and different imaging times. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the overall technical process of the method proposed in this invention.

[0022] Figure 2 This is a structural diagram of the contrastive learning network proposed in this invention. Detailed Implementation

[0023] This invention proposes a remote sensing image representation method based on contrastive learning, the specific flowchart of which is shown below. Figure 1 As shown.

[0024] The specific embodiments and basic principles of the present invention will be further explained below with reference to the accompanying drawings.

[0025] Step 1, remote sensing image preprocessing, performs coordinate unification operation on the remote sensing image. Coordinate unification operation refers to converting the coordinate system of the remote sensing image to the WGS84 coordinate system and cropping the remote sensing image with a large swath width to a small swath width remote sensing sub-map, with the cropping unit being latitude and longitude in minutes.

[0026] Step 2, positive and negative sample construction: Image similarity calculation is performed on the remote sensing sub-images obtained in Step 1, including histogram feature extraction and Euclidean distance calculation of histogram features. Positive and negative sample pairs are constructed based on the imaging time, imaging location, and image similarity information of the remote sensing sub-images. The ratio of positive to negative samples in each pair is 1:1. The specific process for constructing positive and negative sample pairs is as follows:

[0027] right Similarity calculation was performed on the remote sensing sub-graphs to obtain... A similarity metric parameter is used to normalize the similarity metric parameter. Two remote sensing sub-images with a similarity metric parameter less than 0.5 are selected as positive and negative sample candidate pairs. Among the positive and negative sample candidate pairs, two remote sensing sub-images with an imaging time difference greater than 10 years or an imaging location distance greater than 1000 kilometers are selected as positive and negative sample pairs. Data augmentation operations such as random noise addition, random cropping, random rotation, and random cloud and fog addition are performed on the remote sensing sub-images in the positive and negative sample pairs.

[0028] Step 3: Contrastive learning network construction. The input consists of data-augmented positive and negative sample pairs. The positive and negative samples are respectively input into two weight-sharing feature extraction networks. The feature extraction network uses the CSPDarknet53 network, with the feature dimension set to 2048. A two-layer fully connected network is used as the non-linear transformation layer to perform non-linear transformation on the feature representation. The loss calculation feature is set to 256, and the loss function is designed as follows:

[0029]

[0030] in To compare the loss function of the learning network, To match the loss, For mismatch loss, To match the loss coefficients, The mismatch loss coefficient is defined as follows: Matching loss includes the degree of matching between positive samples and between negative samples; mismatch loss includes the degree of matching between positive samples and between negative samples and between positive samples. and The value is between 0 and 1, and + =1, calculate the loss based on the loss function, minimize the loss function through the optimization algorithm, update the parameters of the contrastive learning network, and finally obtain the contrastive learning network;

[0031] Step 4, Remote sensing image representation: Newly acquired remote sensing images that have not participated in training need to be cropped from remote sensing sub-images and then input into the contrastive feature extraction network. There is no need for positive and negative sample construction and data augmentation. The feature extraction network in the contrastive learning network extracts representational features as the representation of the remote sensing image.

Claims

1. A remote sensing image representation method based on contrastive learning, characterized in that, The method comprises the following steps: Step 1, the remote sensing image is subjected to coordinate unification operation, and the remote sensing image with a width exceeding a threshold value is cropped to a set width remote sensing subgraph; Step 2, the remote sensing subgraph obtained in step 1 is subjected to image similarity calculation, and a positive-negative sample pair is constructed according to the imaging time, imaging location and image similarity information of the remote sensing subgraph, and then the positive-negative sample pair is subjected to image enhancement; Step 3, a contrast learning network is constructed, and the input is the positive-negative sample pair, in the contrast learning network, a feature extraction network is first used to extract the representation features of the positive-negative sample pair, then the representation features are subjected to nonlinear transformation to obtain loss function calculation features, and loss calculation is performed according to the loss function, the parameters of the contrast learning network are updated by minimizing the loss function through an optimization algorithm, and finally the contrast learning network is obtained; Wherein, the input of the contrast learning network is the positive-negative sample pair after data enhancement, the positive sample and the negative sample are respectively input into two weight-shared feature extraction networks, the feature extraction network adopts a CSPDarknet53 network, the representation feature dimension is set to 2048, and a double-layer fully connected network is used as a nonlinear transformation layer to perform nonlinear transformation on the representation features. Step 4, the newly imaged remote sensing image not participating in training is input as an input, and the representation features are extracted through the feature extraction network in the contrast learning network, as the representation of the remote sensing image.

2. The contrastive learning-based remote sensing image representation method according to claim 1, wherein, The coordinate unification operation in step 1 is to convert the coordinate system of the remote sensing image into the WGS84 coordinate system, and to crop in units of latitude and longitude per minute.

3. The contrastive learning-based remote sensing image representation method of claim 1, wherein, The image similarity calculation in step 2 includes histogram feature extraction and histogram feature Euclidean distance calculation, the ratio of positive samples to negative samples in the positive-negative sample pair is 1:1, and the positive-negative sample pair construction process is as follows: right Similarity calculation was performed on the remote sensing sub-graphs to obtain... The system uses several similarity metrics to normalize the similarity metrics. Two remote sensing sub-images with similarity metrics less than a set value are selected as positive and negative sample candidate pairs. Among the positive and negative sample candidate pairs, two remote sensing sub-images with an imaging time difference greater than a set duration or an imaging location distance greater than a set kilometer are selected as positive and negative sample pairs. Data augmentation operations are then performed on the remote sensing sub-images in the positive and negative sample pairs, including random noise addition, random cropping, random rotation, and random cloud / fog addition.

4. The remote sensing image representation method based on contrastive learning according to claim 1, characterized in that, In step 3, the loss calculation feature is set to 256, and the loss function is designed as follows: ； wherein is a loss function of the contrast learning network, is a matching loss, is a mismatching loss, is a matching loss coefficient, is a mismatching loss coefficient, the matching loss contains the matching degree between positive sample-positive sample and the matching degree between negative sample-negative sample, the mismatching loss contains the matching degree between positive sample-negative sample and the matching degree between negative sample-positive sample, and is valued between 0 and 1, and + = 1.

5. The contrastive learning based remote sensing image representation method of claim 1, wherein, In step 4, the newly imaged remote sensing image not participating in training is input into the feature extraction network after being cropped into a remote sensing subgraph, and the representation feature dimension of the remote sensing image is consistent with the representation feature dimension in step 3.

Images