A semantic segmentation method for agricultural land in remote sensing images

Abstract

The invention discloses a method for semantic segmentation of agricultural land of remote sensing images, which selects anchor samples in a pair of positive samples with a relatively close distance in a feature space as reference anchor samples; calculates the similarity between all negative samples and the reference anchor samples, and determines the negative samples with the highest similarity to the reference anchor samples as possible false negative samples; uses this part of the true negative samples in the possible false negative samples as true difficult negative samples that the model relies on in the self-supervised comparative learning of remote sensing images; designs false negative sample confidence calibration loss; uses labeled samples and trains a neural network model, and performs back propagation after the prediction of the neural network model and the real label of the data generate losses, and updates the parameters of the neural network model; inputs the remote sensing image into the trained neural network model, and outputs the semantic segmentation result. The invention adds false negative sample self-determination, abandons the method of introducing unsupervised or abandoning the construction of negative samples, and realizes the approximate determination and calibration of false negative samples.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing, and in particular relates to a method for semantic segmentation of agricultural land in remote sensing images. Background Art

[0002] Agricultural land is land used directly or indirectly for agricultural production, also known as agricultural land, including cultivated land, gardens, forests, pastures, aquaculture water surfaces, pond water surfaces, farmland water conservancy facilities, as well as field roads and other land occupied by all agricultural production buildings. In order to protect agricultural land, images collected by remote sensing satellites are needed to identify agricultural land and prevent changes in the use of agricultural land. Self-supervised learning, as a potential learning paradigm, can achieve autonomous learning of invariant features of remote sensing images in a label-free manner. The basic principle of existing self-supervised contrastive learning of remote sensing images is based on learning invariant features of remote sensing images with positive and negative sample pairs as constraints. However, due to the richness of remote sensing image objects and the complexity of context, positive and negative samples coexist in the same patch and are unevenly distributed, which leads to the basic mechanism of contrastive learning: pushing negative samples away and pulling positive samples closer has serious deviations: that is, pushing negative samples away also pushes positive samples away, and vice versa. We call it the Samples Confounding Issue (SCI) problem.

[0003] Deep neural network models trained by supervised learning have made impressive progress in remote sensing image scene classification, object detection, semantic segmentation and other tasks. The reliance of this method on massive, high-quality labeled samples has become a bottleneck for large-scale applications. The rise of self-supervised contrastive learning makes it possible to autonomously learn the invariant features of images from massive unlabeled remote sensing data.

[0004] The core idea of ​​the self-supervised contrastive learning method is to cleverly obtain copies of the same image as positive samples through data augmentation methods such as rotation, scaling, random color distortion, Gaussian blur, and other spatial and spectral transformations, and other non-similar images as negative samples. The supervision signal is constructed by bringing the positive samples closer and pushing the negative samples further away, rather than manually labeling them as supervision signals, forcing the deep neural network to obtain spatial, spectral, and scale-invariant representations.

[0005] However, due to the complexity of scenes, richness of objects, and uneven distribution of remote sensing images, there is a phenomenon that positive and negative samples coexist in the same patch (image block) and are extremely unbalanced, which leads to a serious deviation in the basic mechanism of contrastive learning: pushing negative samples away and pulling positive samples closer: that is, pushing negative samples away also pushes positive samples away, and vice versa. We call this the Samples Confounding Issue (SCI). The performance loss of the model caused by SCI is called the Samples Confounding Effect (SCE), in which the negative sample image patch containing positive samples gives the wrong feedback signal to the model, so it is called a false negative sample.

[0006] At present, the methods to solve the problem of false negative samples in self-supervised contrastive learning are mainly considered from the perspective of samples, which can be divided into two categories: one is to start with improving the quality of negative sample construction, adding other unsupervised methods on the basis of original self-supervised contrastive learning, and using the additional unsupervised results to guide the self-supervised model to construct higher quality positive and negative sample pairs (PCL, etc.). However, in remote sensing image processing, the use of additional unsupervised clustering methods is often difficult to better improve the quality of positive and negative sample pairs. Instead, it may introduce defects in related unsupervised methods, because the objects contained in remote sensing images often have problems such as sample imbalance, intra-class differences and inter-class similarities, resulting in poor results of unsupervised clustering methods.

[0007] The second type of method considers giving up constructing negative samples (such as Barlow twins, etc.), which means that the performance of the model will only depend on the constructed positive samples. Considering that false negative samples are essentially positive samples that already exist in the dataset, although this type of method completely avoids the generation of false negative samples, it also means that the model will not be able to use the positive samples that already exist in the dataset, which to some extent reduces the model's ability to extract invariant features. Summary of the invention

[0008] In view of this, the present invention proposes a self-supervised contrastive learning model for remote sensing image semantic segmentation with self-discrimination of false negative samples. The core idea is derived from the following observation: the deep neural network model has the ability to distinguish between positive and negative samples, and this ability will potentially be continuously enhanced with the training process of the model. Based on this, we designed two main steps to achieve the approximate determination of false negative samples of the model: one is to make a rough determination of false negative samples through the false negative sample self-determination strategy; the other is to design a false negative sample confidence calibration loss function to perform a fine calibration of false negative samples.

[0009] Different from the above methods, the present invention finds that the remote sensing image self-supervised contrast learning model itself has a certain ability to distinguish true and false negative samples. This ability comes from the relatively correct supervision signals given to the model by the real samples and the true negative samples, and this ability will potentially be continuously strengthened as the model is trained. The present invention calls it the false negative sample self-distinguishability (FSD). This observation inspires the present invention to rethink the SCE problem from the perspective of the model, rather than directly from the perspective of the sample.

[0010] The fundamental difficulty in using FSD to determine false negative samples is that self-supervised pre-training is essentially an unsupervised process. The lack of a definable standard for false negative samples leads to theoretical undecidability, so we can only approximate false negative samples through some strategy. (Approximate determination of false negative samples) From the process point of view, it can be divided into two problems: the problem of coarse determination of false negative samples and the problem of fine calibration of false negative samples. The former performs preliminary screening of false negative samples to ensure the comprehensiveness of false negative samples; the latter performs precise selection based on the former to ensure the accuracy of false negative sample selection.

[0011] The present invention proposes a remote sensing image self-supervised contrast learning method (FALSE) for self-differentiation of false negative samples. The false negative sample self-determination strategy is used to achieve the problem of coarse determination of false negative samples. The false negative sample confidence calibration loss function is designed to solve the problem of fine calibration of false negative samples. The false negative sample self-determination strategy uses the anchor sample in the positive sample pair with a relatively close distance in the feature space as a benchmark, and determines the negative sample with the highest similarity to the anchor sample as a possible false negative sample. The false negative sample confidence calibration loss function is designed, and the false negative sample confidence calibration Loss is used to increase the contribution of possible false negative samples to the positive sample term of the loss function, and reduce its contribution to the negative sample term of the loss function, so as to achieve the purpose of alleviating the sample confusion effect.

[0012] The present invention discloses a method for semantic segmentation of agricultural land in remote sensing images, comprising the following steps:

[0013] Acquire multiple remote sensing images as samples;

[0014] Select the anchor sample in the positive sample pair with a closer distance in the feature space as the benchmark anchor sample o key ;

[0015] Calculate all negative samples and benchmark anchor samples o in the feature space key The similarity between them will be compared with the benchmark anchor sample o key The negative sample with the highest similarity is determined as a possible false negative sample;

[0016] The possible false negative samples n pf This part of true negative samples in the is used as the true difficult negative samples that the model relies on in the self-supervised contrastive learning of remote sensing images;

[0017] Design false negative sample confidence calibration Loss, and reduce the possible false negative sample n through confidence weight α pf Contribution to the negative sample term of the model loss function, increasing the possible false negative samples n pf Contribution of positive sample terms to the model loss function;

[0018] Use the labeled samples to train the neural network model. After the loss between the prediction of the neural network model and the real label of the data is generated, back propagate to calculate the gradient and update the parameters of the neural network model.

[0019] The newly acquired remote sensing images are input into the trained neural network model to output the semantic segmentation results of agricultural land.

[0020] Further, key The conditions met are:

[0021] sim(o key ,p)>T

[0022] Among them, T represents the similarity threshold of the positive sample pair, which is used to control the closeness of the positive sample pair in the feature space. key is the anchor sample of the benchmark, and its corresponding positive sample is denoted as p. sim(·,·) represents the calculation of the feature similarity of the two samples.

[0023] Furthermore, all negative samples and benchmark anchor samples o are calculated in the feature space key The similarity between them will be compared with the benchmark anchor sample o key The negative sample with the highest similarity is determined as a possible false negative sample. The determination condition is expressed as follows:

[0024] |sim(o key ,n pf )-sim(o key ,p)|→min|sim(o key ,n)-sim(o key ,p)|

[0025] n represents the negative sample to be judged, n pf represents a possible false negative sample.

[0026] Furthermore, the truly difficult negative sample satisfies:

[0027] n pf =n f +n h

[0028] n f represents a false negative sample, n h Really hard negative examples that represent model dependency.

[0029] Further, when the number of possible false negative samples determined is N pf , N pf < N, the false negative sample confidence calibration loss function is defined as:

[0030]

[0031] Wherein, is the positive sample signal enhancement term, denoted as S EP , is the false negative sample signal weakening term, denoted as S WN , and the two together with the original signal form a new positive sample signal term S P and a new negative sample signal term S N , α is the confidence weight, is the j-th possible false negative sample, n i is the i-th negative sample other than the possible false negative samples, and o is the anchor sample.

[0032] Further, when the confidence weight α = 0, it means that both the positive sample signal enhancement term and the false negative sample signal weakening term for the model's self-distinguishing of false negative samples are 0; when the confidence weight α = 1, it means that full trust is placed in the possible false negative samples self-distinguished by the model, and all the obtained possible false negative samples are adjusted to be closer to positive samples, while completely eliminating the contribution of this part of the possible false negative samples to the negative sample term of the original loss function.

[0033] Further, the neural network model is a self-supervised contrastive learning neural network, including at least one of CPC, DIM, SimCLR, PCL, and Barlow twins.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] (1) Considering and solving the sample confusion problem (SCI) from the perspective of the model rather than the sample, by adding a false negative sample self-determination module, the method of introducing unsupervised or abandoning the construction of negative samples is abandoned, and the false negative sample self-distinguishing ability (FSD) of the model is utilized to achieve approximate determination and calibration of false negative samples.

[0036] (2) Quantitatively characterizing the strength of the false negative sample self-distinguishing ability of the self-supervised contrastive learning model in the form of the confidence weight of the false negative sample confidence calibration loss function.

[0037] (3) Experimental results on ISPRS Potsdam, DGLC CVPR2018 and Xiangtan semantic segmentation datasets show that the mIoU of the proposed method is improved by an average of 0.7% on ISPRS Potsdams and a maximum of 0.88% compared with SimCLR, PCL and Barlow twins; by an average of 12.28% on DGLC CVPR2018 and a maximum of 28.84%; and by an average of 1.17% on Xiangtan and a maximum of 2.48%, effectively alleviating SCE. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the false negative and sample confusion problems in self-supervised contrastive learning for remote sensing image semantic segmentation;

[0039] Figure 2 The remote sensing image self-supervised comparative learning method framework of the present invention;

[0040] Figure 3 Flow chart of the remote sensing image self-supervised comparative learning method of the present invention;

[0041] Figure 4 The average intersection-over-union ratio of false negative samples with different confidence parameters in the Potsdam dataset for semantic segmentation using the self-differentiated self-supervised contrastive learning model;

[0042] Figure 5 The average intersection-over-union ratio of false negative samples with different confidence parameters in the CVPR DGLC dataset for self-differentiation and self-supervised contrastive learning model semantic segmentation;

[0043] Figure 6 Average intersection-over-union ratio of false negative samples for self-differentiation and self-supervised contrastive learning model semantic segmentation with different confidence parameters in Xiangtan dataset. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0045] The self-supervised learning method is based on contrast constraints. It constructs representations by learning to encode the similarities or dissimilarities between two objects. In remote sensing image self-supervised contrast learning, the true negative samples in the true positive samples and negative samples will give the model feedback on the invariance of the image objects under the constraints of pulling the positive samples closer and pushing the negative samples farther away, helping the model learn the invariance characteristics of the image objects. On the contrary, the false negative samples in the negative samples will give the model feedback on the invariance of the image objects under the constraints of pushing the negative samples farther away, resulting in the sample confusion problem (SCI).

[0046] Since self-supervised pre-training is essentially an unsupervised process, the lack of a definable standard for false negative samples leads to theoretical undecidability, so we can only use some strategy to approximate false negative samples. (Approximate determination of false negative samples) From the process point of view, it can be divided into two problems: the problem of coarse determination of false negative samples and the problem of fine calibration of false negative samples. The former performs preliminary screening of false negative samples to ensure the comprehensiveness of false negative samples; the latter performs precise selection based on the former to ensure the accuracy of false negative sample selection.

[0047] The present invention is based on the above basic understanding and is divided into two main parts. The first is to achieve the problem of coarse determination of false negative samples through the false negative sample self-determination strategy; the second is to design a false negative sample confidence calibration loss function to solve the problem of fine calibration of false negative samples.

[0048] The false negative sample self-determination strategy uses the anchor sample in the positive sample pair with a close distance in the feature space as the benchmark, and determines the negative sample with the highest similarity to the anchor sample as the possible false negative sample; designs the false negative sample confidence calibration loss function, and uses the false negative sample confidence calibration Loss to increase the contribution of possible false negative samples to the positive sample term of the loss function, and reduce its contribution to the negative sample term of the loss function, so as to achieve the purpose of alleviating the sample confusion effect (SCE).

[0049] False negative sample rough determination

[0050] 1) Criteria for determination

[0051] Since the basic goal of self-supervised contrastive learning is to bring positive samples closer and push negative samples further away, if the model projects a pair of positive samples to a closer position in the feature space, it means that the model has currently learned the invariant features of the positive sample pair better. Based on this, selecting anchor samples from positive sample pairs that are closer in the feature space as the judgment benchmark can maximize the use of feature extraction information that the current model has learned, thereby minimizing the model's misjudgment of possible false negative samples.

[0052] If the anchor sample used as the benchmark is recorded as o key , the corresponding positive sample is recorded as p, sim(·,·) represents the calculation of the feature similarity of two samples, then o key The conditions met are:

[0053] sim(o key ,p)>T (1)

[0054] In the above formula, T represents the similarity threshold of the positive sample pair, which is used to control the proximity of the positive sample pair in the feature space.

[0055] 2) Judgment conditions

[0056] On the basis of obtaining the judgment criteria satisfying formula (1), all negative samples and the benchmark anchor samples o are calculated in the feature space. key The similarity between them will be compared with the benchmark anchor sample o key The negative sample with the highest similarity is determined as a possible false negative sample. If n represents the negative sample to be determined, n pf represents a possible false negative sample, then the judgment condition can be expressed as follows:

[0057] |sim(o key ,n pf )-sim(o key ,p)|→min|sim(o key ,n)-sim(o key ,p)| (2)

[0058] 3) Analysis of possible false negative samples

[0059] Affected by the false negative samples, the possible false negative samples obtained by the model are not all false negative samples, but also contain some true negative samples. key In fact, it represents the best level of the current model's ability to extract image features. It is difficult for us to use the model's false negative sample self-distinguishability (FSD) to remove these true negative samples from the possible false negative sample set. Note that this type of negative sample is difficult to distinguish for the model. We will pf This part of true negative samples in is called the model-dependent (true) hard negative samples in remote sensing image self-supervised contrastive learning. f represents a false negative sample, n h Represents a difficult negative sample that the model depends on, then it should satisfy:

[0060] n pf =n f +n h (3)

[0061] False negative sample precision calibration

[0062] 1) False negative sample confidence calibration Loss

[0063] To calibrate the obtained possible false negative samples and mitigate the impact of the hard negative samples that the model depends on in the possible false negative samples on the model performance, different from the InfoNCE loss (Equation (4)) of traditional self-supervised contrastive learning, when calibrating the possible false negative sample n pf to a positive sample, we introduce a confidence weight α and design a false negative sample confidence calibration Loss (see Equation (5)). By the confidence weight, we reduce the contribution of the possible false negative sample n pf to the negative sample term of the model loss function and increase the contribution of the possible false negative sample n pf to the positive sample term of the model loss function, achieving the purpose of enhancing the correct signal and weakening the wrong signal.

[0064] When the number of negative samples corresponding to an anchor sample is N, the loss function of the remote sensing image self-supervised contrastive learning model for a single anchor sample is defined as:

[0065]

[0066] When the number of determined possible false negative samples is N pf (N pf < N), the false negative sample confidence calibration loss function is defined as:

[0067]

[0068] Among them, is the positive sample signal enhancement term, denoted as S EP . is the false negative sample signal weakening term, denoted as S WN . The two together with the original signal form a new positive sample signal term S P and a new negative sample signal term S N , α is the confidence weight, is the j-th possible false negative sample, n i is the i-th negative sample other than the possible false negative samples, and o is the anchor sample. In particular, when there are no false negative samples in the dataset, that is, there are no possible false negative samples, N pf = 0, and Equation (6) degenerates into Equation (5), becoming a general remote sensing image self-supervised contrastive learning model.

[0069] 2) Physical meaning of the confidence weight

[0070] The confidence weight α represents the degree of trust in the model's false negative sample self-distinguishing ability (FSD). When the confidence weight α = 0, it means that the positive sample signal enhancement item and the false negative sample signal reduction item of the model's false negative sample self-distinguishing are both 0. At this time, the model is a traditional self-supervised contrastive learning model, and the positive and negative sample construction strategy of the model's false negative sample self-distinguishing is not used; when the confidence weight α = 1, it means that the possible false negative samples obtained by the model self-distinguishing are fully trusted, and all possible false negative samples obtained are adjusted to be closer to positive samples, and the contribution of this part of possible false negative samples to the negative sample items of the original loss function is completely eliminated; and when the confidence weight α takes any value less than 1 and greater than 0, it means that the model increases the contribution of the possible false negative samples obtained by self-distinguishing to the positive sample item of the loss function from 0 to α times the original contribution to the negative sample item, and reduces its contribution to the negative sample item to 1-α times the original.

[0071] The present invention can be applied to various contrast constraint-based self-supervision frameworks, including but not limited to CPC, DIM, SimCLR, PCL and Barlow twins.

[0072] The experiment selected the public remote sensing image semantic segmentation dataset ISPRS Potsdam, the competition remote sensing semantic segmentation dataset CVPR DGLC2018, and the remote sensing semantic segmentation dataset Xiangtan from the Gaofen-2 satellite covering Xiangtan, China. The spatial resolution and number of ground object types of the three datasets are shown in Table 1.

[0073] Table 1 Dataset type and other parameters

[0074] Dataset name Spatial resolution Number of classes Dataset Type ISPRS Potsdam 0.05m 6 Public Datasets CVPR DGLC2018 0.5m 7 Competition Dataset Xiangtan 2m 8 Actual Dataset

[0075] The experiment follows the general paradigm of the self-supervised learning model, that is, based on the fixed image feature extractor (encoder) network weights obtained by self-supervised pre-training, the decoder part is trained using a small amount of labeled data. Self-supervised pre-training is performed for 200 cycles, batch size = 256, and the labeled data is selected from 1% of the pre-training unlabeled data. The decoder is used to obtain the image semantic segmentation results.

[0076] In order to quantitatively explore the false negative self-distinguishing ability (FSD) of the self-supervised contrastive learning model, we conducted multiple experiments with the introduced confidence weight α, and the similarity threshold T of the positive sample pair was set to 0.9.

[0077] See Experiment 1 for the results.

[0078] Experiment 1: Confidence Weight Experiment

[0079] Five confidence weights of 0, 0.3, 0.5, 0.7 and 1 were selected for experiments. Each confidence weight experiment was repeated 5 times while keeping other parameters consistent. Figure 4-Figure 6 The self-supervised contrastive learning model for self-differentiation of false negative samples shows the mIoU comparison on the Potsdam, CVPR DGLC2018, and Xiangtan datasets with five different confidence weights of 0, 0.3, 0.5, 0.7, and 1. When the confidence parameter is 1, that is, when the false negative samples obtained by self-differentiation of the model are fully trusted, the model performs best. Compared with the self-supervised contrastive learning model with the traditional positive and negative sample construction strategy, the mIoU on the Potsdam dataset is improved by 0.72%, the mIoU on the CVPR DGLC dataset is improved by 0.8%, and the mIoU on the Xiangtan dataset is improved by 0.56%. At the same time, we noticed that when the confidence weight takes a value between 0 and 1, that is, when the model partially trusts the suspicious false negative samples obtained by self-differentiation, the semantic segmentation performance of the model becomes unstable and even outlier accuracy statistics appear. Considering that such a confidence weight value actually reduces the contribution of possible false negative samples to both the positive and negative sample terms of the loss function, and possible false negative samples are composed of false negative samples and some difficult negative samples that the model depends on, this phenomenon actually suggests that false negative samples and difficult negative samples that the model depends on affect the stability of the model in extracting image features to a certain extent.

[0080] Table 2 Semantic segmentation IoU (%) of different confidence parameters of Potsdam dataset

[0081] α Imp.Surface Building Low Veg. Tree Car Clutter / background 0 47.87 47.15 41.22 27.94 <![CDATA[ 28.22 ]]> 4.74 0.3 47.87 47.72 42.07 27.37 24.86 5.26 0.5 <![CDATA[ 48.08 ]]> 46.15 42.06 <![CDATA[ 30.31 ]]> 26.42 4.85 0.7 47.82 <![CDATA[ 47.42 ]]> <![CDATA[ 42.22 ]]> 28.05 27.05 <![CDATA[ 4.93 ]]> 1 48.16 46.83 42.38 30.75 28.29 5.26

[0082] Table 3 Semantic segmentation IoU (%) of different confidence parameters in CVPR DGLC2018 dataset

[0083]

[0084]

[0085] Table 4 Semantic segmentation IoU (%) of different confidence parameters of Xiangtan dataset

[0086] α farmland urban Rural areas water woodland grassland roads background 0 63.82 0 16.62 38.97 78.27 1.95 21.52 97.56 0.3 <![CDATA[ 64.10 ]]> 0.31 17.16 <![CDATA[ 41.30 ]]> 78.49 1.86 21.3 98.08 0.5 63.82 0.53 <![CDATA[ 17.23 ]]> 40.94 78.11 1.91 21.01 98.09 0.7 64.06 2.75 18.25 40.42 78.12 1.70 21.68 <![CDATA[ 98.2 ]]> 1 64.45 <![CDATA[ 0.96 ]]> 14.92 43.01 78.36 2.26 <![CDATA[ 21.58 ]]> 98.23

[0087] Table 2-4 shows the semantic segmentation accuracy of various types of objects selected by the model with different confidence weights on the Potsdam, CVPR DGLC2018 and Xiangtan datasets. Compared with traditional self-supervised contrastive learning, the model improves the mIoU of trees in the Potsdam dataset by 2.81%, the mIoU of water bodies in CVPR DGLC2018 by 4.48%, and the mIoU of water bodies in Xiangtan by 4.04%.

[0088] On this basis, we selected the self-supervised contrastive learning model SimCLR, which represents the traditional positive and negative sample construction strategy, the self-supervised contrastive learning model PCL, which represents the clustering, and the self-supervised learning model Barlow twins, which represents the non-construction of negative samples, as the baselines and tested them on Potsdam, DGLC, and Xiangtan. We compared them with the self-supervised contrastive learning of false negative samples with a confidence weight of 1. The results are shown in Experiment 2.

[0089] Experiment 2 Comparison of self-supervised learning models with four different positive and negative sample construction strategies

[0090] Table 5 Comparison of semantic segmentation accuracy of four types of representative self-supervised methods on three datasets

[0091]

[0092]

[0093] Table 5 shows the overall semantic segmentation accuracy of four types of self-supervised learning models with different positive and negative sample strategies on the public datasets ISPRS Potsdam, CVPR DGLC, and Xiangtan datasets. It can be seen that PCL, which represents the construction of positive and negative samples assisted by clustering, and Barlow twins, which represents the abandonment of the construction of negative samples, have improved on the ISPRS Potsdam dataset compared with the self-supervised learning method using traditional positive and negative sample construction, but have poor performance on the public competition datasets CVPR DGLC and Xiangtan datasets, even worse than SimCLR, and the performance results are unstable. The self-supervised comparative learning model of false negative sample self-discrimination has a stable improvement effect on the ISPRS Potsdam, CVPR DGLC, and Xiangtan datasets compared with the other three methods, namely, the self-supervised learning method using the traditional positive and negative sample construction strategy, the self-supervised learning method adding the clustering-assisted positive and negative sample construction strategy, and the self-supervised learning method abandoning the construction of negative samples. The best results among the four methods were achieved.

[0094] The present invention proposes a remote sensing image self-supervised contrastive learning model (FALSE) with self-differentiation of false negative samples, and realizes the approximate determination of false negative samples of the model through rough determination and fine calibration of false negative samples. Experiments show that the remote sensing image self-supervised contrastive learning model with self-differentiation of false negative samples can effectively alleviate the SCE in the self-supervised contrastive learning of remote sensing images. Compared with the self-supervised contrastive learning SimCLR representing the traditional positive and negative sample construction, the self-supervised contrastive learning PCL representing the introduction of clustering to guide the construction of positive and negative samples, and the self-supervised learning Barlow Twins without constructing negative samples, the mIoU on ISPRS Potsdams is improved by an average of 0.7%, and the maximum is improved by 0.88%; on DGLC CVPR2018, the mIoU is improved by an average of 12.28%, and the maximum is improved by 28.84%; on Xiangtan, the mIoU is improved by an average of 1.17%, and the maximum is improved by 2.48%.

[0095] The current method is only a simple implementation of the model's false negative sample self-distinguishing ability. Although it makes full use of the sample similarity information obtained by the model, it introduces manually set confidence weights. For the remote sensing image semantic segmentation task, the confidence weights corresponding to the optimal accuracy of different objects are also different. How to adaptively adjust the confidence weights for different object samples in the dataset and give full play to the model's false negative sample self-distinguishing ability may be an issue that needs to be further considered in the future for the self-supervised contrastive learning model of false negative sample self-distinguishing.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] (1) The sample confusion problem (SCI) is solved from the perspective of the model rather than the sample. By adding a false negative sample self-determination module, the method of introducing unsupervised learning or giving up the construction of negative samples is abandoned. The false negative sample self-distinguishing ability (FSD) of the model is used to achieve approximate determination and calibration of false negative samples.

[0098] (2) The false negative sample self-distinguishing ability of the self-supervised contrastive learning model is quantitatively characterized in the form of the confidence weight of the false negative sample confidence calibration loss function.

[0099] (3) Experimental results on ISPRS Potsdam, DGLC CVPR2018 and Xiangtan semantic segmentation datasets show that the mIoU of the proposed method is improved by an average of 0.7% on ISPRS Potsdams and a maximum of 0.88% compared with SimCLR, PCL and Barlow twins; by an average of 12.28% on DGLC CVPR2018 and a maximum of 28.84%; and by an average of 1.17% on Xiangtan and a maximum of 2.48%, effectively alleviating SCE.

[0100] The word "preferred" as used in the present invention is intended to be used as an example, instance or illustration. Any aspect or design of the present invention described as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included arrangements. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the aforementioned examples.

[0101] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown in the present invention. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0102] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0103] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for semantic segmentation of agricultural land in remote sensing images, characterized in that: The following steps are involved: Acquire multiple remote sensing images as samples; Select the anchor sample in the positive sample pair with a closer distance in the feature space as the benchmark anchor sample o key ; Calculate all negative samples and benchmark anchor samples o in the feature space key The similarity between them will be compared with the benchmark anchor sample o key The negative sample with the highest similarity is determined as a possible false negative sample; The possible false negative samples n pf This part of true negative samples in the is used as the true difficult negative samples that the model relies on in the self-supervised contrastive learning of remote sensing images; Design false negative sample confidence calibration Loss, and reduce the possible false negative sample n through confidence weight α pf Contribution to the negative sample term of the model loss function, increasing the possible false negative samples n pf Contribution of positive sample terms to the model loss function; Use the labeled samples to train the neural network model. After the loss between the prediction of the neural network model and the real label of the data is generated, back propagate to calculate the gradient and update the parameters of the neural network model. The newly acquired remote sensing images are input into the trained neural network model to output the semantic segmentation results of agricultural land.

2. The method for semantic segmentation of agricultural land in remote sensing images according to claim 1, characterized in that: o key The conditions met are: yes(the key ,PT Among them, T represents the similarity threshold of the positive sample pair, which is used to control the closeness of the positive sample pair in the feature space. key is the anchor sample of the benchmark, and its corresponding positive sample is denoted as p. sim(·,·) represents the calculation of the feature similarity of the two samples.

3. The method for semantic segmentation of agricultural land in remote sensing images according to claim 2, characterized in that: Calculate all negative samples and benchmark anchor samples o in the feature space key The similarity between them will be compared with the benchmark anchor sample o key The negative sample with the highest similarity is determined as a possible false negative sample. The determination condition is expressed as follows: |yes(the key ,n pf )-yes(the key ,p)|→min|yes(o key ,n)-yes(o key ,p)| n represents the negative sample to be judged, n pf represents a possible false negative sample.

4. The method for semantic segmentation of agricultural land in remote sensing images according to claim 3, characterized in that: The truly difficult negative samples satisfy: n pf =n f +n h n f represents a false negative sample, n h Really hard negative examples that represent model dependency.

5. The method for semantic segmentation of agricultural land in remote sensing images according to claim 4, characterized in that: When the number of possible false negative samples determined is N pf , N pf < N, where N is the number of negative samples corresponding to an anchor sample, the false negative sample confidence calibration loss function is defined as: in, is the positive sample signal enhancement item, denoted by S EP , is the false negative sample signal attenuation term, denoted by S WN , the two together with the original signal form a new positive sample signal term S P And the new negative sample signal term S N , α is the confidence weight, is the jth possible false negative sample, n i is the i-th negative sample excluding possible false negative samples, and o is the anchor sample.

6. The method for semantic segmentation of agricultural land in remote sensing images according to claim 5, characterized in that: When the confidence weight α=0, it means that the positive sample signal enhancement term and the false negative sample signal weakening term of the model's false negative sample self-differentiation are both 0; when the confidence weight α=1, it means that the possible false negative samples obtained by the model self-differentiation are fully trusted, and all possible false negative samples are adjusted to be closer to positive samples, while completely eliminating the contribution of these possible false negative samples to the negative sample term of the original loss function.

7. The method for semantic segmentation of agricultural land in remote sensing images according to claim 1, characterized in that: The neural network model is a self-supervised contrastive learning neural network, including at least one of CPC, DIM, SimCLR, PCL and Barlow twins.

Images