Multi-modal remote sensing data classification method based on semi-supervised contrast regular network

Abstract

The invention belongs to the technical field of remote sensing image processing, and particularly relates to a multi-modal remote sensing data classification method based on a semi-supervised contrast regular network. The method comprises the following steps: generating high-quality pseudo labels for all label-free multi-modal remote sensing data by adopting a multi-modal label distribution method, screening samples with balanced categories according to the pseudo labels, constructing a label-free sample set and a label sample set, and performing enhancement processing on the sample sets; constructing a semi-supervised contrast regular network; performing supervised learning of the network by using the labeled sample set; performing consistent regular learning of the network by using the reliable unmarked sample; contrastive learning is executed, an anchor sample set, a positive sample set and a negative sample set are established according to reliability judgment, contrastive loss is calculated, and unreliable samples provide negative sample features for the loss; and jointly using three loss functions to jointly optimize the classification performance of the network. The multi-modal remote sensing data classification method is beneficial to realizing category-balanced network optimization in training, enhances the network robustness, improves the network classification performance, and can be widely applied to multi-modal remote sensing data classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image processing, and particularly relates to a multi-modal remote sensing data classification method based on a semi-supervised contrast regular network. BACKGROUND

[0002] Multi-modal remote sensing images can provide multi-type feature expression for ground cover in the same observation scene by integrating data sources from different sensors in different dimensions. The complementarity of hyperspectral images and laser radar images, two kinds of remote sensing data, makes it possible to achieve higher accuracy and robustness in ground object identification, classification and change detection in complex scenes, and the method has been widely applied in natural disaster monitoring, urban planning, and agricultural and forest resource management.

[0003] Deep learning methods have been increasingly applied to multi-modal remote sensing data classification due to their strong feature learning ability and efficient data processing ability. However, the high performance of deep learning models often requires large-scale labeled data to learn cross-modal feature correlation and classification decision rules. However, multi-modal remote sensing data is high in acquisition cost and large in data volume, and the labeling process is complex and time-consuming, which results in only limited labeled samples available for deep learning in many scenarios.

[0004] In order to solve the problem of the scarcity of labeled samples, some methods have been adopted for deep learning-based multi-modal remote sensing data classification, such as self-supervised learning and transfer learning, to reduce the dependence on remote sensing data label truth values. However, these methods usually only use labeled data, making it difficult for unlabeled samples to directly participate in classification task optimization, which limits the further improvement of model performance. In recent years, semi-supervised learning methods based on pseudo labels have been widely applied, which label a large number of unlabeled samples by predicting pseudo labels, and combine a small amount of labeled data to jointly train to improve the classification performance of the model. The main categories include topology-based methods, generative adversarial networks, and consistency regular learning.

[0005] Among them, consistent regularization learning can effectively enhance the learning ability of the model by applying perturbations to unlabeled data and forcing the model to output consistent pseudo labels. For example, "T. Lu, Y. Fang, W. Fu, K. Ding, and X. Kang,. Dual-stream class-adaptive network for semi-supervised hyperspectral image classification,. IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-11, 2024." constructs a dual-stream network based on consistent regularization for hyperspectral remote sensing data classification. The network uses threshold filtering to select high-confidence samples and makes their prediction results stable in the data processing of the two streams to optimize the network. However, this network only uses samples with high reliability to perform training, and unreliable samples are discarded in the classification strategy, where rich remote sensing information cannot be effectively utilized in network optimization, resulting in a decrease in the feature discrimination of the consistent regularization network for each class of samples, which limits the overall classification accuracy of the network.

[0006] In the face of the above problems, some methods realize the potential value of unreliable unlabeled samples, and therefore optimize the learning effect of the classification network by fully mining their useful information. For example, "K. Ding, T. Lu, and S. Li,“Uncertainty-aware contrastive learning for semi-supervised classification of multimodal remote sensing images,”IEEE Transactions on Geoscience and Remote Sensing, 2024." selects training samples from unlabeled data and divides them into reliable and unreliable sets, respectively, to perform soft learning and hard learning training strategies to optimize the classification network. However, the randomly selected unlabeled samples in this method may cause class distribution imbalance of the training data, resulting in insufficient samples for learning of some classes and training bias, making it difficult to identify the features of specific classes and resulting in lower classification accuracy, which affects the overall classification performance of the network. SUMMARY

[0007] The present application provides a multi-modal remote sensing data classification method based on a semi-supervised contrastive regularization network to overcome the problems of class distribution imbalance of training samples and low classification accuracy in the prior art.

[0008] In order to achieve the above object, the technical scheme of the present application is: a multi-modal remote sensing data classification method based on semi-supervised contrast regular network, comprising the following steps:

[0009] Step one, data preparation of training sample set;

[0010] Step two, construction of semi-supervised contrast regular network;

[0011] Step three, performing supervised learning of network by using labeled sample set;

[0012] Step four, performing consistency regular learning of network by using reliable unlabeled sample;

[0013] Step five, performing contrast learning;

[0014] Step six, jointly using three loss functions to optimize the classification performance of network together.

[0015] Further, the above step one specifically comprises the following steps:

[0016] 1.1 performing superpixel segmentation after multi-modal data fusion, obtaining three possible superpixels, including 1) only labeled pixels, 2) labeled and unlabeled pixels, 3) only unlabeled pixels;

[0017] 1.2 generating pseudo-labels for unlabeled samples in superpixels by using multi-modal label assignment method;

[0018] 1.3 constructing class-balanced unlabeled sample set according to pseudo-labels, and constructing labeled sample set at the same time;

[0019] 1.4 adding different degrees of enhancement processing to training samples to complete data preparation.

[0020] Further, in the above step 1.2, the pseudo-labels of the pixels in the superpixels in the three different cases can be determined by using the multi-modal label assignment method, specifically:

[0021] Case 1: All pixels in the superpixel have clear labels, and no pseudo-labels need to be generated.

[0022] Case 2: Some pixels have labels, select the most pixel class in the superpixel, and assign this class to the unlabeled pixels as their pseudo-labels, while the original labels of the labeled pixels remain unchanged.

[0023] Case 3: There is no label in the superpixel, then calculate the average spectrum and average elevation of the superpixel in two modalities respectively, and measure their feature similarity with the labeled samples, and the closest class is the pseudo-label of all pixels in the current superpixel.

[0024] Further, the semi-supervised contrastive regular network in step two comprises three modules: a multi-modal feature expression module, a classification module, and a consistency contrast module.

[0025] Further, in step two, the multi-modal feature expression module comprises a hyperspectral submodule and a lidar submodule, which are used for independent feature expression of the two modalities, respectively; the data of the two modalities are extracted features after passing through the corresponding submodules, and then the features of the two modalities are fused through element multiplication, and the generated multi-modal feature expression is the output information of the module.

[0026] The classification module consists of a convolutional layer and a fully connected layer.

[0027] The consistency contrast module consists of a convolutional layer and two fully connected layers.

[0028] Further, in step three, the weakly augmented labeled sample set output by step one is extracted feature information through the multi-modal feature expression module in the network, and then the prediction label y of the sample is generated through the classification module. s The prediction label is combined with the true value label y of the sample to establish a supervised loss L CE , and the specific formula is:

[0029]

[0030] wherein, N l is the number of labeled samples with true value y, and CE(·) is a cross-entropy function.

[0031] Further, in step four, the weakly augmented unlabeled samples and the strongly augmented unlabeled samples output by step one are sent into the strong and weak branches of the network to perform class prediction, and the reliability of the prediction is judged in the weakly augmented branch to filter the samples, and the prediction results of the same reliable sample in the two branches are used to calculate a consistency regular loss; the formula of the consistency regular loss function is:

[0032]

[0033] wherein, the function CE(·) represents a cross-entropy function, N UR represents the number of reliable unlabeled samples, represents the reliable prediction in the weakly augmented branch, represents the prediction label of the corresponding sample in the strongly augmented branch.

[0034] Further, the specific steps of step five are as follows:

[0035] 5.1, an anchor sample set is established;

[0036] 5.2, positive samples are established:

[0037] 5.3, Establishing negative sample set:

[0038] Negative samples come from three sample sets: labeled set X kl , reliable unlabeled set and unreliable unlabeled set Wherein, the negative samples from the unreliable unlabeled set require to produce lower prediction probability on the class k; the selection method of the negative samples of the k class on the three sample sets is:

[0039]

[0040] In the formula, A nk is the negative sample set, K is the number of classes, and the sets composed of negative samples from three sources are respectively The similarity is distinguished by the order threshold r, P represents the prediction class confidence of the current sample, and Q(P, k) is the confidence order of the class k in descending order;

[0041] 5.4, Establishing contrast loss function.

[0042] Further, the contrast loss function defined in the above 5.4 is:

[0043]

[0044] Wherein N k represents the number of negative samples, The function represents the calculation of two-parameter cosine similarity.

[0045] Compared with the prior art, the beneficial effects of the present application are:

[0046] 1. The present application proposes a multi-modal label assignment method, which generates high-quality pseudo-labels for unlabeled samples in multi-modal remote sensing data before network training, so as to construct sample sets with consistent number of classes by using pseudo-labels; wherein, for the superpixels with only unlabeled pixels, the average spectrum and average elevation of the superpixels in two modalities are calculated respectively, and the feature similarity with the labeled samples is measured, and finally the class with the closest feature is assigned as the pseudo-label to all pixels in the current superpixel. This method is beneficial to realize the network optimization of class balance in training and enhance the network robustness.

[0047] 2. The present application establishes a semi-supervised contrast regular network, wherein the multi-modal feature expression module for feature extraction designs two separate sub-modules to perform independent feature expression of two modalities, so that the network fully learns the multi-modal features of remote sensing data and enhances the generalization ability of the classification network.

[0048] 3.The present application proposes a contrast learning method based on unreliable samples, which effectively utilizes the potential negative features of unreliable unlabeled samples to optimize the classification network. Specifically, the contrast learning method promotes the anchor sample features to be closer to the positive samples and further away from the negative samples by constructing higher quality anchor sample, positive sample and negative sample sets, thereby improving the class distinction of the network; wherein, for the negative sample selection from the unreliable unlabeled set, the method requires the sample to produce a lower prediction probability on the class k, i.e. k is the least likely class of the sample. The present application can deeply mine potential information from unreliable unlabeled pixels. When the most likely predicted class is not prominent, the least likely class can be effectively identified to provide relevant negative information, thereby enhancing the discriminability of the network.

[0049] 4.The present application adopts three loss functions to jointly optimize the classification network, and improves the global classification accuracy of the network through supervised learning, consistency regularization learning and contrast learning. A large number of experiments on existing data sets show that, compared with existing methods or methods using a single loss function, the present application can significantly improve the network classification performance and can be widely used in multi-modal remote sensing data classification. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The flow chart of the multi-modal remote sensing data classification method based on the semi-supervised contrast regularization network provided by the embodiment of the present application is shown in the figure.

[0051] Figure 2 The schematic diagram of the multi-modal label allocation method designed by the present application is shown in the figure.

[0052] Figure 3 The classification result graph of the contrast method provided by the embodiment of the present application and the method of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments.

[0054] The method of the present application is described in detail below with reference to the drawings. Figure 1 The design idea is as follows:

[0055] Step 1. Data preparation.

[0056] Step 2. Building a semi-supervised contrast regularization network: the network includes three modules of multi-modal feature expression module, classification module and consistency contrast module.

[0057] Step 3. Supervised learning of the network using the labeled sample set: the labeled sample is input into the network to extract features and predict the label, and the labeled sample and its true value are used to establish a supervised loss.

[0058] Step four. Perform consistent regularization learning of the network with reliable no-label samples: the reliable no-label samples that are weakly and strongly enhanced are input into different modules of the two-branch network, and the output class prediction results are used to calculate the consistent regularization loss. Among them, the reliability discrimination is given in the weak enhancement branch.

[0059] Step five. Perform contrastive learning: according to the reliability discrimination, establish the anchor sample, positive sample and negative sample set, and calculate the contrastive loss, wherein the unreliable sample provides negative sample features for the loss.

[0060] Step six. Jointly use three loss functions to jointly optimize the classification performance of the network.

[0061] Based on the above idea, the application provides a multi-modal remote sensing data classification method based on a semi-supervised contrastive regularization network, which specifically includes the following steps:

[0062] Step one. Data preparation of the training sample set.

[0063] In this step, a multi-modal label assignment method is used to generate high-quality pseudo labels for all unlabeled multi-modal remote sensing data, and class-balanced samples are selected according to the pseudo labels to construct a no-label sample set and a labeled sample set, and the sample set is enhanced. The specific steps are as follows:

[0064] 1.1 Perform superpixel segmentation after multi-modal data fusion to obtain three kinds of superpixels. The specific steps are as follows:

[0065] First, the multi-modal data can be represented as follows. The hyperspectral and lidar data set can be represented as {X H ,X L ,y}, wherein is a hyperspectral image, is a lidar image. Here, h and w represent the spatial dimensions of the image data, v represents the number of spectral channels, and y is the true value of the pixel label. {X H ,X L ,y} can be further divided into a labeled set {X HL ,X LL ,y} and an unlabeled set {X HU ,X LU}.

[0066] Second, the steps of multi-modal data fusion are as follows: multiply the lidar data X L and the hyperspectral data X H channel by channel to obtain two modal fusion data X F , that is:

[0067] X F =X H ⊙repeat(XL ,v)

[0068] The function repeat(·) extends X along the spectral dimension. L To match X H The size of the circle is indicated by ⊙, which represents element-wise multiplication.

[0069] Finally, the fused data is segmented into superpixels. The superpixel segmentation algorithm SLIC is applied to the two-modal fused data, X F Divided into a series of superpixels S i ,but i = 1, 2, ..., I, where I is the total number of superpixels. Assume the i-th superpixel S... i If it contains J pixels, then Where s ij Let S represent the j-th pixel in the i-th superpixel. Based on the superpixel S... i The internal pixels have a label status. Each superpixel has three possible states: 1) only labeled pixels, 2) labeled and unlabeled pixels, and 3) only unlabeled pixels.

[0070] 1.2 Generating pseudo-labels for unlabeled samples in superpixels using a multimodal label assignment method, see [link to documentation]. Figure 2 :

[0071] In remote sensing images, land cover information of the same category often forms continuous regions. When the spatial size of a superpixel is small enough, pixels within a superpixel can be approximated as land cover of the same category. Therefore, for superpixels under three different conditions, the pseudo-labels of each pixel within them can be determined by a multimodal labeling method. This method is specifically as follows:

[0072] Case 1: All pixels in a superpixel have explicit labels, so there is no need to generate pseudo-labels.

[0073] Case 2: Some pixels have labels. The pixel category with the most pixels in the superpixel is selected and assigned to unlabeled pixels as their pseudo-labels, while the original labels of labeled pixels remain unchanged.

[0074] Case 3: If no label exists in the superpixel, calculate the average spectrum and average elevation of the superpixel in both modalities, and measure their feature similarity to labeled samples. The closest category is the pseudo-label of all pixels within the current superpixel. The specific algorithm is:

[0075] The feature similarity between the i-th superpixel and class k is defined by the Euclidean distance between the superpixel average spectrum / elevation and the class average spectrum / elevation. Assume the fused data is like X. F There are N labeled pixels of class k, where the spectrum and elevation of the nth sample are respectively... and The Euclidean distance of two modalities is:

[0076]

[0077] where, is the average spectrum of the ith superpixel, is the average spectrum of the labeled sample of class k, and similar parameters correspond to the average elevation represented by the lidar data. A smaller Euclidean distance means a greater similarity, therefore, the smallest and the smallest values are compared, and the pseudo-label is determined by the modality with the shortest distance. Assuming that in all classes, the superpixel S i is closest to class a in spectrum and closest to class b in elevation, i.e. then each pixel in S i is assigned the same pseudo-label

[0078]

[0079] 1.3 Construct a class-balanced unlabeled sample set according to the pseudo-labels, and construct a labeled sample set at the same time.

[0080] Through the above algorithm, all unlabeled pixels in the superpixel can be assigned a pseudo-label. According to the pseudo-label, an equal number of samples are selected from each class to construct an unlabeled sample set, to ensure that the training process does not favor any particular class, thereby improving network robustness and overall accuracy. In addition, the data of the labeled pixels with true values constitute the labeled sample set. In subsequent semi-supervised learning, both the labeled sample set and the unlabeled sample set are used for network training to provide rich and diverse training data.

[0081] 1.4 Add different degrees of enhancement processing to the training samples to complete data preparation.

[0082] The labeled sample set is weakly enhanced by scaling, mirroring, and other processing methods to generate a weakly enhanced labeled sample set. The unlabeled sample set performs the same weak enhancement operation to generate a weakly enhanced unlabeled sample set, and in addition, the unlabeled sample set also increases multi-angle rotation and other operations for strong enhancement to generate a strongly enhanced unlabeled sample set. The three data-enhanced sample sets are used as training data to input the subsequent network for semi-supervised learning.

[0083] Step 2. Construct a semi-supervised contrastive regular network.

[0084] The designed semi-supervised contrastive regular network includes three modules: a multi-modal feature expression module, a classification module, and a consistency contrast module. According to the different input sample sets, the multi-modal feature expression module is connected in series with the classification module or the consistency contrast module, so as to output the predicted label of the sample.

[0085] The multi-modal feature expression module includes a hyperspectral submodule and a lidar submodule, which are used for independent feature expression of the two modalities. Each submodule includes 4 identical computing units connected in series, each unit consisting of a convolution layer (Conv), a batch normalization (BN), a rectified linear unit (ReLU), and a max pooling layer (Max Pooling). The sample data is input into the multi-modal feature expression module, and the data of the two modalities is extracted after passing through the corresponding submodule. Then the features of the two modalities are fused by element multiplication, and the generated multi-modal feature expression is the output information of the module.

[0086] The classification module consists of a convolution layer and a fully connected layer, which is used to predict the sample label from the multi-modal feature expression.

[0087] The consistency contrast module consists of a convolution layer and two fully connected layers, which has the same function as the classification module and is used to predict the label.

[0088] Step three. Perform supervised learning of the network using the labeled sample set.

[0089] The weakly enhanced labeled sample set output from step one is input into the semi-supervised contrastive regular network to perform supervised learning. The weakly enhanced labeled sample is extracted through the multi-modal feature expression module in the network to obtain the feature information, and then the classification module is used to generate the predicted label y s of the sample. The predicted label is combined with the true value label y of the sample to establish a supervised loss, and the specific formula of the supervised loss L CE is as follows:

[0090]

[0091] Where N l is the number of labeled samples with true value y, and CE(·) is the cross-entropy function.

[0092] Step four. Determine the prediction reliability of the unlabeled sample through the weakly enhanced branch, and perform consistency regular learning of the network using reliable unlabeled samples.

[0093] The weak augmented unlabeled sample set and the strong augmented unlabeled sample set generated in step one are sent into the network for consistency regularization learning. Consistency regularization learning requires stable model prediction on unlabeled data under difference perturbation, thereby reducing the dependence on labeled samples. Therefore, the consistency regularization learning of the network will utilize different modules to predict the labels of the same unlabeled sample under different augmentations in the two branches, and force the prediction results of the two branches to be consistent through a consistency regularization loss function, thereby improving the robustness of the network.

[0094] 4.1 Consistency regularization learning is performed on the same unlabeled sample in the two branches, and the specific steps are as follows:

[0095] In the weak augmented branch, the weak augmented unlabeled sample set is input into the multi-modal feature expression module and the classification module. The data processing process in this branch is consistent with step three, and two network modules are shared. Based on the shared multi-modal expression module and classification module constructed by supervised learning, the unlabeled sample in the weak augmented branch generates a predicted label

[0096] In the strong augmented branch, the strong augmented unlabeled sample set is input into the multi-modal feature expression module and the consistency comparison module of the network, and the predicted label y u is output. Compared with the weak augmented branch, the strong augmented branch uses a consistency comparison module instead of a classification module to provide model prediction labels of the same sample under different perturbations and modules. The predicted label y u obtained by the strong augmented branch is used together with the predicted label obtained by the weak augmented branch to drive consistency regularization learning.

[0097] 4.2 Based on the class prediction probability of the weak augmented branch, reliable samples are screened.

[0098] Not all predicted labels are reliable in training, and samples with low reliability may affect the classification accuracy of the network. Therefore, the network identifies the reliability of the prediction results in the weak augmented branch, so as to adopt different training strategies for reliable and unreliable samples.

[0099] In the weak augmented branch, according to the reliability of the output label , the corresponding unlabeled sample can be divided into two categories: samples with reliable labels can be used for consistency regularization learning, and samples with unreliable labels can provide distinguishing information from different categories for contrastive learning. The above two kinds of samples can be distinguished by a threshold θ k specific to category k, which is calculated as where θ0 is the initial threshold, and α∈(0,1] is the weight coefficient for adjusting . For evaluating the classification difficulty of class k, the high-probability sample number μ of the current class k is calculated by k Divided by the maximum sample number μ in all classes c . The high-probability sample number μ is calculated by is the estimated probability when the unlabeled sample is predicted as class k, and I{·} is the indicator function indicating the sample count when the condition in the curly braces is met. μ k The function of μ is to compare the maximum probability of the predicted class with θ0, and the larger μ k indicates that the prediction probability of most unlabeled samples is high, meaning that the current class k is a class with relatively low classification difficulty. Therefore, if the above strategy produces a higher threshold θ k , it leads to fewer predicted labels being considered reliable, and vice versa. The labels with prediction probability exceeding θ k can be considered reliable, and the corresponding reliable unlabeled samples are used for consistent regularization learning.

[0100] 4.3 Constructing a consistent regularization loss by constraining the reliable outputs of the strong and weak enhancement branches through a cross-entropy function to perform consistent regularization learning of the network. The formula of the consistent regularization loss function is:

[0101]

[0102] where the function CE(·) represents the cross-entropy function, N UR represents the number of reliable unlabeled samples, represents the reliable prediction in the weak enhancement branch, represents the predicted label of the corresponding sample in the strong enhancement branch.

[0103] In summary, the present application optimizes the three modules of the network through consistent regularization learning of reliable samples, thereby fully utilizing unlabeled samples to improve the classification performance of the network.

[0104] Step five. Perform contrastive learning of the network using anchor samples, positive samples and negative samples, and calculate the contrastive loss, wherein the unreliable samples provide negative sample features for the loss.

[0105] Through the sample reliability screening of step 4.2, the labels of all samples can be divided into three types based on the prediction results: true value labels y, reliable labels and unreliable labels Correspondingly, the training samples with class k labels can be classified into three sample sets, namely the labeled set X kl , the reliable unlabeled set and the unreliable unlabeled set ​

[0106] This step designs a contrastive learning method based on unreliable samples. This method delves into the latent negative features of specific categories from unreliable samples, enabling the network to better distinguish land cover categories in remote sensing images. Specifically, the contrastive learning method constructs higher-quality sets of anchor samples, positive samples, and negative samples, causing anchor sample features to align more closely with positive samples and less closely with negative samples, thereby improving the network's classification discriminative ability. The specific method is as follows:

[0107] 5.1 Establishing the anchor sample set:

[0108] Anchor samples can be obtained from the labeled set X kl and reliable unlabeled sets Choose from the following. In the current training batch, samples with the true label k or reliably predicted as class k constitute the anchor sample set. Let A k Let m be the set of anchor samples. Having feature vectors

[0109] 5.2 Establishing positive samples:

[0110] A generated in step 5.1 k In the set, each anchor sample Sharing the same positive sample s pk The value is the average of the k-class anchor samples, i.e. Where M is A k Number of anchor samples. Positive samples s pk The eigenvectors are F pk .

[0111] 5.3 Establishing a negative sample set:

[0112] Negative samples come from three sample sets X kl , and By employing different sample selection mechanisms in each set, high-quality negative samples are selected, thereby improving the data quality of the training samples. The specific negative sample selection method is as follows:

[0113] For those from the scalar set X kl negative samples The requirement is that the true value of a feature does not belong to category k, but at the same time has high similarity to category k.

[0114] For those from reliable unstandard sets negative samples It should meet the requirement that the predicted label is not k, but has a high confidence level in category k.

[0115] For those from unreliable unstandard sets negative samples The prediction probability of the class k is required to be low, i.e. k is the least likely class of the sample.

[0116] In summary, assuming that the sets of negative samples of the above three sources are defined as A respectively, and the similarity is distinguished by the ranking threshold r, then the negative sample set A nk can be expressed as:

[0117]

[0118] where K is the number of classes, P represents the predicted class confidence of the current sample. Q(P, k) is the confidence order of class k in descending order, and the larger the value represents the lower probability of being predicted as class k. A nk The mth negative sample in the set can be expressed as: The feature vector of which is:

[0119] 5.4, Establish a contrast loss function.

[0120] Based on the above-mentioned selection method of anchor samples, positive samples and negative samples, a contrast loss function can be constructed to guide the features of anchor samples to be closer to positive samples and farther away from negative samples. The contrast loss function is defined as:

[0121]

[0122] where N k represents the number of negative samples, The function represents the calculation of the cosine similarity of two parameters.

[0123] According to the above-mentioned contrast learning method based on unreliable samples, the present application can deeply mine potential information from unreliable unlabeled pixels. When the predicted most likely class is not prominent, the least likely class can be effectively identified to provide relevant negative information, thereby enhancing the discriminability of the network.

[0124] Step six. Jointly use three loss functions to jointly optimize the classification performance of the network.

[0125] The semi-supervised contrast regular network includes the optimization process of three loss functions. The supervised loss is used to optimize the supervised learning process of the labeled sample set, the consistency regular loss is used to learn the data features of reliable unlabeled samples, and the contrast loss is used to learn the potential negative features of unreliable samples. Therefore, the total loss function L for optimizing the classification performance of the network is composed of three parts, including the supervised loss L CE , the consistency regular loss L CR and the contrast loss L CL , i.e.

[0126] L = L CE + LCR +L CL .

[0127] The technical effects of the present application are described in detail below in combination with simulation experiments:

[0128] The simulation experiment of the present application is realized based on PyTorch, and the hardware platform is NVIDIA GeForce RTX4090. The multi-modal data of hyperspectral and laser radar used in the simulation experiment of the present application adopts the Houston2013 dataset collected by the United States National Airborne Laser Mapping Center, covering the Houston University and its surrounding areas, and was published on the IEEE GRSS data fusion challenge in 2013. It contains single-channel laser radar data with elevation information and hyperspectral images providing spectral information, and the spatial dimension size is 349x1905 pixels, and the spatial resolution is 2.5m / pix. The hyperspectral image has 144 spectral channels, covering a wavelength range of 0.364-1.046μm. The dataset provides 15 land cover categories, a total of 15029 labeled samples.

[0129] Experimental parameter setting: in the training stage, 10 labeled samples are randomly selected from each class as training samples, and the other samples are test sets. The superpixel size J of the Houston2013 dataset is set to 90. The optimizer adopts Adam, and the learning rate is set to 10 -4 . The maximum training round of the model is set to 200, and the batch size is set to 30. The initial threshold θ0 is set to 0.85, and the weight coefficient α of the classification difficulty is set to 0.8.

[0130] In the experiment, the overall accuracy (OA), average accuracy (AA) and kappa coefficient are used for classification performance evaluation. These indicators are widely used in the field of multi-modal remote sensing data classification, and are helpful for comprehensive evaluation of the network model from the objective level.

[0131] Experimental content and result analysis:

[0132] The experiment adopts eight existing methods as a comparison to verify the effectiveness of the present application, and the classification performance is shown in Table 1. It can be seen that the method proposed in the present application shows accurate judgment for each category, and compared with other methods, the highest accuracy is achieved on three evaluation indexes, that is, AA = 92.98%, OA = 94.01%, and Kappa = 92.41%, which is more than two percentage points higher than the suboptimal method GLT-Net. This shows that the improvement of the classification accuracy of the present application is not limited to a few categories, but is effective for all 15 categories in the data set. Although some of the most advanced methods (such as CALC, NNCNet and FHCT) also show high accuracy on specific categories, their lower accuracy on other categories lowers the global index. In contrast, the present application shows superior and robust performance globally, and almost all categories have an accuracy of more than 84%, even on the more difficult to distinguish class 2, the best result (96.38%) is obtained. The samples of the data set are complex and the spatial distribution is sparse, and the present application overcomes the challenge of sample complexity by effectively using the contrast learning strategy, and by using the negative information in unreliable samples, the classification and recognition ability of the network is significantly improved, thereby enhancing the generalization ability of the network. Figure 3 is a classification result map, and compared with other methods, the present application can obtain prediction results closer to the true value on each category, especially the right side of the figure shows the difficult to distinguish categories 1 and 2, the present application can still achieve accurate classification, which proves the superior performance of the present application in multi-modal remote sensing data classification.

[0133] Table 1. Comparison of classification performance of the present application and other methods

[0134]

[0135] The multi-modal label assignment strategy designed in the present application aims to assign high-quality pseudo labels to all multi-modal data samples, thereby providing a class-balanced and sufficient number of unlabeled sample sets for subsequent network training. In order to verify the effectiveness of the multi-modal label assignment strategy, an ablation experiment on the strategy was performed on the Houston2013 dataset, and the training process of the present application includes the multi-modal label assignment strategy. In contrast, the simulation experiment does not contain the strategy, but randomly selects the same number of unlabeled samples, and the experimental results are shown in Table 2. As can be seen from the classification performance indicators, the performance of the present application is significantly improved by introducing the multi-modal label assignment strategy, with OA, AA and Kappa increasing by 1.63%, 1.37% and 1.76%, respectively. The superior performance of the method proposed in the present application in the global range is due to the high-quality pseudo labels generated by the multi-modal label assignment strategy, which generates the most likely estimate of the pixel label by comparing the cross-modal features between the superpixels and the labeled samples, and then performs class-balanced selection according to the label estimation results to provide rich training samples for the network.

[0136] Table 2. Ablation experiment results of multi-modal label assignment strategy

[0137]

[0138] The network designed in the present application is trained by jointly optimizing three loss functions, namely supervised loss L CE , consistency regularization loss L CR and contrastive loss L CL . The influence of the three loss functions on the classification accuracy is demonstrated by ablation experiments, and the experimental results are shown in Table 3. As can be seen, when only the supervised loss is used as the only loss function, the OA, AA and Kappa of the network on the Houston2013 dataset reach 89.68%, 91.45% and 88.84%, respectively. By introducing the consistency regularization loss and the contrastive loss, the performance is significantly improved in these three indicators. When the consistency regularization loss is added to the supervised loss, the OA and AA reach 91.40% and 92.18%, respectively. If the three losses required by the present application are tested, the OA reaches the highest value of 92.98%, and the corresponding AA and Kappa are 94.01% and 92.41%, respectively, which are higher than the results using fewer loss functions. Ablation experiments show that adding additional consistency regularization loss and contrastive loss to supervised learning helps to improve network performance. Experiments prove that the method proposed in the present application can effectively utilize unlabeled samples, i.e. reliable sample features are extracted by consistency regularization learning, and negative features of unreliable samples are extracted by contrastive learning strategy, thereby improving the network's ability to distinguish sample classes.

[0139] Table 3. Ablation experiment results of three loss functions

[0140]

[0141] This embodiment is a specific implementation of the present application, not a limitation of the present application. The skilled in the art of the technical field can also make corresponding equivalent technical solutions without departing from the scope of the present application, so all equivalent technical solutions should be included in the protection scope of the present application.

Claims

1. A multi-modal remote sensing data classification method based on a semi-supervised contrastive regular network, characterized in that: The method comprises the following steps: Step one, data preparation of training sample set; Step two, constructing semi-supervised contrastive regular network; Step three, performing supervised learning of the network by using labeled sample set; Step four, performing consistency regular learning of the network by using reliable unlabeled samples; Step five, performing contrastive learning; Step six, jointly using three loss functions to optimize the classification performance of the network.

2. The method of claim 1, wherein the method is implemented by a semi-supervised contrastive regular network. The step one specifically comprises the following steps: 1.1 performing superpixel segmentation after multi-modal data fusion to obtain three possible superpixels, including 1) only labeled pixels, 2) labeled and unlabeled pixels, and 3) only unlabeled pixels; 1.2 generating pseudo-labels for unlabeled samples in the superpixels by using a multi-modal label assignment method; 1.3 constructing a class-balanced unlabeled sample set according to the pseudo-labels, and constructing a labeled sample set; 1.4 adding different degrees of enhancement processing to the training samples to complete data preparation.

3. The method of claim 2, wherein the semi-supervised contrastive regular network-based multi-modal remote sensing data classification method is characterized by: In the step 1.2, the pseudo-labels of the pixels in the superpixels in the three different cases are determined by the multi-modal label assignment method, specifically as follows: Case 1: All pixels in the superpixel have clear labels, and no pseudo-labels need to be generated; Case 2: Part of the pixels in the superpixel have labels, and the most common pixel class in the superpixel is selected, and the class is assigned to the unlabeled pixels as their pseudo-labels, while the original labels of the labeled pixels remain unchanged; Case 3: There is no label in the superpixel, then the average spectrum and average elevation of the superpixel in the two modalities are calculated respectively, and the similarity of the features with the labeled samples is measured, and the closest class is the pseudo-label of all pixels in the current superpixel.

4. The method of claim 1, wherein the method is implemented by a semi-supervised contrastive regular network based multi-modal remote sensing data classification method. In the step two, the semi-supervised contrastive regular network comprises three modules: a multi-modal feature expression module, a classification module, and a consistency contrast module.

5. The method of claim 4, wherein the semi-supervised contrastive regular network-based multi-modal remote sensing data classification method is characterized by: In the step two, the multi-modal feature expression module comprises a hyperspectral submodule and a lidar submodule, which are respectively used for independent feature expression of two modalities; the data of the two modalities are extracted features after passing through the corresponding submodules, then the features of the two modalities are fused by element multiplication, and the generated multi-modal feature expression is the output information of the module; The classification module is composed of a convolutional layer and a fully connected layer; The consistency contrast module is composed of a convolutional layer and two fully connected layers.

6. The method of claim 1, wherein the method is implemented by a semi-supervised contrastive regular network based multi-modal remote sensing data classification method. In the step three, the weakly augmented sample set output by the step one extracts feature information through the multi-modal feature expression module in the network, and then generates a prediction label y of the sample through the classification module s The prediction label is combined with the true value label y of the sample to establish a supervision loss L CE The specific formula is: where N l is the number of labeled samples with true value y, and CE(·) is the cross-entropy function.

7. The method of claim 1, wherein the method is implemented by a semi-supervised contrastive regular network based multi-modal remote sensing data classification method. In the step four, the weakly enhanced unlabeled samples and the strongly enhanced unlabeled samples output by the step one are sent into the strong and weak branches of the network to perform class prediction respectively, and the reliability of the prediction in the weakly enhanced branch is discriminated to screen the samples, and the prediction results of the same reliable samples in the two branches are used to calculate the consistency regular loss; the formula of the consistency regular loss function is: where the function CE(·) denotes the cross-entropy function, N UR denotes the number of reliable unlabeled samples, denotes the reliable prediction in the weak augmented branch, denotes the predicted label of the corresponding sample on the strong augmented branch.

8. The method of claim 1, wherein the method is implemented by a semi-supervised contrastive regular network based multi-modal remote sensing data classification method. The specific steps of the step five are as follows: 5.1, establishing an anchor sample set; 5.2, establishing positive samples; 5.3, establishing a negative sample set; The negative samples come from three sample sets: labeled set X kl , reliable unlabeled set , and unreliable unlabeled set where the negative samples from the unreliable unlabeled set require a lower predicted probability on class k; The selection method of the k-class negative samples on the three sample sets is as follows: wherein A nk is the set of negative samples, K is the number of classes, and the sets of negative samples from the three sources are The similarity is distinguished by a ranking threshold r, P is the predicted class confidence of the current sample, and Q(P, k) is the confidence order of class k in descending order. 5.4, establishing a contrastive loss function.

9. The method of claim 8, wherein the semi-supervised contrastive regular network-based multi-modal remote sensing data classification method is characterized by: The contrastive loss function constructed in the step 5.4 is defined as: where N k denotes the number of negative samples, The function denotes a two-parameter cosine similarity calculation.

Images