A Small-Sample Hyperspectral Image Semantic Segmentation Method and System Based on Multiple Classifiers

Abstract

A semantic segmentation method and system for small-sample hyperspectral remote sensing images based on multiple classifiers, including its basic feature extraction module, multiple classifier module, and loss function module. The basic feature extraction module can effectively utilize the category channel information of hyperspectral images, establish the relationship features between marked points and context, enhance the overall category's ability to express features, relatively effectively distinguish the categories to which some features belong, and improve the overall accuracy of semantic segmentation of small-sample hyperspectral images; the multiple classifiers ensure classifier diversity and prediction consistency for unlabeled data by maximizing and minimizing adversarial losses, improve the classification accuracy of unlabeled data, and thus improve the overall accuracy of semantic segmentation of small-sample hyperspectral images.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, is used for the classification of small-sample remote sensing images, and specifically relates to a semantic segmentation method for small-sample hyperspectral images based on multiple classifiers. Background Art

[0002] Hyperspectral remote sensing images are radiometric data acquired by imaging spectrometers across dozens or even hundreds of continuous spectral channels. They possess high spectral resolution and can capture a wealth of spectral, radiometric, and spatial information about the Earth's surface. They hold enormous potential for application in geology, the environment, agriculture, hydrology, military affairs, and the atmosphere. With the continuous advancement of my country's resource satellite exploration technology, acquiring hyperspectral remote sensing image data is no longer as difficult as it once was. However, hyperspectral remote sensing image data with sufficient labeled samples remains valuable. Therefore, achieving high-precision hyperspectral remote sensing image classification using less labeled data is of great research significance.

[0003] With the continuous development of computer science and technology in recent years, deep learning has made significant progress in the field of machine learning. Research on hyperspectral image classification using deep learning has increased. Compared to conventional machine learning, which requires manual feature design, deep learning can automatically extract features from data. Therefore, many researchers have proposed hyperspectral image classification algorithms based on deep learning. Convolutional Neural Networks (CNNs) are currently the most widely used method for hyperspectral image classification. CNNs require that the input data are all labeled pixel blocks. They perform feature extraction and knowledge learning on small image blocks containing the sample point and its surrounding pixels, obtaining a classification label as the category label for the center pixel. However, due to the small number of samples in the dataset, the learned knowledge cannot represent the overall data distribution, causing the CNN network to overfit the small sample data and poor overall image classification performance. Furthermore, the pixel-by-pixel segmentation of the small image blocks for classification results in image-level labels, which means that surrounding pixels other than the center pixel are repeatedly classified, increasing system overhead. Furthermore, the size of the segmented small image blocks is limited, making it impossible to model the larger contextual information within the hyperspectral image.

[0004] To address these issues, using fully convolutional networks (FCNs) for hyperspectral image classification is an effective solution. Semantic segmentation algorithms based on fully convolutional networks predict and classify each pixel in an image. Hyperspectral image classification is also pixel-level classification, making it feasible to use fully convolutional network semantic segmentation algorithms for hyperspectral image classification. Fully convolutional networks do not restrict the shape and size of input and output data, nor do they require all input data to be labeled. The network's receptive field is not limited to the size of image patches. The network's input and output correspond pixel-by-pixel, and the image patches can contain multiple sample points and a large amount of unlabeled data. In the case of small sample sizes, the contribution of labeled data to the network is limited, leaving the vast amount of unlabeled data in the image containing feature information that needs to be utilized. While many semantic segmentation methods, thanks to the powerful feature representation capabilities of fully convolutional networks (FCNs), can achieve good segmentation results for high-resolution remote sensing images, these methods are not suitable for hyperspectral images. The large number of stacked convolutional layers can easily lead to overfitting when the amount of hyperspectral image data is small. The large number of pooling layers in the network can lead to rough edge classification and misclassification. The semantic segmentation networks currently used in hyperspectral images do not take into account the sparse label characteristics of hyperspectral images. Due to the limitation of a small number of labels, they cannot represent the overall data distribution, which affects the image segmentation effect. Summary of the Invention

[0005] In response to the above problems or improved algorithms in the prior art, the present invention provides a semantic segmentation method and system for small-sample hyperspectral remote sensing images based on multiple classifiers, which aims to solve the problem that the existing semantic segmentation algorithm has poor classification effect on small-sample hyperspectral images.

[0006] In order to achieve the above object, the present invention provides a multi-classifier-based semantic segmentation method for small-sample hyperspectral remote sensing images, comprising the following steps:

[0007] S1 builds a semantic segmentation network for small-sample hyperspectral images, which includes a basic feature extraction module, a multi-classification module, and a loss function;

[0008] The basic feature extraction module is used to extract basic features from the input hyperspectral image and output the basic features of the image. The basic feature extraction module consists of four layers, each of which consists of a 3×3 convolutional layer, a BN layer and a ReLU layer. A maximum pooling layer is added only after the second layer to increase the receptive field size of the network and avoid excessive downsampling.

[0009] The multi-classifier module includes two classifiers, each of which is composed of a 1×1 convolutional layer; after the basic feature extraction module, the features are up-sampled by bilinear interpolation before being input into the classifier so that the output feature size is the same as the input data block size;

[0010] The loss function includes a cross - entropy loss function, an adversarial loss function, and a manifold - constraint regularization loss function. The overall loss function is expressed as:

[0011] L total = L ce (X L , Y L ) + αL mr (X S ) - βL adv (X U )

[0012] where L ce is the cross - entropy loss function, L mr is the manifold - constraint regularization, L adv is the adversarial loss function, X S represents the randomly selected sample points from the labeled samples and unlabeled samples, a small amount of labeled input data X L , the labeled information Y L of the data, and a large amount of unlabeled samples X U ;

[0013] S2 uses the small - sample hyperspectral remote - sensing images captured by drones or satellites as the training set, and iteratively trains the semantic segmentation network to obtain a semantic segmentation model;

[0014] S3 presets the corresponding number of categories, and inputs the hyperspectral remote - sensing image to be detected into the trained semantic segmentation model to obtain the category to which each pixel point in the hyperspectral remote - sensing image to be classified belongs.

[0015] Furthermore, the basic feature extraction module downsamples the feature map to 1 / 2.

[0016] Furthermore, the adversarial loss function is defined as:

[0017]

[0018]

[0019] where p1 and p2 respectively represent the prediction results of G c1 and G c2 , p1 c and p2 c represent the probability that the sample belongs to the C - th category, and S represents the softmax layer;

[0020] The manifold - constraint regularization constraint function is expressed as:

[0021]

[0022] Among them, W represents the graph adjacency matrix, L = D - W represents the graph Laplacian matrix, D represents the diagonal matrix, Xs are the selected samples in the input image block, N represents the number of selected samples, Tr represents taking the rank, and the adjacency matrix W is defined as:

[0023]

[0024] Among them, σ is the heat kernel parameter. For each pixel x i , the Euclidean distance is used to select the neighbor x j , and W ij represents the similarity between them. Minimize the manifold constraint regularization loss function, and constrain p i and p j are also similar.

[0025] Furthermore, the training steps of the semantic segmentation network include:

[0026] ① Minimize the cross-entropy loss function and the manifold constraint loss. The training goal is that both classifiers can correctly classify the labeled samples, and the predictions for the labeled data and the unlabeled data both follow the manifold constraint. The objective function is:

[0027]

[0028] ② Maximize the adversarial loss function. Fix the network parameters of the basic feature extraction module, and train the difference between the two classifiers. At the same time, they can correctly classify the labeled data and satisfy the manifold constraint. The objective function is:

[0029]

[0030] ③ Minimize the adversarial loss function. Fix the parameters of the two classifiers, and train the basic feature extraction module to minimize the prediction difference. The objective function is:

[0031]

[0032] Minimizing the prediction difference is equivalent to maximizing the prediction consistency. Since adversarial learning requires the initial classifier to be able to correctly classify the labeled samples, the network is trained in step ① in the first 20 epochs, and then the three steps are repeatedly iteratively trained until the network converges;

[0033] Finally, the two classification features output by the two classifiers are calculated by the softmax function to obtain the classification probability matrix of H×W×C. The average value is taken for the number of category dimensions to obtain the classification probability corresponding to each pixel point for each category, and the maximum probability is taken as the category to which the pixel point belongs as the final determination category of this system.

[0034] Based on the same inventive concept, the present invention also designs an electronic device, which is characterized by including:

[0035] One or more processors;

[0036] A storage device for storing one or more programs;

[0037] When the one or more programs are executed by the one or more processors, the one or more processors implement the described small-sample hyperspectral remote sensing image semantic segmentation method based on multiple classifiers.

[0038] Based on the same inventive concept, the present invention also designs a computer-readable medium, on which a computer program is stored, and is characterized in that: when the program is executed by a processor, it implements the described small-sample hyperspectral remote sensing image semantic segmentation method based on multiple classifiers.

[0039] Generally speaking, compared with the existing solutions through the above technical solutions conceived by the present invention, the following effects can be achieved.

[0040] The basic feature extraction module in the present invention can effectively utilize the category channel information of the hyperspectral image, establish the relationship features between the marked points and the context, enhance the overall category's ability to express features, relatively effectively distinguish the categories to which some features belong, and improve the overall accuracy of small-sample hyperspectral image semantic segmentation;

[0041] The multiple classifier module in the present invention can effectively extract the discriminative features of unlabeled data, ensure the diversity of classifiers and the prediction consistency for unlabeled data by maximizing the minimum adversarial loss, improve the classification accuracy of unlabeled data, and thus improve the overall accuracy of small-sample hyperspectral image semantic segmentation;

[0042] The manifold constraint regularization in the present invention can clearly constrain the smoothness of the classification between unlabeled sample data, further utilize unlabeled data, and improve the overall accuracy of small-sample hyperspectral image semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is the network framework diagram of the hyperspectral remote sensing image semantic segmentation method provided by the embodiment of the present invention

[0044] Figure 2 is the flowchart of the hyperspectral image semantic segmentation method provided by the embodiment of the present invention

[0045] Figure 3 is the structural schematic diagram of the multiple classifier module in the embodiment of the present invention DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The present invention will be further described below with reference to the drawings and embodiments.

[0047] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementations described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] A small-sample hyperspectral remote sensing image semantic segmentation method based on multiple classifiers provided by the present invention includes:

[0049] Construct a small-sample hyperspectral image semantic segmentation network, which includes a basic feature extraction module, a loss function part, and a multiple classifier module.

[0050] The basic feature extraction module is used to extract basic features of the input hyperspectral image and output the basic features of the image.

[0051] The loss function part includes three parts, namely the cross-entropy loss function, the adversarial loss function, and the manifold constraint regularization loss function. Among them, for a small number of labeled samples, the cross-entropy loss function can obtain a feature extraction module that can better classify the labeled data by supervising the training of this part of the data; for unlabeled samples, the manifold constraint regularization is used to supervise the network to learn the manifold structures of the input labeled samples and unlabeled samples, and improve the classification accuracy of unlabeled samples.

[0052] The multiple classifier module is used to classify the basic features of the extracted image. Each classifier will obtain a corresponding classification probability map. By constraining two different classifiers to have consistent prediction results for each unlabeled sample, the features of the extracted unlabeled data are made more discriminative, that is, according to the classification difference, the network can identify the data that is not easy to classify and make such data far from the decision boundary, prompting the network to extract more discriminative features. Finally, the multiple discriminative features output by the multiple classifiers are fused to obtain a fused feature, and each pixel point in the fused feature is classified to obtain the classification result of the final image.

[0053] Taking the small-sample hyperspectral remote sensing images captured by drones or satellites as the training set and iteratively training the semantic segmentation network, a semantic segmentation model can be obtained;

[0054] Inputting the hyperspectral remote sensing image to be detected into the trained semantic segmentation model, the category to which each pixel point in the hyperspectral remote sensing image to be classified belongs can be obtained. Since the corresponding number of categories is preset, the points in the hyperspectral image can be divided into specific categories such as water bodies, roads, houses, shrubs, etc.

[0055] AsFigure 1 As shown in Figure 1 , an invention of semantic segmentation of small - sample hyperspectral remote - sensing images based on multiple classifiers provided by the present invention includes:

[0056] Construct a semantic segmentation network;

[0057] Specifically, the small - sample hyperspectral remote - sensing images in the present invention generally refer to the "three - dimensional data blocks" containing dozens to hundreds of continuous and fine - divided spectral bands captured by drones and remote - sensing satellites, and the labels are sparse.

[0058] The semantic segmentation network includes a basic feature extraction module, a loss function part, and a multiple classifier module:

[0059] (1) The basic feature module is used to perform basic feature extraction on the small - sample hyperspectral remote - sensing images input into the system; among which the input data has only a small amount of labeled data, which can be expressed as X L = [x1, x2, …, x L ∈ R D×L , the label information is Y L ∈ R C×L , D represents the spectral dimension, L represents the number of labeled sample locations, C represents the number of categories, and Y L is the one - hot encoding of the label information. A large number of unlabeled samples can be denoted as X U ∈ R D×U , where U is the number of unlabeled data. The basic feature module is labeled as G f .

[0060] Furthermore, the size of the data block input into the network is 128×128×C’ or 64×64×C’, where C’ represents the number of spectral channels; among which only a very small number of samples in each data block have label information; most are unlabeled samples;

[0061] Furthermore, the data block is input into the basic feature extraction module. The basic feature module can be divided into four layers, and each layer consists of a 3×3 convolutional layer, a BN layer, and a ReLU layer. In addition, only after the second layer, a max - pooling layer is added to increase the receptive field size of the network and avoid excessive down - sampling. Before the features are input into the classifier, an up - sampling operation is performed by bilinear interpolation to make the output feature size the same as the input data block size; finally, the size of the feature map output by the basic feature extraction module is 256×H×W, where H and W are the sizes of the feature map, specifically 256×128×128 or 256×64×64.

[0062] (2) The loss function part includes three parts, namely the cross - entropy loss function, the adversarial loss function, and the manifold - constraint regularization loss function. The overall loss function is expressed as:

[0063] L total = L ce (X L , Y L ) + αL mr (X S ) - βL adv (X U )

[0064] Among them, L ce is the cross - entropy loss function, L mr is the manifold constraint regularization, L adv is the adversarial loss function, X S represents the randomly selected sample points from labeled samples and unlabeled samples. The parameter α is used to balance the manifold constraint loss function, and the parameter β is used to balance the adversarial loss function.

[0065] (3) The multi - classifier module classifies the features to be classified. Each classifier is composed of a 1×1 convolutional layer, and the output result is C×H×W. Here, the two classifiers are respectively denoted as G c1 and G c2 . The adversarial loss function can be defined as:

[0066]

[0067]

[0068] Among them, p1 and p2 respectively represent the prediction results of G c1 and G c2 . p1 c and p2 c represent the probability that the sample belongs to the C - th class, S represents the softmax layer, and E represents taking the mean.

[0069] Furthermore, maximizing the adversarial loss function ensures the diversity of the two classifiers, and minimizing the adversarial loss function trains the basic feature extraction module to obtain consistent predictions for unlabeled data. The training method of the adversarial loss function is:

[0070]

[0071] (4) The manifold constraint regularization constrains that the prediction results have the same smoothness as the data manifold, which can be expressed as:

[0072]

[0073] Among them, W represents the graph adjacency matrix, L = D - W represents the graph Laplacian matrix, D represents the diagonal matrix, Xs is the selected sample in the input image patch, N represents the number of selected samples, and the adjacency matrix W is defined as:

[0074]

[0075] Among them, σ is the thermal kernel parameter. For each pixel x i , the nearest neighbor x is selected using the Euclidean distance j , and W ij represents the similarity between them. The manifold constraint regularization loss function is minimized, and p i and p j are also similar.

[0076] (5) The specific training steps can be expressed as follows:

[0077] ① Minimize the cross-entropy loss function and the manifold constraint loss. The training objective is that both classifiers can correctly classify the labeled samples, and the predictions for the labeled data and the unlabeled data both obey the manifold constraint. The objective function is:

[0078]

[0079] ② Maximize the adversarial loss function. Fix the network parameters of the basic feature extraction module, and train the difference between the two classifiers. At the same time, they can correctly classify the labeled data and satisfy the manifold constraint. The objective function is:

[0080]

[0081] ③ Minimize the adversarial loss function. Fix the parameters of the two classifiers, and train the basic feature extraction module to minimize the prediction difference. The objective function is:

[0082]

[0083] Minimizing the prediction difference is equivalent to maximizing the prediction consistency. Since adversarial learning requires the initial classifier to be able to correctly classify the labeled samples, the network is trained in step ① for the first 20 epochs, and then the three steps are repeatedly iteratively trained until the network converges.

[0084] Finally, the two classification features output by the two classifiers are calculated by the softmax function to obtain a classification probability matrix of H×W×C. The average value is taken for the number of classes dimension to obtain the classification probability corresponding to each pixel point for each class, and the maximum probability is taken as the class to which the pixel point belongs as the final determination class of this system.

[0085] The training process is as follows:

[0086] Using the hyperspectral remote sensing image target detection dataset as the training set, the above semantic segmentation network can be iteratively trained to obtain a hyperspectral remote sensing image semantic segmentation model;

[0087] Specifically, this method uses the Pavia University hyperspectral image dataset with a spatial resolution of 1.5 m. This data has 115 bands. After removing uncalibrated bands, noisy bands, and spectrally overlapping bands, 103 bands are left for experiments. The size of this data is 610×340, with a total of 2,207,400 pixels, among which 42,776 pixels carry label information. These pixels contain a total of 9 types of ground objects, including asphalt roads, pastures, gravel, trees, painted metal plates, bare soil, asphalt, bricks, shadows, etc. Randomly select 10 labeled samples in each category as training data, and the remaining data as test data. Since there is only one hyperspectral image in each dataset, the image is cropped into sub-images with a size of 128×128. Each sub-image is centered on a labeled sample to ensure that the classification loss of cross-entropy can be calculated.

[0088] In this training, the cut-out sub-images are randomly divided into training batches with the same number of images. Each training batch consists of 8 images, and data augmentation is performed on the sub-images of each training batch to improve the richness of the data, specifically including random horizontal flipping and random vertical flipping. Train one training batch each time, and when all training batches are trained, it is the end of one round of iteration. Repeat the above operations until the upper limit of the iteration rounds is reached to obtain the weight parameters of the final network training and achieve the training of the small-sample hyperspectral image semantic segmentation network. In this method, the upper limit of the iteration rounds is 50 times.

[0089] The training process of one batch is as follows: Use the forward propagation algorithm and backward propagation algorithm of the neural network to train the above-mentioned semantic segmentation network parameters. Calculate the loss function corresponding to each training batch through forward propagation, and the gradient corresponding to this training batch will be obtained through backward propagation. Use the cross-entropy loss function and the manifold constraint loss function to calculate the total loss L = L ce +αL mr ,L ce is the cross-entropy loss function, and L mr is the manifold constraint loss function; the parameter α is used to balance the manifold constraint loss function, and the value is preset to 0.0001. Use the adaptive moment estimation gradient descent algorithm to minimize the loss function, and then update the network weights. In the second stage, fix the network weights of the basic feature module, and use the adversarial loss function to calculate the total loss L = L ce +αL mr -βL adv ,L adv is the adversarial loss function, and the parameter β is used to balance the adversarial loss function, and the value is preset to 1. Use the adaptive moment estimation gradient descent algorithm to minimize the loss function, and then update the network weights of the two classifiers. In the third stage, fix the network weights of the classifier and use the adversarial loss function to calculate the total loss L = Ladv , L adv For the adversarial loss function, the adaptive moment estimation gradient descent algorithm is used to minimize the loss function, and then the network weights of the basic feature module are updated. The learning rate is preset to 0.01 and decreased in the form of exponential decay. After updating the network training parameters, the updated network parameters are used as the initial values for the next training batch.

[0090] After preprocessing the hyperspectral image to be segmented, it is input into the trained hyperspectral remote sensing image semantic segmentation model to obtain the land cover class to which each pixel in the hyperspectral remote sensing image to be segmented belongs.

[0091] The small-sample hyperspectral remote sensing image semantic segmentation algorithm proposed by this invention can be applied to vegetation detection, urban development planning, environmental science management and other fields.

Claims

1. A semantic segmentation method for small-sample hyperspectral remote sensing images based on multiple classifiers, characterized by: S1 builds a semantic segmentation network for small-sample hyperspectral images, which includes a basic feature extraction module, a multi-classification module, and a loss function; The basic feature extraction module is used to extract basic features from the input hyperspectral image and output the basic features of the image. The basic feature extraction module consists of four layers, each of which consists of a 3×3 convolutional layer, a BN layer and a ReLU layer. A maximum pooling layer is added only after the second layer to increase the receptive field size of the network and avoid excessive downsampling. The multi-classifier module includes two classifiers, each of which is composed of a 1×1 convolutional layer; after the basic feature extraction module, the features are up-sampled by bilinear interpolation before being input into the classifier so that the output feature size is the same as the input data block size; The loss function includes a cross entropy loss function, an adversarial loss function, and a manifold constraint regularization loss function; Among them, the adversarial loss function is defined as: Among them, the two classifiers are respectively denoted as G c1 and G c2 , basic feature module G f , p1 and p2 represent G c1 and G c2 The prediction results, p 1 c and p 2 c Indicates that the sample belongs to C The probability of the class, S represents the softmax layer; The manifold constraint regularization constraint function is expressed as: Where W is the graph adjacency matrix, L = DW is the graph Laplacian matrix, D is the diagonal matrix, X is the s is the sample selected from the input image block, N Indicates the number of selected samples, Tr Indicates the rank, and the adjacency matrix W is defined as: Where σ is the heat kernel parameter, for each pixel x i , use Euclidean distance to select the nearest neighbor x j , W ij Represents the similarity between them, minimizes the manifold constraint regularization loss function, and constrains p i and p j It is also similar; The overall loss function is expressed as: in, L ce is the cross entropy loss function, L mr is the manifold-constrained regularization, L adv is the adversarial loss function, X S Represents random sample points selected from labeled samples and unlabeled samples, a small amount of labeled input data X L , data labeling information Y L , a large number of unlabeled samples X U , α Used to balance the manifold constraint loss function, β Used to balance the adversarial loss function; S2 uses a small sample of hyperspectral remote sensing images taken by drones or satellites as a training set, iteratively trains the semantic segmentation network, and obtains a semantic segmentation model; S3 pre-sets the corresponding number of categories, inputs the hyperspectral remote sensing image to be detected into the trained semantic segmentation model, and obtains the category to which each pixel in the hyperspectral remote sensing image to be classified belongs.

2. The multi-classifier-based semantic segmentation method for small-sample hyperspectral remote sensing images according to claim 1, characterized in that: The basic feature extraction module downsamples the feature map to 1 / 2.

3. The multi-classifier-based semantic segmentation method for small-sample hyperspectral remote sensing images according to claim 1, characterized in that: The semantic segmentation network training steps include: ① Minimize the cross entropy loss function and manifold constraint loss. The training goal is that both classifiers can correctly classify labeled samples, and the predictions of labeled and unlabeled data are subject to manifold constraints. The objective function is: ② Maximize the adversarial loss function, fix the network parameters of the basic feature extraction module, and train the differences between the two classifiers. At the same time, they can correctly classify the labeled data and satisfy the manifold constraints. The objective function is: ③ Minimize the adversarial loss function, fix the parameters of the two classifiers, and train the basic feature extraction module to minimize the prediction difference. The objective function is: Minimizing the prediction difference is equivalent to maximizing the prediction consistency. Since adversarial learning requires the initial classifier to correctly classify the labeled samples, the network is trained in step ① for the first 20 epochs, and then the three steps of iterative training are repeated until the network converges. Finally, the two classification features output by the two classifiers are calculated by the softmax function to obtain the classification probability matrix: H×W×C , the average value of the category number dimension is taken to obtain the classification probability of each pixel corresponding to the category, and the maximum probability is taken as the category to which the pixel belongs as the final judgment category of this system.

4. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

5. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images