Hyperspectral image segmentation method based on spectral and spatial sparse feature representation

Abstract

The application discloses a hyperspectral image segmentation method based on spectral and spatial sparse feature representation, constructs a basic network model to preliminarily extract visual features, then obtains multiple groups of spectral features by grouping in the spectral dimension, uses a spectral convolutional neural network in each group of features to obtain the relationship between each group of spectral internals, then uses a sparse Transformer module to interact between the multiple spectral groups to obtain sequence information between the spectrums, then uses a spatial convolution module to extract spatial local features of the hyperspectral image, uses a sparse Transformer module to extract spatial global features of the hyperspectral image, and finally segments the obtained spectral spatial features.

Description

Technical Field

[0001] This invention relates to image processing technology, specifically to a hyperspectral image segmentation method based on spectral and spatially sparse feature representation. Background Technology

[0002] Hyperspectral remote sensing images, due to their continuous imaging of objects across a wide spectral band, possess advantages such as high spectral resolution and rich information content. Therefore, hyperspectral images have been widely used in the field of remote sensing. Hyperspectral image segmentation has important applications in geological exploration, food safety, and environmental monitoring, and is also the foundation for other applications in hyperspectral remote sensing. Its purpose is to classify pixels from continuous imaging across a wide spectral band into different land cover categories according to their semantics.

[0003] In recent years, with the rapid improvement of parallel computing capabilities and the exponential growth of various types of data, deep neural networks, as a powerful feature extraction method, have attracted widespread attention from scholars both domestically and internationally. They have been successfully applied to feature extraction of image, text, speech, and video data, generally achieving better results than traditional methods. Therefore, they have also been introduced into hyperspectral image segmentation. Currently, methods for hyperspectral image segmentation based on deep neural networks include those based on stacked autoencoders (SAE), deep belief networks (DBN), and deep convolutional neural networks (DCNN).

[0004] Zhouhan Lin et al., in their paper "Deep Learning-Based Classification of Hyperspectral Data," used SAE (Synthetic Image Processing) for hyperspectral image segmentation. This method divides the original hyperspectral image into 7×7 blocks, then uses Principal Component Analysis (PCA) to reduce the spectral dimension of each block. The dimensionality-reduced image is then stretched into a vector and used as input to the SAE. The SAE extracts nonlinear features, and finally, a Support Vector Machine (SVM) is used for segmentation. TongLi et al., in their paper "Classification of hyperspectral image based on deep belief networks," used DBN (Deep Belief Networks) for hyperspectral image segmentation. This method reduces the image dimensionality using PCA, divides each pixel into 7×7 blocks, stretches these blocks into vectors, and concatenates them with the original pixel vectors. The resulting long vector is used as input to a Deep Belief Network (DBN), and finally, a Logistic Regression (LR) segmenter is used for segmentation.

[0005] However, both SAE and DBN require one-dimensional vectors as input data. In other words, when using SAE and DBN for hyperspectral image segmentation, the input data must first be converted into a one-dimensional vector form. DCNN can accept two-dimensional image input. Therefore, while preserving pixel spatial information, DCNN has lower input complexity and is thus the most widely used in hyperspectral image segmentation applications.

[0006] In their paper "Deep supervised learning for hyperspectral data classification through convolutional neural networks," Konstantinos Makantasis et al. used DCNN for hyperspectral image segmentation. Their method first reduces the dimensionality of the original hyperspectral image using randomized principal component analysis (R-PCA), then divides the dimensionality-reduced image into blocks, creating training, validation, and test sets. Finally, the training set is used to train the DCNN. While convolutional processing typically deals with dense data, some data, such as point clouds, is extremely sparse. Directly using dense convolutional networks on such sparse data is highly inefficient. Benjamin Graham et al., in their paper "Submanifold Sparse Convolutional Networks," introduced a sparse convolution operation tailored for handling sparse data. Unlike previous work on sparse convolutional networks, this operation operates strictly on submanifolds, rather than extending observations to every layer of the network. In their paper "Explicitsparse transformer: Concentrated attention through explicit selection", Guangxiang Zhao et al. proposed a sparse Transformer architecture for image processing. By introducing sparsity into the Transformer architecture, they reduced memory consumption and computational power through the attention matrix in the sparse Transformer model.

[0007] A major challenge in hyperspectral image classification is extracting the rich and effective information from hyperspectral image data. Most existing methods employ convolutional neural networks to extract and fuse spatial and spectral information from the image, obtaining spatial-spectral information for classification. With the development of Transformers in computer vision, some works have used Transformers to explore the relationships between different spectral bands in hyperspectral images. However, due to the extremely high spectral dimensionality of hyperspectral images, the number of spectra requiring interaction is enormous, and spectra with small differences often contain largely the same information, leading to the extraction of a large amount of redundant information. Summary of the Invention

[0008] Purpose of the Invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a hyperspectral image segmentation method based on spectral and spatial sparse feature representations. From the perspective of information enhancement, this invention utilizes spectral Transformer and spectral convolution modules to extract spectral feature representations on the one hand, and spatial convolution and spatial Transformer modules to extract spatial feature representations on the other hand. Since hyperspectral data contains a large amount of redundant information and the distribution of truly useful data is sparse, this invention employs sparse Transformer interaction to obtain a better feature representation to improve segmentation performance.

[0009] Technical solution: The present invention provides a hyperspectral image segmentation method based on spectral and spatial feature extraction, comprising the following steps:

[0010] Step (1): Input a randomly selected hyperspectral image of size h×w×c to be segmented, where h is the height of the hyperspectral image, w is the width of the hyperspectral image, and c is the spectral dimension of the hyperspectral image; usually, the spectral dimension of a hyperspectral image is between 100 and 1000.

[0011] Step (2): Randomly divide the hyperspectral image data to be segmented into training set, validation set and test set;

[0012] Step (3): Construct a basic network model, using two convolutional neural networks with different parameters as the backbone network. Extract the initial visual features of the hyperspectral image through the convolution operation of the backbone network, as shown in formula (1) below: R=f(H) (1)

[0013] Where H represents the hyperspectral image to be segmented, f(·) represents the convolutional layer, and R represents the initial visual features;

[0014] Step (4): Group the initial visual features obtained in step (3) along the spectral dimension. Extract spectral features from each group using one-dimensional convolution in the spectral dimension. Then, put each group of spectral features into a sparse Transformer for interaction to obtain spectral feature representation.

[0015] Step (5): Divide the spectral feature representation obtained in step (4) into blocks along the spatial dimension. Extract the spatial features of the image by using two-dimensional convolution in the spatial dimension for each block. Then, use a sparse Transformer to perform feature interaction on all spatial features to obtain the final feature map representation.

[0016] Step (6): Compare the final feature map representation obtained in step (5) with the ground truth map to obtain the final segmentation result through multiple iterations of optimization.

[0017] Furthermore, in step (2), when dividing the dataset, linear discriminant analysis is first used to enhance the hyperspectral image data, and samples with different proportions of labels are selected as training samples, validation samples, and test samples.

[0018] Further, in step (4), the spectral Transformer module extracts spectral features. To preserve the local features between each spectrum, the hyperspectral image is first divided into several channel groups according to the spectral dimension. Then, spectral convolution is used to extract local features within each channel group. During grouping, neighboring spectra with similar information are grouped together, and information interaction is performed between groups. After that, each group of spectra is regarded as a whole to avoid interference from redundant information of adjacent spectra and to explore the relationship between different spectral groups. All group features are fused by calculating the attention of each group of local features with the local features of each other group. The specific working process is as follows:

[0019] First, within each channel group, spectral convolution is used to perform feature interactions between groups:

[0020] X=δ(WR+b) (2)

[0021] Where δ(·) is the activation function, and W and b are the parameters of the convolutional neural network;

[0022] Then, a sparse Transformer is used to facilitate interaction between channel groups: first, three linear mapping layers map each feature X to K, Q, and V. K represents the key, used to retrieve the index of information in memory; Q represents the query, used to find the key of information in memory; and V represents the value, used to update the new content of the current information. The attention score is calculated as follows:

[0023] P = QK T (3)

[0024] Unlike the traditional Transformer, the Sparse Transformer reduces memory consumption and computational cost by sparsifying the attention matrix in the Transformer model. After calculating the attention score, the model does not enter the softmax module. Instead, it first selects the k points with the highest scores P and retains them, discarding the other attention values. Then, it performs a softmax operation on these k scores.

[0025]

[0026] Among them, P ij t represents the score in the i-th row and j-th column of the attention matrix. i This represents the k-th largest value in row u of P;

[0027]

[0028]

[0029]

[0030] Where LN(·) represents the column normalization layer, and F represents the extracted spectral features.

[0031] In summary, if we let F represent the final spectral characteristics of the sparse Transformer module after processing by equations (3)-(7), then the sparse Transformer module can be expressed as:

[0032] F=SpareTrasnformer(R) (8).

[0033] Furthermore, in step (5), the extraction of spatial features cannot extract long-distance spatial relationships due to the limitations of the receptive field of the convolutional kernel. Furthermore, objects of the same category in a hyperspectral image are not distributed together, but rather randomly distributed throughout the entire hyperspectral image. Therefore, long-distance spatial information is crucial for the entire image. Specifically, this invention first divides the features obtained in step (4) into 4×4 blocks in the spatial dimension, and uses a convolutional neural network to extract the local spatial information of each feature block. The calculation method is as follows:

[0034]

[0035] Where f(·) represents a convolutional neural network; F is the spectral feature; Represents the obtained spatial local features

[0036] Then, to explore the long-distance spatial relationships of each sample on the graph, each spatial feature is used as a token to calculate a sparse attention matrix in the spatial dimension. The attention matrix is ​​then used to guide the learning of long-distance dependencies between each sample. The calculation method is as follows:

[0037]

[0038] The final feature representation O is obtained for classification. The output features are fed into a softmax classifier to obtain the final prediction result, and the cross-entropy loss is calculated with the ground truth map. After multiple iterations, the final trained model is obtained.

[0039] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0040] (1) The training model constructed in this invention is a novel feature enhancement network that mines hyperspectral image features at multiple levels.

[0041] (2) The present invention utilizes the Dilute Transformer module to facilitate interaction between features, thereby alleviating the problem of high dimensionality and redundant data information in hyperspectral image data.

[0042] (3) This invention uses spectral dimension grouping processing. First, convolution is used to extract information within the group. Then, each group of features is treated as a whole to avoid the calculation of redundant information between adjacent spectra. Finally, sparse Transformer is used to interact and obtain the spectral features of the image.

[0043] (4) When processing spatial dimension blocks, this invention uses convolution to extract the short-range information of each sample in spatial dimension. Since the distribution positions of similar samples in hyperspectral images are different, it also uses sparse Transformer to extract the long-range dependency relationship in spatial dimension. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the overall segmentation process of the present invention;

[0045] Figure 2 This is a diagram of the network model in this invention;

[0046] Figure 3 This is a visualization of the dataset in the example;

[0047] Figure 4 This is a truth graph of the dataset in the example. Detailed Implementation

[0048] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0049] like Figure 1 As shown, a hyperspectral image segmentation method based on spectral and spatial feature extraction according to the present invention includes the following steps:

[0050] Step (1): Input a randomly selected hyperspectral image of size h×w×c to be segmented.

[0051] Step (2): Randomly divide the hyperspectral image data into training set, validation set and test set;

[0052] Step (3): Construct a basic network model and initially extract visual features; We use a simple backbone network to obtain the initial features of the hyperspectral image data;

[0053] Step (4): Group the initial visual features obtained in step (3) along the spectral dimension. Extract spectral features from each group using one-dimensional convolution in the spectral dimension. Then, put each group of spectral features into a sparse Transformer for interaction to obtain spectral feature representation.

[0054] Step (5): Divide the spectral feature representation obtained in step (4) into blocks along the spatial dimension. Extract the spatial features of the image by using two-dimensional convolution in the spatial dimension for each block. Then, use a sparse Transformer to perform feature interaction on all spatial features to obtain the final feature map representation.

[0055] Step (6): Compare the final feature map representation from step (5) with the ground truth map to obtain the final segmentation result through multiple iterations of optimization.

[0056] Example 1:

[0057] This embodiment includes the following steps:

[0058] Step (1): Input a hyperspectral image to be segmented with dimensions of 145×145×200, where 200 is the spectral dimension;

[0059] Step (2): For the hyperspectral image data, randomly divide it into training set, validation set and test set; select 10% of the samples as training set, 1% of the samples as validation set and 89% of the samples as test set.

[0060] Step (3): Construct a basic network model and initially extract visual features; we use a two-layer convolutional neural network to extract initial features from the hyperspectral image data. The dimension of the initial features is...

[0061] 256×256×512;

[0062] Step (4): Divide the initial visual features obtained in step (3) into 16 groups along the spectral dimension. The feature dimension of each group is 256×256×32. Extract spectral features of each group using one-dimensional convolution in the spectral dimension. Then, transform each group of features into K, Q, and V using three linear transformations with different parameters. Calculate the spectral attention matrix P. The dimension of the P matrix is ​​32×32. Select the first 32 attention scores in the attention matrix and keep them unrepresented. Set the remaining attention scores to 0. Obtain the sparsified attention matrix through the softmax function and extract spectral feature representations through sparse attention.

[0063] Step (5): Divide the spectral feature representation in step (4) into 4×4 blocks along the spatial dimension. The feature dimension of each block is 64×64×32. Use two-dimensional convolution to extract the spatial features of the image in the spatial dimension of each block. Then calculate the sparse attention matrix in the spatial dimension. Guide the extraction of spatial features through the sparse attention matrix to obtain the final feature map representation.

[0064] Step (6): Compare the final feature map representation from step (5) with the ground truth map to obtain the final segmentation result through multiple iterations of optimization.

[0065] The dataset used in this experiment is the Indian Pine Dataset, which was collected in 1992 by an Airborne Visible / Infrared Imaging Spectrometer (AVIRIS) sensor in northwestern Indiana, USA. We reduced the label size to 145×145, resulting in a total of 21,025 pixels in our experiment. Note that 10,366 pixels are feature pixels, and the remaining 10,659 pixels are background pixels. There are 16 categories, and the sample distribution between categories is extremely uneven. It contains 220 bands. For the training and testing phases, we only need to use 200 bands by removing 20 interfering bands. Figure (3) shows the visualization of the dataset, and Figure (4) shows the ground truth map of the dataset.

[0066] In this embodiment, the stochastic gradient method is used for 600 iterations during iterative training. The batch size for each iteration is set to 32, and the model learning rate is set to 0.0005. We use Adam as the optimizer. After every ten iterations, the result no longer changes, and the final model is saved for model testing.

Claims

1. A hyperspectral image segmentation method based on spectral and spatial feature extraction, characterized in that: Comprising the following steps: Step (1), inputting an optional one of hyperspectral images to be segmented with a size of , is high of the hyperspectral image, is wide of the hyperspectral image, and c is a dimension of a spectral dimension of the hyperspectral image; Step (2), randomly divide the hyperspectral image data to be segmented into training set, validation set and test set; Step (3), build a basic network model, use two layers of convolutional neural networks with different parameters as the backbone network, and extract the initial visual features of the hyperspectral image through the convolution operation of the backbone network, formula (1) as follows: (1) wherein, represents a hyperspectral image, represents a convolutional layer, represents an initial visual feature; Step (4), group the initial visual features obtained in step (3) along the spectral dimension, and extract spectral features in each group along the spectral dimension using one-dimensional convolution, then put each group of spectral features as a whole into the sparse Transformer for interaction to obtain spectral feature representation, the specific method is: The spectral Transformer module extracts spectral features, first divides the hyperspectral image into several channel groups according to the spectral dimension, then uses spectral convolution to extract local features within each channel group, and then calculates the attention between each group of local features and other groups of local features to fuse all group features, the specific working process is as follows: First, for each channel group, use spectral convolution for feature interaction between groups: (2) wherein is an activation function, and are parameters of a convolutional neural network; Then inter-group interaction is performed for each group of channels by a sparse Transformer module: first, each group of features is mapped by three linear mapping layers to , , represent a key, an index to look up information in memory; represent a query, a key to find information in memory; represent a value, new content to update the current information; then the attention score is calculated: (3) According to the score Choose the highest score We retain this one point, discarding other attention values, and then apply this to... Perform softmax operation on each fraction: (4) wherein, denotes the score of the th row and th column in the attention matrix, denotes the th largest value of the th row, th column. (5) (6) (7) wherein, represents a column normalization layer, represents the extracted spectral features; In summary, the final spectral feature represented after processing by the sparse Transformer module represented by equations (3) - (7) is The sparse Transformer module can then be represented as: (8)； Step (5), divide the spectral feature representation obtained in step (4) into blocks along the spatial dimension, and extract spatial features of the image in each block along the spatial dimension using two-dimensional convolution, then perform feature interaction on all spatial features through the sparse Transformer to obtain the final feature map representation; Step (6), the final feature map representation obtained in step (5) is loss with the true value map, and the final segmentation result is obtained through multiple iterations of optimization.

2. The method of claim 1, wherein the method is based on spectral and spatial feature extraction. When dividing the data set in step (2), linear discriminant analysis is first used to enhance the hyperspectral image data, and samples with different proportions are selected as training samples, validation samples and test samples. 3.The hyperspectral image segmentation method based on spectral and spatial feature extraction of claim 1, wherein: The specific working process of extracting spatial features in step (5) is as follows: First, use spatial convolution to extract spatial local features of the image: (9) wherein, represents a convolutional neural network; refers to spectral features; represents the obtained spatial local features; Then, the obtained features are divided into blocks along the spatial dimension, and for each feature after blocking, the spatial features are extracted using the sparse Transformer: (10) Further, the final spatial feature representation is obtained ; The segmentation loss function used in the training stage is cross-entropy loss.

Images