Efficient hyperspectral image classification method based on deep fusion of multi-scale space and spectral features
Through the deep fusion method of multi-scale spatial and spectral features, combined with principal component analysis and support vector machine, the problem of insufficient feature extraction and generalization capabilities in hyperspectral image classification is solved, efficient feature extraction and precise classification are achieved, and the Computational complexity and dependency on labeled data.
Patent Information
- Application Number
- CN202411286357.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2025-01-03
AI Technical Summary
The existing hyperspectral image classification methods rely on local information and are susceptible to noise and environmental changes. It is difficult to fully explore the complementarity between spatial features and spectral features. In addition, deep learning methods require a large amount of computing resources and labeled data, and the generalization ability is insufficient.
The deep fusion method of multi-scale spatial and spectral features is adopted to extract the original shallow spectral features through principal component analysis and whitening treatment, and the convolution kernel group size is adaptively selected, and the stochastic patch is used as the 3D convolution kernel to extract the depth multi-scale space and spectra. Features, principal component analysis and fusion are carried out, and classification is carried out in combination with support vector machines.
The generalization ability of the model is improved, and the spatial and spectral characteristics of hyperspectral images can be effectively extracted and characterized, the spatial and spectral characteristics of hyperspectral images are retained to the maximum extent, the computational complexity is reduced, the classification accuracy and robustness are improved, and the dependence on labeled data is reduced.
Smart Images
Figure CN119251557A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of remote sensing image technology, and in particular to a hyperspectral image classification method that is highly efficient and deeply integrates multi-scale spatial and spectral features. Background Art
[0002] Hyperspectral image classification technology is an advanced technology for processing and analyzing hyperspectral data. Its main task is to classify and identify spectral images containing hundreds to thousands of bands. Compared with traditional multispectral images, hyperspectral images have higher spectral resolution, which enables hyperspectral images to provide continuous spectral information on each pixel, thereby capturing rich ground object information. Therefore, hyperspectral image classification technology has shown important value in many application fields, including remote sensing, agricultural environmental monitoring, urban planning, mineral resource exploration, forest fire monitoring, and military reconnaissance. In these application scenarios, the core challenge of hyperspectral image classification technology is how to extract effective feature information from high-dimensional spectral data and accurately classify objects of different categories.
[0003] The process of hyperspectral image classification usually includes several key steps: feature extraction, data dimensionality reduction, classifier design, and post-processing. The purpose of feature extraction is to extract information features that can effectively distinguish different categories from the original hyperspectral data, which is the basis of the entire classification process. Since hyperspectral data usually has very high dimensions, this will lead to high computational complexity and serious data redundancy, so data dimensionality reduction processing is required. Common data dimensionality reduction methods include principal component analysis (PCA), linear discriminant analysis (LDA), and local linear embedding (LLE). These methods reduce the redundancy and complexity of data by projecting high-dimensional data into low-dimensional space, while retaining the most useful information features for classification tasks. Classifier design involves selecting a suitable algorithm or model to map the extracted features to specific categories. Commonly used classification methods include support vector machine (SVM), multinomial logistic regression (MLR), k-nearest neighbor algorithm (KNN), and sparse representation classifier (SRC). These methods achieve the classification of hyperspectral images by constructing discriminant models in feature space. Finally, the post-processing step is used to optimize and enhance the classification results to improve the accuracy and robustness of the classification.
[0004] In the prior art, in the development of hyperspectral image classification technology, early research mainly relied on spectral information for feature extraction and classification. Since hyperspectral images can provide detailed spectral information on each pixel, these spectral information can reflect the reflection characteristics of objects at different wavelengths, thus effectively distinguishing different materials and types of objects. However, methods that rely solely on spectral information for classification often face some challenges, such as spectral aliasing and hyperspectral similarity. These problems will lead to a decrease in classification accuracy, especially in complex scenes, where the spectral characteristics of different objects may be very similar and difficult to distinguish through spectral information.
[0005] To overcome these problems, researchers have gradually realized the importance of deep fusion methods that combine spatial information with spectral information. Spatial information provides additional information about the distribution and shape of objects, which can help reduce the impact of spectral similarity on classification results, thereby improving classification accuracy and robustness. By comprehensively utilizing spectral information and spatial information, deep fusion methods can more accurately classify objects in complex scenes and improve classification performance.
[0006] In addition, due to the high-dimensional characteristics of hyperspectral data, the problems of data redundancy and computational complexity are particularly prominent. In order to meet these challenges, dimensionality reduction techniques are widely used in hyperspectral image classification. Common dimensionality reduction techniques include principal component analysis (PCA), linear discriminant analysis (LDA) and local linear embedding (LLE). These methods reduce data redundancy and noise by mapping high-dimensional data to a low-dimensional space, thereby improving classification efficiency and accuracy. In terms of classifier design, traditional machine learning methods, such as support vector machine (SVM), multinomial logistic regression (MLR), k-nearest neighbor algorithm (KNN) and sparse representation classifier (SRC), are still widely used in hyperspectral image classification. These methods can effectively classify hyperspectral images by constructing discriminant models in feature space.
[0007] In recent years, the rapid development of deep learning technology has provided new methods and ideas for hyperspectral image classification. Deep learning models, especially convolutional neural networks (CNNs), graph neural networks (GNNs) and recurrent neural networks (RNNs), have been successfully introduced into the hyperspectral image classification task. These deep learning models can automatically learn complex feature representations and effectively combine spectral information and spatial information, thereby significantly improving the accuracy and robustness of classification. Convolutional neural networks can extract local spatial features through multi-layer convolution operations, which are suitable for capturing local spatial structures in spectral data; graph neural networks can capture more complex spectral and spatial relationships by modeling non-Euclidean structures in data; and recurrent neural networks are good at processing sequence data and can effectively utilize the continuity characteristics of spectral information. By applying these deep learning techniques to hyperspectral image classification, researchers can more effectively solve the spectral aliasing and high dimensionality problems encountered in traditional methods, and further improve the classification performance and robustness of the algorithm.
[0008] However, traditional spatial and spectral hyperspectral image classification methods mainly rely on the extraction of local information, which makes them susceptible to noise and environmental changes. In addition, such methods are difficult to fully exploit the complementarity between spatial and spectral features, resulting in insufficient generalization capabilities, especially when facing classification tasks in different scenes.
[0009] Although existing deep learning classification methods can extract deeper features, these features are often difficult to directly associate with the semantic information in the image. In addition, most existing deep learning methods rely on pre-training and fine-tuning stages, which not only require a lot of computing resources and time costs, but also have a high demand for large-scale annotated data. In practical applications, it is often difficult to obtain a sufficient number of annotated samples to train deep models.
[0010] To this end, we introduce an efficient hyperspectral image classification method with deep fusion of multi-scale spatial and spectral features. Summary of the invention
[0011] The purpose of the present invention is to provide an efficient hyperspectral image classification method with deep fusion of multi-scale spatial and spectral features to solve the problems raised in the above background technology.
[0012] To achieve the above object, the present invention provides the following technical solution: an efficient hyperspectral image classification method with deep fusion of multi-scale spatial and spectral features, comprising the following steps:
[0013] S1: Perform principal component analysis (PCA) and whitening on the input high-dimensional spectral remote sensing data to reduce the data dimension and extract the original shallow spectral features;
[0014] S2: Adaptively select the size of the convolution kernel group according to the size of the dataset to ensure the appropriateness and effectiveness of feature extraction;
[0015] S3: By extracting random patches of different scales from the original shallow spectrum, 3D convolution kernels are obtained without training, and convolved with image bands respectively to extract deep multi-scale spatial and spectral features;
[0016] S4: The generated feature map is subjected to principal component analysis and whitening again to further optimize the feature representation, and the processed feature map is re-input into step S3 for recursive operation. The feature extraction network adopts an efficient parallel cascade architecture, in which each branch focuses on extracting features of a specific scale.
[0017] S5: The deep spatial and spectral features extracted at each scale and level are fused through principal component analysis. At the same time, the original spectral features are also processed by principal component analysis to extract shallow spectral features. Finally, the multi-scale deep spatial and spectral features are deeply fused with the original shallow spectral features to make full use of the complementarity of spatial and spectral information.
[0018] S6: Use support vector machine (SVM) to classify the deeply fused features to obtain the final classification results.
[0019] Preferably, the input hyperspectral remote sensing image is dimensionality reduced and preprocessed through principal component analysis and whitening operations to reduce redundant information and extract original shallow spectral features. Secondly, by introducing large kernel convolutions of different sizes and random blocks as 3D convolution kernel strategies, the extraction of multi-scale spatial and spectral features of the image can be achieved.
[0020] Preferably, the size of the data set is adaptively selected by selecting a large convolution kernel. For a data set of size r×c, the smaller of the length and width is selected as the benchmark:
[0021] x=min(r,c)
[0022] The three convolution kernels in the convolution kernel group are respectively and It not only ensures that the convolution kernel is larger, but also can extract features of different spatial scales, maximally retain the spatial information in the hyperspectral image, and thus better identify complex spatial patterns.
[0023] Preferably, the 3D convolution kernels are divided into: large convolution kernels and small convolution kernels, the large convolution kernel branch mainly extracts spatial features, while the smaller convolution kernel is more inclined to extract spectral diversity features, three parallel levels of different scales:
[0024] In the first level, a random function is used to randomly select k pixels from the extracted original shallow spectral features, and a patch of size w1×w1×p is selected around each pixel, where w1 is the smaller of w1, w2, and w3, in order to extract more detailed local features to obtain k random blocks: The pixels at the edge are filled by mirroring the image, and then k random blocks are used as 3D convolution kernels. The preprocessed data is convolved with the convolution kernel to obtain k feature maps:
[0025]
[0026] F(M)=max(0,MT);
[0027] T=[t2,...,t2]∈H rc×k ;
[0028] The features of the first layer can be expressed as Z (1) =F(M)∈R rc×k In order to extract deeper features, the feature map of the first layer is processed by PCA and whitening and regarded as the new input H. Similar extraction operations are continued on the first layer. Finally, we can get the feature set of the first level: Z1 = Z (1) ,Z (2) ,...,Z (n) .
[0029] Preferably, M i ∈R r×c is the feature map mapping of the i-th dimension, is the whitened data of the jth dimension, * represents the two-dimensional convolution operator, is the jth dimension of the i-th random block;
[0030] where t2∈R rc×1 represents the average vector of M in the second dimension, and T is the average matrix composed of k repetitions of t2.
[0031] Preferably, in the second level, the whitened data H whitening Repeat the operation of the first level, but this time the patch size w2 is selected to be larger than w1 in order to focus on a wider range of spatial features. The second-level feature set Z2 can be obtained by extracting random patches and performing convolution operations.
[0032] Preferably, in the third level, the patch size w3 is adjusted to capture feature information in a wider range. After the convolution operation, the third-level feature sets Z3 of different scales are obtained. Through the features extracted at different scales, three feature sets Z = {Z1, Z2, Z3} of different scales are obtained. By using 3D convolution kernels of different sizes at different levels, the constructed multi-scale random patch network can extract and characterize the spatial and spectral features of the image in all directions, thereby showing significant advantages in improving the performance, robustness and generalization ability of the model.
[0033] Preferably, the spatial and spectral feature sets of three different scales are denoted as Z PCA , Z PCA Compared with the original shallow spectral features H based on principal component analysis PCA Perform deep fusion to comprehensively utilize the advantages of spectral and spatial information.
[0034] Compared with the prior art, the beneficial effect of the present invention is that the present invention flexibly selects three 3D convolution kernels of different scales according to the size of the data set, thereby improving the generalization ability of the model. This strategy can effectively extract features of different spatial scales, maximize the retention of spatial and spectral information in hyperspectral images, and more accurately identify complex spatial patterns. At the same time, the present invention uses a random function to randomly extract pixel blocks from the image as 3D convolution kernels, and convolve with each band of the image without pre-training and fine-tuning. It combines the features of shallow convolution and deep convolution, and can effectively extract multi-scale and rich spatial and spectral features. In addition, the present invention also fuses spatial and spectral features of different scales and levels through principal component analysis, and deeply fuses them with the original shallow spectral features obtained by principal component analysis, thereby making full use of the comprehensive advantages of spectral and spatial information. These innovative technical means enable the present invention to have significant advantages in feature extraction and recognition of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a schematic diagram of the overall network structure of the present invention;
[0036] Figure 2 Schematic diagram of random block extraction operation of the present invention. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] See also Figure 1-2 The present invention provides a technical solution: an efficient hyperspectral image classification method with deep fusion of multi-scale spatial and spectral features, comprising the following steps:
[0039] S1: Perform principal component analysis (PCA) and whitening on the input high-dimensional spectral remote sensing data to reduce the data dimension and extract the original shallow spectral features;
[0040] S2: Adaptively select the size of the convolution kernel group according to the size of the dataset to ensure the appropriateness and effectiveness of feature extraction;
[0041] S3: By extracting random patches of different scales from the original shallow spectrum, 3D convolution kernels are obtained without training, and convolved with image bands respectively to extract deep multi-scale spatial and spectral features;
[0042] S4: The generated feature map is subjected to principal component analysis and whitening again to further optimize the feature representation, and the processed feature map is re-input into step S3 for recursive operation. The feature extraction network adopts an efficient parallel cascade architecture, in which each branch focuses on extracting features of a specific scale.
[0043] S5: The deep spatial and spectral features extracted at each scale and level are fused through principal component analysis. At the same time, the original spectral features are also processed by principal component analysis to extract shallow spectral features. Finally, the multi-scale deep spatial and spectral features are deeply fused with the original shallow spectral features to make full use of the complementarity of spatial and spectral information.
[0044] S6: Use support vector machine (SVM) to classify the deeply fused features to obtain the final classification results.
[0045] First, the input hyperspectral remote sensing image is reduced in dimension and preprocessed through principal component analysis and whitening operations to reduce redundant information and extract the original shallow spectral features; secondly, we introduce large kernel convolutions of different sizes and random blocks as 3D convolution kernel strategies to achieve the extraction of multi-scale spatial and spectral features of the image. Inspired by the latest research on Visual Transformer (Vits), we believe that using a small number of large convolution kernels is a more powerful paradigm. Therefore, we adaptively select large convolution kernels according to the size of the dataset. Specifically, for a dataset of size r×c, we choose the smaller of the length and width as the benchmark:
[0046] x=min(r,c)
[0047] Then the three convolution kernels in the convolution kernel group are taken and This method not only ensures that the convolution kernel is larger, but also can extract features of different spatial scales, maximally retain the spatial information in the hyperspectral image, and thus better identify complex spatial patterns.
[0048] The feature extraction network adopts an efficient parallel cascade architecture, in which each branch focuses on extracting features at a specific scale. Specifically, the large convolution kernel branch mainly extracts spatial features, while the smaller convolution kernel is more inclined to extract spectral diversity features.
[0049] The entire feature extraction network is divided into three parallel layers of different scales. In the first layer, we randomly select k pixels from the extracted original shallow spectral features using a random function, and select a patch of size w1×w1×p around each pixel (w1 is the smallest of w1, w2, and w3, aiming to extract more detailed local features) to obtain k random blocks. Process Figure 2 As shown. The pixels at the edge are filled by mirroring the image. Then these k random blocks are used as 3D convolution kernels, and the preprocessed data is convolved with the convolution kernel to obtain k feature maps:
[0050]
[0051] Among them, M i ∈R r×c is the feature map mapping of the i-th dimension, is the whitened data of the jth dimension, * represents the two-dimensional convolution operator, is the jth dimension of the ith random block.
[0052] F(M)=max(0,MT)
[0053] T=[t2,...,t2]∈H rc×k
[0054] where t2∈R rc×1 represents the average vector of M in the second dimension, and T is the average matrix composed of k repetitions of t2. Finally, the features of the first layer can be expressed as Z (1) =F(M)∈R rc×k In order to extract deeper features, we treat the feature map of the first layer as the new input H after PCA and whitening, and continue to perform similar extraction operations on the first layer. In this way, we can finally get the feature set of the first level:
[0055] Z1=Z (1) ,Z (2) ,...,Z (n)
[0056] In the second level, the whitened data Hwhitening Repeat the first level operation, but this time the patch size w2 is larger than w1 to focus on a wider range of spatial features. The second-level feature set Z2 can be obtained by extracting random patches and performing convolution operations. In the third level, we adjust the patch size w3 to capture a wider range of feature information. After the convolution operation, we get the third-level feature set Z3 of different scales.
[0057] By extracting features at different scales, we obtain three feature sets Z = {Z1, Z2, Z3} at different scales. By using 3D convolution kernels of different sizes at different levels, the constructed multi-scale random patch network can fully extract and characterize the spatial and spectral features of the image, thus showing significant advantages in improving the performance, robustness and generalization ability of the model.
[0058] In the feature fusion stage, we perform principal component analysis on the features extracted from each layer to retain the effective features containing spatial and spectral information. The spatial and spectral feature sets of three scales are denoted as Z PCA , Z PCA Compared with the original shallow spectral features H based on principal component analysis PCA Perform deep fusion to comprehensively utilize the advantages of spectral and spatial information.
[0059] Finally, the support vector machine (SVM) classifier is used to evaluate the performance of the proposed method. As a supervised learning model, SVM is widely used in hyperspectral image classification due to its strong classification ability and adaptability to high-dimensional data. The algorithm achieves excellent classification results by finding the optimal hyperplane to distinguish samples of different categories. Therefore, we use the fused feature vector as the input of the SVM classifier to evaluate its classification effect.
[0060] Three 3D convolution kernels of different sizes are flexibly selected according to the size of the data set, thereby improving the generalization ability of the model. This strategy can effectively extract features of different spatial scales, maximize the retention of spatial and spectral information in hyperspectral images, and more accurately identify complex spatial patterns. At the same time, the present invention uses a random function to randomly extract pixel blocks from the image as 3D convolution kernels, and convolve with each band of the image without pre-training and fine-tuning. It combines the features of shallow convolution and deep convolution, and can effectively extract multi-scale and rich spatial and spectral features. In addition, the present invention also fuses spatial and spectral features of different scales and levels through principal component analysis, and deeply fuses them with the original shallow spectral features obtained by principal component analysis, thereby making full use of the comprehensive advantages of spectral and spatial information.
[0061] PCA and whitening operations are used on the original hyperspectral data to reduce the data dimension and extract the original shallow spectral features;
[0062] By selecting three 3D convolution kernels of different sizes according to the size of the data set, the generalization ability of the model is improved, and it can extract features of different spatial scales, maximize the retention of spatial and spectral information in hyperspectral images, and thus better identify complex spatial patterns.
[0063] No training is required during the feature extraction process. Pixel blocks are randomly extracted from the original shallow spectral features as 3D convolution kernels using random functions and convolved with each band of the image respectively. This method does not require pre-training and fine-tuning, and combines the features of shallow convolution and deep convolution to extract rich multi-scale spatial features.
[0064] The spatial and spectral features of different scales and levels are fused through principal component analysis and then deeply fused with the original shallow spectral features obtained by principal component analysis, thereby comprehensively utilizing the advantages of spectral and spatial information.
[0065] PCA and whitening operations are used to extract the original shallow spectral features for subsequent feature extraction and fusion.
[0066] An adaptive large convolution kernel group size selection strategy based on the dataset size is designed to improve the generalization ability of the model.
[0067] During the feature extraction process, patches in the original shallow spectrum are directly extracted through random functions as 3D convolution kernels without training, and convolution operations are performed with each band of the image to extract multi-scale deep spatial-spectral features.
[0068] Convolution kernels of different sizes are designed to extract spatial features of different scales respectively, so as to more comprehensively capture and represent the diverse features in hyperspectral images.
[0069] The spatial-spectral features of different scales and levels are fused through principal component analysis, and deeply fused with the original shallow spectral features after principal component analysis, which effectively enhances the model's ability to capture global and local information of the image.
[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An efficient hyperspectral image classification method with deep fusion of multi-scale spatial and spectral features, characterized in that: The following steps are involved: S1: Perform principal component analysis (PCA) and whitening on the input high-dimensional spectral remote sensing data to reduce the data dimension and extract the original shallow spectral features; S2: Adaptively select the size of the convolution kernel group according to the size of the dataset; S3: By extracting random patches of different scales from the original shallow spectrum, 3D convolution kernels are obtained without training, and convolved with image bands respectively to extract deep multi-scale spatial and spectral features; S4: The generated feature map is subjected to principal component analysis and whitening again to further optimize the feature representation, and the processed feature map is re-input into step S3 for recursive operation. The feature extraction network adopts a parallel cascade architecture, in which each branch extracts features of a specific scale; S5: The deep spatial and spectral features extracted at each scale and level are fused through principal component analysis. At the same time, the original spectral features are also processed by principal component analysis to extract shallow spectral features. Finally, the multi-scale deep spatial and spectral features are deeply fused with the original shallow spectral features. S6: Use support vector machine (SVM) to classify the deeply fused features to obtain the final classification results.
2. According to claim 1, an efficient hyperspectral image classification method with deep fusion of multi-scale spatial and spectral features is characterized by: The input hyperspectral remote sensing image is reduced in dimension and preprocessed through principal component analysis and whitening operations, and the original shallow spectral features are extracted.
3. The method for hyperspectral image classification with high efficiency and deep fusion of multi-scale spatial and spectral features according to claim 1 is characterized by: The size of the dataset is adaptively selected with a large convolution kernel. For a dataset of size r×c, the smaller of the length and width is selected as the benchmark: x=min(r,c) The three convolution kernels in the convolution kernel group are respectively and 4. The method for hyperspectral image classification with high efficiency and deep fusion of multi-scale spatial and spectral features according to claim 1 is characterized by: The 3D convolution kernels are divided into: large convolution kernels and small convolution kernels. The large convolution kernel branches extract spatial features, and the small convolution kernel extracts spectral diversity features. There are three parallel levels of different scales: In the first level, a random function is used to randomly select k pixels from the extracted original shallow spectral features, and a patch of size w1×w1×p is selected around each pixel, where w1 is the smaller of w1, w2, and w3, to extract more detailed local features and obtain k random blocks: The pixels at the edge are filled by mirroring the image, and then k random blocks are used as 3D convolution kernels. The preprocessed data is convolved with the convolution kernel to obtain k feature maps: T=[t2,...,t2]∈H rc×k ; The first layer feature is represented as Z (1) =F(M)∈R rc×k , the feature map of the first layer is processed by PCA and whitening and regarded as the new input H, and the first layer extraction operation is continued to be performed, and finally the feature set of the first level is obtained: Z1=Z (1) ,Z (2) ,...,Z (n) .
5. The method for highly efficient hyperspectral image classification with deep fusion of multi-scale spatial and spectral features according to claim 4 is characterized by: M i ∈R r×c is the feature map mapping of the i-th dimension, is the whitened data of the jth dimension, * represents the two-dimensional convolution operator, P i (j) is the jth dimension of the i-th random block; t2∈R rc×1 represents the average vector of M in the second dimension, and T is the average matrix composed of k repetitions of t2.
6. The method for highly efficient hyperspectral image classification with deep fusion of multi-scale spatial and spectral features according to claim 4 is characterized by: In the second level, the whitened data H whitening Repeat the operation of the first level, select a patch size w2 that is larger than w1, and obtain the second-level feature set Z2 by extracting random patches and performing convolution operations.
7. The method for highly efficient hyperspectral image classification with deep fusion of multi-scale spatial and spectral features according to claim 4 is characterized by: In the third level, the patch size w3 is adjusted to capture feature information. After the convolution operation, the third-level feature sets Z3 of different scales are obtained. Through the features extracted at different scales, three feature sets Z = {Z1, Z2, Z3} of different scales are obtained.
8. The method for highly efficient hyperspectral image classification with deep fusion of multi-scale spatial and spectral features according to claim 4 is characterized by: The spatial and spectral feature sets of three different scales are denoted as Z PCA , Z PCA Compared with the original shallow spectral features H based on principal component analysis PCA Carry out deep integration.