An image classification method based on multi-scale space spectrum extreme learning machine

Abstract

The application provides an image classification method based on a multiscale space spectrum extreme learning machine, which extracts classification features through hierarchical space spectrum feature extraction and cross-modal fusion to realize accurate classification of hyperspectral remote sensing images. A multiscale space feature extraction module is used to generate a space feature vector, a residual spectrum feature extraction module is used to simulate the receptive field response of the spectrum dimension, the shallow spectrum trend and the deep feature are combined, and a correlation-driven adaptive weighted fusion module is used to fuse the space feature and the spectrum feature, and finally classification is realized. For five hyperspectral remote sensing images with different resolutions, the SS-MSLRF-ELM classification model and other five models are compared in the experiment, and the overall accuracy OA and the Kappa coefficient are the highest among the six models. The experimental results show that the SS-MSLRF-ELM classification model can provide more accurate and stable classification results, and can improve the classification accuracy while considering the classification speed.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral remote sensing image classification, and specifically to an image classification method based on a multi-scale spatial-spectral limit learning machine. Background Technology

[0002] Hyperspectral imagery (HSI) comprehensively presents ground scenes in the form of data cubes through spectral and spatial dimensions. Due to its high spectral resolution, large information content, and integrated image-spectral structure, it is widely used in fields such as ecological environment monitoring, mineral exploration, surface parameter inversion, and target detection and identification. On the other hand, HSI data has a complex structure and high spectral dimension, making it susceptible to interference from data redundancy, noise, mixed pixels, and the Hughes phenomenon, which makes accurate interpretation difficult and limits its application capabilities.

[0003] Classification is one of the important directions for HSI interpretation and application. Machine learning models such as Support Vector Machines, Random Forests, and Markov Conditional Random Fields have been widely used in HSI classification and have achieved good classification results. However, for complex scenarios, the feature information extracted by the above classification models has limited expressive power and poor adaptability, making it difficult to improve classification performance. In traditional neural network models, the weight parameters to be learned generally grow exponentially with the increase of nodes and layers. Using the back propagation (BP) model to adjust the weight parameters requires a lot of computation, making it far less efficient in terms of training time than machine learning models.

[0004] Huang et al. proposed the Extreme Learning Machine (ELM). This method can randomly generate input weights and hidden layer biases to improve learning speed, and determines output weights through generalized inverse. This method not only has good generalization performance but also has a faster computation speed compared to gradient-based backpropagation (BP) models, and is widely used in classification, prediction, and regression. To directly utilize ELM for image processing, Huang et al. proposed a Local Receptive Field based ELM (ELM-LRF). ELM-LRF uses random convolutional nodes and pooling structures to learn local and translation-invariant features and analyzes and calculates output weights, providing a simple deterministic solution with high training efficiency. However, ELM-LRF uses a fixed-size local receptive field, lacks multi-scale feature extraction capabilities, focuses on spatial local correlations, and lacks a spectral feature extraction mechanism, thus having certain limitations in HSI classification tasks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an image classification method based on a multi-scale spatial-spectral limit learning machine. This model uses spatial and spectral branches to extract HSI spatial and spectral features respectively. After fusing the spatial and spectral features, the output weights are calculated and the model is classified.

[0006] An image classification method based on a multi-scale spatial-spectral limit learning machine specifically includes the following steps: Step 1: Dimensionality reduction using PCA (Principal Component Analysis) PCA dimensionality reduction is performed on the input data to obtain a unified input unit that simultaneously contains spatial neighborhood structure information and pixel spectral information.

[0007] Step 2: Divide the network input units The dimensionality-reduced data is divided into several spatial neighborhood image patches. Each pixel is centered, and its spatial size is selected as... d × d The neighborhood of the image is used as a single input image block.

[0008] Step 3: Multi-scale receptive field spatial feature extraction Step 3.1: Orthogonalize the convolution kernel The convolution kernels are randomly generated from a standard Gaussian distribution and orthogonalized using singular value decomposition (SVD).

[0009] Step 3.2: Parallel Convolution with Multi-Scale Spatial Receptive Field Convolutional operations are performed in parallel for receptive fields of 3×3, 5×5, and 7×7 scales. After generating multi-scale convolutional feature maps, deep fusion of multi-scale features is achieved through feature map concatenation operations, generating 16 feature maps for each scale receptive field, for a total of 48 feature maps.

[0010] Step 3.3: Parallel processing of multiple pooling strategies The concatenated feature map is processed in parallel using three strategies: square / square root pooling, global average pooling, and global max pooling. The outputs of the three pooling strategies are concatenated along the channel dimension to form the pooled spatial feature vector.

[0011] Step 4: Residual-enhanced spectral receptive field feature extraction; Step 4.1: Select the spectral vector as input The spectral dimension is feature-mapped using a fully connected network with fixed weights, with the input being local neighborhood blocks. x patch C×d×d The spectral vector of the center pixel, where CThis refers to the spectral dimension after dimensionality reduction. Its spectral vector is selected as input, covering the spectral information of all bands after dimensionality reduction.

[0012] Step 4.2: Perform feature transformation on the spectral vectors within the spectral receptive field using a residual fully connected network. All fully connected layer weights are orthogonally initialized to ensure the integrity of information during feature transformation, and the weights are fixed and do not participate in training. The first layer mapping reduces the spectral dimension. C The input is mapped to a 64-dimensional hidden space, and nonlinearity is introduced by ReLU (Rectified Linear Unit) activation. The second layer of mapping further transforms the 64-dimensional features nonlinearly while maintaining the dimensionality. The third layer performs a linear transformation to adjust the feature space distribution. Then, the output of the third layer is fused with the residual of the output of the first layer to enhance feature propagation.

[0013] Step 5: Feature classification; Step 5.1: Construct an adaptive weight allocation mechanism based on feature correlation spatial features f spatial With spectral characteristics f spec L2 norm normalization was performed separately to eliminate scale differences. Then, for each sample... i Calculate the cosine similarity between its spatial features and spectral features, by... N The average similarity of each sample is used to obtain a global relevance measure. Based on The physical meaning of the value is determined, a piecewise weight function is designed, and finally, a fusion feature is generated by weighted summation.

[0014] Step 5.2: Output weight calculation and classification After extracting the fusion features from the model, an image classification model based on multi-scale spatial-spectral extreme learning machine (SS-MSLRF-ELM) was used for classification. The output weight matrix was calculated using the least squares method, ultimately yielding the category of the test sample.

[0015] Beneficial technical effects of the present invention: This invention effectively captures the spatial feature correlations at different scales by employing a multi-scale spatial receptive field module (integrating 3×3, 5×5, and 7×7 pixel receptive fields) and enhanced spatial pooling (fusing square / square root pooling, adaptive average pooling, and max pooling). Simultaneously, it leverages residual connections and a fully connected architecture to deeply mine spectral feature sequence attributes. Furthermore, an L2 norm normalization mechanism is introduced to balance the dimensions of spatial and spectral features, and a correlation-driven adaptive weighted fusion module is used to fuse spatial and spectral features, ultimately achieving classification. Experimental results show that the proposed SS-MSLRF-ELM classification model provides more accurate and stable classification results, while simultaneously improving classification accuracy and speed. Attached Figure Description

[0016] Figure 1 Five HSI false-color composite images and standard classification images from embodiments of the present invention; Figure 2 Five HSI-related parameters from embodiments of the present invention; Figure 3 Embodiment 5 of the present invention sets the size of HSI input image blocks. Figure 4 Classification accuracy of SS-MSLRF-ELM on 5 HSI images with different image block sizes in embodiments of the present invention; Figure 5 The impact of different regularization coefficients C on the classification accuracy of SS-MSLRF-ELM in embodiments of the present invention; Figure 6 The embodiments of this invention are based on ablation experiment results using overall classification accuracy (OA); Figure 7 The embodiments of this invention are based on ablation experiment results using the Kappa coefficient; Figure 8 Classification results of embodiments of the present invention; Figure 9 The classification accuracy of Pavia University in this invention embodiment; Figure 10 The classification accuracy of Indian Pines in this invention embodiment; Figure 11 The Salinas classification accuracy in the embodiments of the present invention; Figure 12 The classification accuracy of Xuzhou in this embodiment of the invention; Figure 13 LongKou classification accuracy in embodiments of the present invention; Figure 14 Training time for different models in embodiments of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments; To address the problem that traditional extreme learning machines (ELMs) cannot simultaneously utilize spatial and spectral information in hyperspectral remote sensing image classification, this invention proposes an image classification method based on a multi-scale spatial-spectral extreme learning machine. The core idea is to extract spatial-spectral features hierarchically and use a cross-modal fusion strategy to generate classification features to achieve accurate classification of hyperspectral remote sensing images. The method utilizes a multi-scale spatial feature extraction module to generate spatial feature vectors and a residual spectral feature extraction module to simulate the receptive field response of the spectral dimension. It combines shallow spectral trends with deep features and employs a correlation-driven adaptive weighted fusion module to fuse spatial and spectral features, finally achieving classification. The specific steps include: Step 1: PCA dimensionality reduction HSI of the test sample Dimensionality reduction by performing PCA ,in, M It is the height of HSI. N It is the width of HSI. B It is the spectral dimension of the original HSI. C It refers to the spectral dimension after dimensionality reduction, in this embodiment. C=10 .

[0018] Step 2: Divide the network input units The dimensionality-reduced data is divided into several spatial neighborhood image patches. Each pixel is centered, and its spatial size is selected as... d × d neighborhood x patch As a single input image block.

[0019] Step 3: Multi-scale receptive field spatial feature extraction Step 3.1: Orthogonalize the convolution kernel Singular value decomposition is used to orthogonalize the convolution kernel. Orthogonalization allows the network to extract more complete features than non-orthogonal features.

[0020] 1) Randomly generate initial weights and orthogonalize them. To obtain a sufficient representation of the input, the following method is used. K Different input weights are used to obtain K Given distinct feature maps, an initial weight matrix is ​​randomly generated. , , r Indicates the spatial size of the local receptive field. It is the first k The initial weight vector is The k List.

[0021] 2) Use SVD on the initial weight matrix Perform orthogonalization. The orthogonalized matrix is ​​denoted as... Each of its columns is An orthogonal basis. When r 2 < K In this case, it is necessary to complete the process of transposing, orthogonalizing, and transposing again.

[0022] Step 3.2: Parallel Convolution with Multi-Scale Spatial Receptive Field After generating multi-scale convolutional feature maps, deep fusion of multi-scale features is achieved through feature map concatenation. Each receptive field uses 16 convolutional kernels, resulting in a total of 48 convolutional kernels, consistent with the original algorithm. The feature matrix after convolution is as follows:

[0023] in, f t To utilize t × t The feature matrix generated after feature extraction from the receptive field. This represents a basic unit in the characteristic matrix, signifying a specific eigenvalue. i and j These are the coordinates of this eigenvalue in the spatial dimension. i This represents the row index on the feature map. j Represents column index, p It is the index of the feature value in the channel dimension. express t × t The number of feature maps generated by the receptive field. f 3. f 5 and f 7. The three elements are concatenated along the channel dimension to form a feature matrix that integrates multi-scale information:

[0024] The number of feature maps is the sum of the number of feature maps generated by the three receptive field sizes, which is 48, while the spatial size remains unchanged. It preserves the spatial structure of local neighborhood blocks and integrates multi-scale spatial features from fine-grained to global, providing input for enhanced pooling.

[0025] Step 3.3: Parallel processing of multiple pooling strategies The concatenated feature matrix is ​​processed in parallel using three strategies: square root pooling, global average pooling, and global max pooling.

[0026] Step 4: Residual-enhanced spectral receptive field feature extraction; Step 4.1: Select the spectral vector as input Let the spatial coordinate range of the neighboring block be [0, ... d -1]×[0, d -1], then the coordinates of the center pixel are Select its spectral vector x spec The input is:

[0027] Step 4.2: Perform feature transformation on the spectral vectors within the spectral receptive field using a residual fully connected network. Next, a three-layer fully connected network with residual connections is used to perform feature transformation on the spectral vectors within the spectral receptive field. The first layer maps the 10-dimensional input to a 64-dimensional hidden space, and introduces nonlinearity through ReLU activation, i.e.:

[0028] in, W 1 represents the first-layer weight matrix. b 1 represents the bias term, ReLU = max(0, z This is used to filter negative responses and enhance significant spectral features. The second layer mapping further performs a nonlinear transformation on the 64-dimensional features while maintaining dimensionality invariance, i.e.:

[0029] in, W 2 represents the weight matrix of the second layer. b 2 represents the bias term, which enhances the capture of spectral details, such as subtle fluctuations in reflectance with wavelength. The third layer performs a linear transformation to adjust the feature space distribution as follows:

[0030] in, W 3 represents the third-layer weight matrix. b 3 is the bias term, which captures subtle spectral patterns that were not fully extracted in the first two layers. Then, the output of the third layer is fused with the residual output of the first layer to enhance feature propagation.

[0031] Step 5: Feature classification; Step 5.1: Construct an adaptive weight allocation mechanism based on feature correlation An adaptive weight allocation mechanism based on feature correlation is constructed, which dynamically adjusts the fusion weights by quantifying the global consistency between spatial features and spectral features. First, spatial features With spectral characteristics Perform L2 norm normalization separately to eliminate scale differences, i.e.:

[0032] in, This is the Euclidean norm. After normalization, the feature vectors are projected onto the unit hypersphere, highlighting their directional information and laying the foundation for subsequent similarity calculations. Its value is as follows:

[0033] Then for each sample i The cosine similarity between its spatial features and spectral features is calculated as follows:

[0034] in, This represents the angle between two feature vectors in the feature space. This value quantifies the degree of consistency between the spatial and spectral features of a single sample. By analyzing... N The average similarity of each sample yields the global relevance measure as follows:

[0035] The value reflects the overall degree of coordination between spatial and spectral features in the dataset, revealing the essential characteristic distribution of the dataset. Based on The physical meaning of the value, and the design of the piecewise weight function are as follows:

[0036] The final fused features are generated through a weighted summation, i.e.:

[0037] This fusion method not only preserves the directional information of the features, but also optimizes the discriminative structure of the feature space through adaptive weights.

[0038] Step 5.2: Output weight calculation and classification After extracting the fusion features from the model, SS-MSLRF-ELM is used for classification. The objective function is to minimize the training error and the output weight norm, i.e.:

[0039] in, H The hidden layer output matrix is ​​for fusing features. T This is the label matrix. The output weight matrix is ​​calculated using the least squares method, i.e.:

[0040] Finally, the category of the test sample is obtained.

[0041] like Figures 1 to 14As shown, to verify the feasibility and effectiveness of the proposed SS-MSLRF-ELM, five HSI datasets from Pavia University, Indian Pines, Salinas, Xuzhou, and LongKou were selected for systematic experimental verification. The evaluation was conducted from four aspects: experimental parameters, ablation experiments, experimental results, and training time. On the five HSI datasets from Pavia University, Indian Pines, Salinas, Xuzhou, and LongKou, SS-MSLRF-ELM, compared to ELM, ELM-LRF, 1D-CNN, 2D-CNN, and 3D-CNN, can better suppress salt-and-pepper noise and can also effectively identify and maintain relatively complete contours for difficult-to-segment categories, demonstrating superior classification accuracy and robustness. Furthermore, leveraging the characteristics of the ELM framework, it shows a significant advantage in training time efficiency compared to deep CNN models, achieving an effective balance between classification performance and computational efficiency.

Claims

1. An image classification method based on a multi-scale spatial-spectral limiting learning machine, characterized in that, Includes the following steps: Step 1: PCA dimensionality reduction HSI of the test sample PCA dimensionality reduction yields ,in, M It is the height of HSI. N It is the width of HSI. B It is the spectral dimension of the original HSI. C It is the spectral dimension after dimensionality reduction; Step 2: Divide the network input units The dimensionality-reduced data is divided into several spatial neighborhood image patches. Each pixel is centered, and its spatial size is selected as... d × d neighborhood x patch As a single input image block; Step 3: Multi-scale receptive field spatial feature extraction Step 3.1: Orthogonalize the convolution kernel Singular value decomposition is used to orthogonalize the convolution kernel; orthogonalization allows the network to extract more complete features than non-orthogonal features; Step 3.2: Parallel Convolution with Multi-Scale Spatial Receptive Field Convolutional operations on receptive fields of 3×3, 5×5 and 7×7 scales are performed in parallel. After generating multi-scale convolutional feature maps, deep fusion of multi-scale features is achieved through feature map concatenation operations. 16 feature maps are generated for each receptive field, for a total of 48 feature maps. Step 3.3: Parallel processing of multiple pooling strategies The concatenated feature map is processed in parallel using three strategies: square / square root pooling, global average pooling, and global max pooling. The outputs of the three pooling strategies are concatenated along the channel dimension to form the pooled spatial feature vector. Step 4: Residual-enhanced spectral receptive field feature extraction; Step 4.1: Select the spectral vector as input The spectral dimension is feature-mapped using a fully connected network with fixed weights, with the input being local neighborhood blocks. x patch C ×d×d The spectral vector of the center pixel; its spectral vector is selected as input to cover the spectral information of all bands after dimensionality reduction; Step 4.2: Perform feature transformation on the spectral vectors within the spectral receptive field using a residual fully connected network. All fully connected layer weights are orthogonally initialized to ensure the integrity of information during feature transformation, and the weights are fixed and do not participate in training; the first layer mapping reduces the spectral dimension. C The input is mapped to a 64-dimensional hidden space, and nonlinearity is introduced through ReLU activation; the second layer of mapping further transforms the 64-dimensional features nonlinearly while maintaining the dimension; the third layer performs a linear transformation to adjust the feature space distribution, and then the output of the third layer is fused with the output residual of the first layer to enhance feature propagation; Step 5: Feature classification; Step 5.1: Construct an adaptive weight allocation mechanism based on feature correlation spatial features f spatial With spectral characteristics f spec L2 norm normalization was performed separately to eliminate scale differences; then, for each sample i Calculate the cosine similarity between its spatial features and spectral features, by... N The average similarity of each sample is used to obtain a global relevance measure; based on The physical meaning of the value is determined, a piecewise weight function is designed, and finally, a fusion feature is generated by weighted summation. Step 5.2: Output weight calculation and classification Classification is performed based on fused features, and the output weight matrix is ​​calculated through closed-form solution to finally obtain the category of the test sample.

2. The image classification method based on a multi-scale spatial-spectral limit learning machine according to claim 1, characterized in that, Step 3, the orthogonalization of convolution kernels in the multi-scale receptive field spatial feature extraction, specifically includes: Singular value decomposition is used to orthogonalize the convolution kernel; orthogonalization allows the network to extract more complete features than non-orthogonal features; 1) Randomly generate initial weights and orthogonalize them; to obtain a sufficient representation of the input, the following is adopted: K Different input weights are used to obtain K Given distinct feature maps, an initial weight matrix is ​​randomly generated. , , r Indicates the spatial size of the local receptive field. It is the first k The initial weight vector is The k List; 2) Use SVD on the initial weight matrix Perform orthogonalization; the orthogonalized matrix is ​​denoted as... Each of its columns is orthogonal basis; when r 2 < K In this case, it is necessary to complete the process of transposing, orthogonalizing, and transposing again.

3. The image classification method based on a multi-scale spatial-spectral limit learning machine according to claim 1, characterized in that, Step 3, multi-scale spatial receptive field spatial feature extraction, specifically includes multi-scale spatial receptive field parallel convolution, which includes: After generating multi-scale convolutional feature maps, deep fusion of multi-scale features is achieved through feature map concatenation. Sixteen convolutional kernels are used for each receptive field, for a total of 48 kernels. The feature matrix after convolution is as follows: ， in, f t To utilize t × t The feature matrix generated after feature extraction from the receptive field This represents a basic unit in the characteristic matrix, signifying a specific eigenvalue. i and j These are the coordinates of this eigenvalue in the spatial dimension. i This represents the row index on the feature map. j Represents column index, p It is the index of the feature value in the channel dimension. express t × t The number of feature maps generated by the receptive field; f 3. f 5 and f 7. The three elements are concatenated along the channel dimension to form a feature matrix that integrates multi-scale information: The number of feature maps is the sum of the number of feature maps generated by the three receptive field sizes, which is 48, while the spatial size remains unchanged. It preserves the spatial structure of local neighborhood blocks and integrates multi-scale spatial features from fine-grained to global, providing input for enhanced pooling.

4. The image classification method based on a multi-scale spatial-spectral limit learning machine according to claim 1, characterized in that, Step 3, multi-scale receptive field spatial feature extraction, involves parallel processing of multiple pooling strategies, specifically including: The concatenated feature matrix is ​​processed in parallel using three strategies: square root pooling, global average pooling, and global max pooling. 。 5. The image classification method based on a multi-scale spatial-spectral limit learning machine according to claim 1, characterized in that, Step 5, feature classification, involves constructing an adaptive weight allocation mechanism based on feature correlation, which specifically includes: An adaptive weight allocation mechanism based on feature correlation is constructed, which dynamically adjusts the fusion weights by quantifying the global consistency between spatial features and spectral features. First, spatial features With spectral characteristics Perform L2 norm normalization separately to eliminate scale differences, i.e.: ,in, Here, is the Euclidean norm; after normalization, the eigenvectors are projected onto the unit hypersphere, highlighting their directional information and laying the foundation for subsequent similarity calculations. Their values ​​are as follows: Then for each sample i The cosine similarity between its spatial features and spectral features is calculated as follows: ,in, This represents the angle between two feature vectors in the feature space; this value quantifies the degree of consistency between the spatial and spectral features of a single sample. Spatiotemporal-spectral high coordination, Spatiotemporal and spectral aspects are independent. There is a conflict between spatiotemporal and spectral dimensions; through analysis of... N The average similarity of each sample yields the global relevance measure as follows: , The value reflects the overall degree of coordination between spatial and spectral features in the dataset. The time period represents a high degree of consistency in spatial spectral density (dominated by pure ground features). The time signature indicates strong spatial spectral complementarity, resulting in mixed pixel phenomena. This index reveals the essential distribution characteristics of the dataset; based on The physical meaning of the value is explained below, and the piecewise weight function is designed as follows: .

Images