Hyperspectral image classification method and system based on double-branch lightweight algorithm

Abstract

The invention discloses a hyperspectral image classification method and system based on a double-branch lightweight algorithm, belongs to the technical field of computer vision and image processing, and solves the problems that an existing image classification model is complex in structure, so that the calculated amount is large, the occupied memory is high, and balance between classification precision and model lightweight is difficult to achieve. The method comprises the steps of collecting a hyperspectral image, preprocessing the hyperspectral image, performing iterative training on a double-branch lightweight algorithm model based on a training set, executing the double-branch lightweight algorithm model, and classifying a test set by the double-branch lightweight algorithm model. According to the hyperspectral image classification method, the algorithm complexity is greatly reduced, the high precision of hyperspectral image classification is ensured, the lightweight space-spectrum Transformer is introduced into the double-branch lightweight algorithm model, and through the optimization design of the whole structure and the lightweight efficient Attention-Aware mechanism, the classification accuracy of the hyperspectral image is improved. The complexity of the model is reduced, and the characteristic relation between the modeling space and the spectrum sequence can be better established.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and particularly relates to a hyperspectral image classification method and system based on a dual-branch lightweight algorithm. Background Technique

[0002] Hyperspectral images are obtained through hyperspectral imaging technology, which records the reflection characteristics of surface substances in different spectral bands, forming a three-dimensional data cube containing rich spectral information. The classification mechanism is mainly based on the differences in the spectral characteristics of substances, and the recognition of ground object categories is realized by analyzing spectral curves. Hyperspectral image classification has wide applications in fields such as agricultural monitoring, environmental protection, and military reconnaissance, and is crucial for precision agriculture management, resource exploration, and target detection. Therefore, in-depth research on hyperspectral image classification technology is of great significance.

[0003] Currently, convolutional neural networks (CNNs) and Transformers are two mainstream technologies in hyperspectral image classification. CNNs extract spatial features through local convolutional operations and achieve efficient classification by combining pooling and fully connected layers; while Transformers use self-attention mechanisms to capture global spectral-spatial dependencies and have strong capabilities in modeling long-range feature relationships. These two technologies have their own advantages and jointly promote the development of hyperspectral image classification technology.

[0004] However, CNNs and Transformers also have obvious drawbacks in hyperspectral image classification. They usually have numerous parameters and complex model structures, resulting in large computational amounts and high memory occupancy, and are difficult to be deployed in environments with limited resources. At the same time, the pursuit of high accuracy often comes with the expansion of the model scale, making it difficult to achieve a balance between classification accuracy and model lightweight. To address the above problems, we propose a hyperspectral image classification method and system based on a dual-branch lightweight algorithm. Summary of the Invention

[0005] The purpose of the present invention is to provide a hyperspectral image classification method and system based on a dual-branch lightweight algorithm for the deficiencies of the existing technology, and solve the problems that the existing image classification model has a complex structure, resulting in large computational amounts, high memory occupancy, being difficult to be deployed in environments with limited resources, and it is difficult to achieve a balance between classification accuracy and model lightweight.

[0006] The present invention is implemented as follows. A hyperspectral image classification method based on a dual-branch lightweight algorithm, the hyperspectral image classification method based on a dual-branch lightweight algorithm includes:

[0007] Collect hyperspectral images and preprocess the hyperspectral images;

[0008] Load the preprocessed hyperspectral image. Taking the sample pixels in the hyperspectral image as the center points, extract samples from the hyperspectral image to obtain a modeling sample set. When extracting samples from the hyperspectral image, use the window step size S as the side length unit to extract s a sample cube of s×L as a sample;

[0009] Obtain the modeling sample set, and divide the modeling sample set into a training set, a validation set, and a test set according to a ratio;

[0010] Pre-build a lightweight algorithm model with a dual-branch structure. Based on the training set, iteratively train the dual-branch lightweight algorithm model, calculate the loss function, optimize the parameters of the dual-branch lightweight algorithm model, and evaluate and optimize the dual-branch lightweight algorithm model based on the validation set, and output the converged dual-branch lightweight algorithm model;

[0011] Taking the test set as the input, execute the dual-branch lightweight algorithm model, and the dual-branch lightweight algorithm model classifies the test set.

[0012] Preferably, the method for preprocessing the hyperspectral image includes:

[0013] Use a hyperspectral camera to collect the hyperspectral image, calibrate the hyperspectral image, and eliminate the differences in the hyperspectral images obtained at different time periods and with different sensor parameters;

[0014] Based on the linear regression empirical method, eliminate the influence of emissivity caused by different factors, and achieve the correction of the hyperspectral image;

[0015] Load the corrected hyperspectral image, use wavelet transform to denoise the hyperspectral image, and at the same time use image enhancement technology to enhance the hyperspectral image;

[0016] Segment and label the hyperspectral image, segment the hyperspectral image into several sub-regions, and label the pixels.

[0017] Preferably, the dual-branch lightweight algorithm model is based on a lightweight dual-branch network architecture, integrates the convolutional neural network CNN and the Transformer model, improves the Transformer model, introduces the ResidualBottleneck Block to replace the multi-layer perceptron, and introduces a lightweight self-attention mechanism. The lightweight self-attention mechanism is used to efficiently and accurately capture the intricate correlations and subtle differences between spectra, achieve in-depth and refined modeling of the high-dimensional spectral space features of HSI data, and fuse the gated recurrent unit Gate Recurrent Unit with the encoder module of the LSS Transformer to obtain the CNN branch and the Transformer branch.

[0018] Preferably, the method for classifying the test set by the lightweight algorithm model with dual branches specifically includes:

[0019] Obtain the test set, and use a sliding window of size s×s to split the original HSI data X in the test set to obtain N small blocks of size s×s×L;

[0020] Perform an equal division of the sample tensor P in the channel dimension to generate two complementary sub-tensors X′ and X″, and respectively input them into the Transformer branch and the CNN branch. The Transformer branch and the CNN branch respectively perform in-depth and detailed feature extraction on the tensors X′ and X″;

[0021] Load the sub-tensor X′, and the Transformer branch performs feature extraction on the sub-tensor X′ to obtain the feature matrix Y Trfm ;

[0022] Load the sub-tensor X″, and the CNN branch performs feature extraction on the sub-tensor X″ to obtain the feature matrix Y cnn ;

[0023] Obtain the feature matrix Y cnn and the feature matrix Y Trfm Merge the feature matrix Y cnn and the feature matrix Y Trfm Input the merged feature matrix Y cnn and the feature matrix Y Trfm into the multi-layer perceptron MLP of the lightweight algorithm model with dual branches. The multi-layer perceptron MLP classifies the feature matrix Y cnn and the feature matrix Y Trfm and outputs the image classification result.

[0024] Preferably, the method for the Transformer branch to perform feature extraction on the sub-tensor X′ specifically includes:

[0025] Obtain the sub-tensor X′, and the dimension transformation module Dimension Reduction Block in the Transformer branch performs channel dimension compression on the sub-tensor X′, and the compressed sub-tensor X′ is output as X DR ;

[0026] Use the patch embedding module Patch Embedding Block to perform spatial flattening on the dimension-reduced feature tensor X DR and convert it into a vector sequence through a linear transformation operation

[0027] Load the vector sequence Perform on the vector sequence Add learnable positional embedding P pos to obtain the embedding sequence X Tin The embedding sequence X Tin is expressed as:

[0028]

[0029] Obtain the embedding sequence X Tin The LSS Transformer encoder performs long-range spectral-spatial modeling on the embedding sequence X Tin to obtain the feature matrix Y Trfm .

[0030] Preferably, the method for the LSS Transformer encoder to perform long-range spectral-spatial modeling on the embedding sequence X Tin specifically includes:

[0031] Load the embedding sequence X Tin The embedding sequence X Tin is input into the LayerNorm layer. After being processed by the LayerNorm layer, the embedding sequence X Tin is transmitted to the Multi-head self-attention unit using a residual connection;

[0032] Among them, after being processed by the LayerNorm layer, the embedding sequence X Tin is transmitted to the Multi-head self-attention unit using a residual connection, including:

[0033] Perform linear mapping processing on the embedding sequence X Tin to obtain Q, A, K, V through linear mapping, where A represents the proxy token obtained through pooling; First, A is used as a query to perform attention calculation with the key K and value V to aggregate the proxy feature V A ; Subsequently, using A as the key, perform attention calculation again to transfer the global information of the proxy feature to each query token, thereby obtaining the output O of the first stage sa1 ;

[0034] Concatenate the input feature O sa1 with the query vector Q; Subsequently, perform a channel shuffle operation and divide the result into two parts, X1 and X2. The two parts are respectively processed by a linear layer. On the main branch, a sigmoid function is used to evaluate the correlation of the hyperspectral sequence features;

[0035] X1 and X2 are sent to the Residual Bottleneck Block after passing through the LayerNorm layer.

[0036] Preferably, the method for the CNN branch to extract features from the sub-tensor X″ specifically includes:

[0037] Obtain the sub-tensor X″, perform a channel shuffle operation on the sub-tensor X″, and divide the sub-tensor X″ into three parts: X″1, X2″, and X″′3;

[0038] Perform 1×1 convolution calculation on X″1, and perform batch normalization and PReLUde operations;

[0039] Then perform a channel shuffle operation on X″1, split it into two paths, perform atrous convolution and depthwise separable convolution respectively, and then sum them element-wise to obtain Y1.

[0040] Overlay Y1 with the original X2″, and then perform batch normalization and PReLUde operations again;

[0041] Perform spectral and spatial local feature modeling through the ECA attention module. Among them, the ECA module internally includes a Global Pooling module, a convolution module, a sigmoid module, and uses residual connection for output.

[0042] On the other hand, the present invention also provides a hyperspectral image classification system based on a dual-branch lightweight algorithm. The hyperspectral image classification system based on the dual-branch lightweight algorithm specifically includes:

[0043] An image acquisition module, which is used to acquire hyperspectral images and preprocess the hyperspectral images;

[0044] A sample extraction module, which is used to load the preprocessed hyperspectral images, take the sample pixels in the hyperspectral images as the center points, extract samples from the hyperspectral images to obtain a modeling sample set. Among them, when extracting samples from the hyperspectral images, a sample cube of s×s×L is extracted with the window step S as the side length unit as a sample;

[0045] A model construction module, which is used to obtain the modeling sample set, divide the modeling sample set into a training set, a validation set, and a test set according to a ratio, pre-construct a lightweight algorithm model with a dual-branch, iteratively train the dual-branch lightweight algorithm model based on the training set, calculate the loss function, optimize the parameters of the dual-branch lightweight algorithm model, and evaluate and optimize the dual-branch lightweight algorithm model based on the validation set, and output a converged dual-branch lightweight algorithm model;

[0046] An image classification module, which is used to take the test set as the input, execute the dual-branch lightweight algorithm model, and the dual-branch lightweight algorithm model classifies the test set.

[0047] Preferably, the image acquisition module includes:

[0048] The image calibration unit uses a hyperspectral camera to collect hyperspectral images and calibrates the hyperspectral images to eliminate the differences in hyperspectral images obtained at different time periods and with different sensor parameters.

[0049] The image correction unit eliminates the influence of emissivity caused by different factors based on the linear regression empirical method to achieve the correction of hyperspectral images.

[0050] The image enhancement unit is used to load the corrected hyperspectral images, denoise the hyperspectral images using wavelet transform, and simultaneously enhance the hyperspectral images using image enhancement techniques.

[0051] The image segmentation unit is used to segment and label the hyperspectral images, segment the hyperspectral images into several sub-regions, and label the pixels.

[0052] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

[0053] The present invention not only greatly reduces the algorithm complexity, but also ensures the high accuracy of hyperspectral image classification, and has good portability, facilitating deployment and application on embedded devices or mobile devices. Moreover, the dual-branch lightweight algorithm model introduces a lightweight spatial-spectral Transformer (LSS Transfomer). Through the optimized design of the entire structure and the lightweight and efficient Attention-Aware mechanism, the model can better model the feature relationship between spatial and spectral sequences while reducing the complexity.

[0054] In the present invention, the Gated Recurrent Unit network architecture is incorporated into the LSS Transformer encoder. By tightly coupling the GRU module with the encoder module, the encoder can capture the global dependency relationship while more carefully capturing the temporal dependency of spectral features, thus significantly improving the feature extraction ability and classification accuracy of the algorithm.

[0055] The present invention can be applied to many different fields, such as precision agriculture, food safety, environmental detection, mineral exploration, urban planning, biomedical imaging, military reconnaissance and many other fields, with broad application prospects and great potential for commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the implementation process of the hyperspectral image classification method based on the dual-branch lightweight algorithm provided by the present invention.

[0057] Figure 2 It shows the architecture diagram of the dual-branch lightweight algorithm model.

[0058] Figure 3Shows the architecture diagram of the lightweight spectral enhancement self-attention algorithm in the Multi-head self-attention unit.

[0059] Figure 4 Shows the architecture diagram of the CNN branch in the lightweight algorithm model with a dual-branch structure.

[0060] Figure 5 Shows the classification comparison result diagram of the lightweight algorithm model with a dual-branch structure of the present invention and seven other methods on the Salinas dataset.

[0061] Figure 6 Shows the classification comparison result diagram of the lightweight algorithm model with a dual-branch structure of the present invention and seven other methods on the Indian Pines dataset. Detailed implementation manners

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above accompanying drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above accompanying drawings are used to distinguish different objects and not to describe a specific order.

[0063] Referring to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The phrase appears at various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0064] The existing image classification models have complex structures, resulting in large computational amounts and high memory occupancy, making it difficult to be deployed in environments with limited resources. Moreover, it is difficult to achieve a balance between classification accuracy and model lightweighting. To address the above problems, we propose a hyperspectral image classification method and system based on a dual-branch lightweight algorithm. The method includes collecting hyperspectral images, preprocessing the hyperspectral images, taking the sample pixels in the hyperspectral images as the center points, extracting samples from the hyperspectral images to obtain a modeling sample set, iteratively training the lightweight algorithm model of the dual-branch based on the training set, executing the lightweight algorithm model of the dual-branch, and the lightweight algorithm model of the dual-branch classifying the test set. The present invention not only greatly reduces the algorithm complexity, but also ensures high accuracy in hyperspectral image classification, and has good portability, facilitating deployment and application on embedded devices or mobile devices. Moreover, the lightweight algorithm model of the dual-branch introduces a lightweight spatial-spectral Transformer (LSSTransformer). Through the optimized design of the entire structure and the lightweight and efficient Attention-Aware mechanism, the model can better model the feature relationships of spatial and spectral sequences while reducing the complexity.

[0065] An embodiment of the present invention provides a hyperspectral image classification method based on a dual-branch lightweight algorithm. Figure 1 The schematic diagram of the implementation process of the hyperspectral image classification method based on the dual-branch lightweight algorithm is shown. The hyperspectral image classification method based on the dual-branch lightweight algorithm specifically includes:

[0066] Step S10, collect hyperspectral images and preprocess the hyperspectral images;

[0067] Step S20, load the preprocessed hyperspectral images, take the sample pixels in the hyperspectral images as the center points, extract samples from the hyperspectral images to obtain a modeling sample set. Among them, when extracting samples from the hyperspectral images, with the window step size S as the side length unit, extract s a sample cube of ×s×L as a sample;

[0068] Step S30, obtain the modeling sample set, and divide the modeling sample set into a training set, a validation set, and a test set according to a ratio;

[0069] Step S40, pre-construct a lightweight algorithm model of the dual-branch, iteratively train the lightweight algorithm model of the dual-branch based on the training set, calculate the loss function, optimize the parameters of the lightweight algorithm model of the dual-branch, and evaluate and optimize the lightweight algorithm model of the dual-branch based on the validation set, and output a converged lightweight algorithm model of the dual-branch;

[0070] Step S50, take the test set as the input, execute the lightweight algorithm model of the dual-branch, and the lightweight algorithm model of the dual-branch classifies the test set.

[0071] The present invention not only significantly reduces the algorithm complexity, but also ensures high accuracy in hyperspectral image classification, and has good portability, facilitating deployment and application on embedded devices or mobile devices. Moreover, the lightweight algorithm model with dual branches introduces a lightweight spatial-spectral Transformer (LSS Transfomer). Through the optimized design of the entire structure and the lightweight and efficient Attention-Aware mechanism, the model can better model the feature relationship between spatial and spectral sequences while reducing complexity.

[0072] The embodiment of the present invention provides a method for preprocessing hyperspectral images. The method for preprocessing hyperspectral images specifically includes:

[0073] Step S101: Use a hyperspectral camera to collect hyperspectral images, calibrate the hyperspectral images, and eliminate the differences in hyperspectral images obtained at different time periods and with different sensor parameters.

[0074] Step S102: Based on the linear regression empirical method, eliminate the influence of emissivity caused by different factors to achieve the correction of hyperspectral images.

[0075] Step S103: Load the corrected hyperspectral images, use wavelet transform to denoise the hyperspectral images, and simultaneously use image enhancement technology to enhance the hyperspectral images.

[0076] Step S104: Segment and label the hyperspectral images, segment the hyperspectral images into several sub-regions, and label the pixels.

[0077] In this embodiment, the lightweight algorithm model with dual branches is based on a lightweight dual-branch network architecture, integrating the convolutional neural network CNN and the Transformer model. The Transformer model is improved by introducing ResidualBottleneck Block to replace the multi-layer perceptron, and a lightweight self-attention mechanism is introduced. The lightweight self-attention mechanism is used to efficiently and accurately capture the intricate correlations and subtle differences between spectra, realizing deep and refined modeling of the high-dimensional spectral space features of HSI data. The gated recurrent unit Gate Recurrent Unit is fused with the encoder module of LSS Transformer to obtain the CNN branch and the Transformer branch.

[0078] Figure 2The figure shows the architecture diagram of a lightweight algorithm model with two branches. The lightweight algorithm model with two branches combines the unique advantages of convolutional neural networks (CNNs) in fine local feature extraction and the extraordinary capabilities of Transformers in global context information modeling. The core of this lightweight algorithm is the design of a novel lightweight spatio-spectral Transformer (LSS Transformer) structure. We have deeply improved the traditional Transformer architecture by introducing ResidualBottleneck Block to replace the conventional multi-layer perceptron (MLP). This not only reduces the burden of model parameters but also enhances the hierarchy and depth of feature representation, improving the model's ability to process complex information. At the same time, in view of the unique properties of hyperspectral images, a brand-new lightweight self-attention mechanism is designed. This mechanism fully considers the unique spectral characteristics of hyperspectral images and can efficiently and accurately capture the intricate correlations and subtle differences between spectra, achieving in-depth and refined modeling of the high-dimensional spectral space features of HSI data and greatly enhancing the feature extraction and analysis capabilities. In addition, to further strengthen the model's ability to capture the temporal dependence of spectral features, we have innovatively integrated the gated recurrent unit (GRU) with the encoder module of the LSS Transformer. This design not only endows the model with the ability to process time series data but also significantly improves its in-depth understanding and expression ability of spectral feature sequences, while effectively improving the convergence speed of the algorithm. Through this coupling mechanism, this lightweight algorithm can more accurately grasp the spatio-temporal dynamic characteristics in HSI data, laying a solid foundation for accurate classification.

[0079] The embodiment of the present invention provides a method for classifying a test set by a lightweight algorithm model with two branches. The method for classifying a test set by the lightweight algorithm model with two branches specifically includes:

[0080] Step S201, obtain a test set, and split the original HSI data X in the test set using a sliding window of size s×s to obtain N small blocks of size s×s×L;

[0081] Among them, the sample feature extraction formula is as follows:

[0082] N = (H - s + 1)(H - s + 1)

[0083] Step S202, perform an equal division of the sample tensor P in the channel dimension to generate two complementary sub-tensors X′ and X″, where And respectively input them into the Transformer branch and the CNN branch. The Transformer branch and the CNN branch respectively perform in-depth and detailed feature extraction on the tensors X′ and X″;

[0084] Step S203: Load the sub-tensor X′, and the Transformer branch extracts features from the sub-tensor X′ to obtain the feature matrix Y Trfm ;

[0085] Step S204: Load the sub-tensor X″, and the CNN branch extracts features from the sub-tensor X″ to obtain the feature matrix Y cnn ;

[0086] Step S205: Obtain the feature matrix Y cnn and the feature matrix Y Trfm Merge the feature matrix Y cnn and the feature matrix Y Trfm Input the merged feature matrix Y cnn and the feature matrix Y Trfm into the multi-layer perceptron MLP of the dual-branch lightweight algorithm model. The multi-layer perceptron MLP classifies the feature matrix Y cnn and the feature matrix Y Trfm and outputs the image classification result.

[0087] In this embodiment, the method for the Transformer branch to extract features from the sub-tensor X′ specifically includes:

[0088] Obtain the sub-tensor X′. The dimension reduction module Dimension Reduction Block in the Transformer branch compresses the channel dimension of the sub-tensor X′, and the compressed sub-tensor X′ is output as X DR ;

[0089] X DR ∈R h×w×d (d ≤ l);

[0090] Use the patch embedding module Patch Embedding Block to perform spatial flattening on the dimension-reduced feature tensor X DR and convert it into a vector sequence through a linear transformation operation

[0091] The vector sequence is expressed as:

[0092]

[0093] Load the vector sequence To accurately retain the spatial position information in the sequence representation of hyperspectral data, add the learnable position embedding P to the vector sequence pos to obtain the embedding sequence X Tin, the embedded sequence X Tin is represented as:

[0094]

[0095] Obtain the embedded sequence X Tin , the LSS Transformer encoder performs long-distance spectral-spatial modeling on the embedded sequence X Tin to obtain the feature matrix Y Trfm .

[0096] In the embodiment of the present invention, in the encoder, the Gated Recurrent Unit (GRU) is innovatively integrated with the encoder module of the LSSTransformer. It not only endows the model with the ability to process time series data, but also significantly improves its in-depth understanding and expression ability of spectral feature sequences, and at the same time effectively improves the convergence speed of the algorithm.

[0097] In the embodiment of the present invention, the Gated Recurrent Unit network architecture is incorporated into the LSS Transformer encoder. By tightly coupling the GRU module with the encoder module, the encoder can capture the temporal dependence of spectral features more carefully while capturing global dependencies, thus significantly improving the feature extraction ability and classification accuracy of the algorithm.

[0098] In the embodiment of the present invention, the method for the LSS Transformer encoder to perform long-distance spectral-spatial modeling on the embedded sequence X Tin specifically includes:

[0099] Load the embedded sequence X Tin , the embedded sequence X Tin is input into the LayerNorm layer. After being processed by the LayerNorm layer, the embedded sequence X Tin is transmitted to the Multi-head self-attention unit using a residual connection;

[0100]

[0101] where Y Tout , X' represent the output and intermediate variables respectively, and LNorm(·), MHSA(·), and RBB(·) represent the LayerNorm(), Multi-Head Self-Attention(), and Residual Bottleneck Block() functions respectively.

[0102] Among them, after being processed by the LayerNorm layer, the embedded sequence X TinTransmitted to the Multi-head self-attention unit, including:

[0103] Perform linear mapping processing on the embedded sequence X Tin to obtain Q, A, K, and V through linear mapping, where A represents the proxy token obtained through pooling; first, A is used as a query to perform attention calculation with key K and value V to aggregate the proxy feature V A ; subsequently, using A as the key, perform attention calculation again to transfer the global information of the proxy feature to each query token, thereby obtaining the output O of the first stage sa1 ;

[0104] Among them, the proxy feature V A has the following calculation formula:

[0105]

[0106] where Soft(·) and φ(·) represent the softmax() and linear() functions respectively.

[0107] Concatenate the input feature O sa1 with the query vector Q; then perform a channel shuffle operation and split the result into two parts, X1 and X2. The two parts are processed through linear layers respectively. On the main branch, use the sigmoid function to evaluate the correlation of the hyperspectral sequence features;

[0108]

[0109] Among them Sig(·) and Cat(·) represent the Linear(), Sigmoid(), and Concatenate() functions respectively.

[0110] After passing through the LayerNorm layer, X1 and X2 are sent into the Residual Bottleneck Block.

[0111] It should be noted that when X1 and X2 are sent into the Residual Bottleneck Block after passing through the LayerNorm layer, the entire process uses a residual connection. We have conducted an innovative exploration, that is, using the Residual Bottleneck Block to replace the traditional MLP layer. The inside of this block uses a depthwise separable convolution with a kernel size of 3x3 to efficiently fuse and refine the spectral feature information with residual and bottleneck operations, and significantly reduce the model parameters, solving the problem of large number of parameters caused by multiple FC layers in the MLP layer.

[0112] It should be noted that the core of the Multi-head self-attention unit is the lightweight spectral enhancement self-attention algorithm. Figure 3 Figure Figure 3 shows the architecture diagram of the lightweight spectral enhancement self-attention algorithm in the Multi-head self-attention unit. This lightweight spectral enhancement self-attention algorithm can be finely divided into two stages, aiming to efficiently process the complex features of hyperspectral images.

[0113] The first stage focuses on global feature modeling, using the softmax function as the basis to capture the initial correlation between features. However, due to the inherent limitations of the softmax function in representing negative correlations in the attention map, this may limit the in-depth understanding of the fine affinity relationship between the spectral-spatial multi-dimensions of hyperspectral images.

[0114] Therefore, we have carefully designed the second stage, which makes full use of the global dependency framework established in the first stage and combines the dynamic characteristics of the query vector. By deeply exploring the interaction between these global dependencies and the query vector, we have achieved a significant enhancement and refined extraction of the spectral feature representation. Specifically, this stage not only strengthens the connection between features within the spectral dimension but also promotes the accurate capture of the cross-affinity between the spectral and spatial dimensions, thus greatly improving the algorithm's ability to analyze the complex spectral characteristics of hyperspectral images.

[0115] The embodiment of the present invention provides a method for the CNN branch to extract features from the sub-tensor X″. The method for the CNN branch to extract features from the sub-tensor X″ specifically includes:

[0116] Step S301, obtain the sub-tensor X″, perform channel cleaning on the sub-tensor X″, and divide the sub-tensor X″ into three parts: X″1, X2″, and X″′3;

[0117] Step S302, perform 1×1 convolution calculation on X″1, and perform batch normalization (BN) and PReLUde operations;

[0118] Step S303, perform channel shuffle operation on X″1, split it into two paths, perform atrous convolution and depthwise separable convolution respectively, and then add them element-wise to obtain Y1;

[0119] Step S304, stack Y1 with the original X2″ and then perform batch normalization (BN) and PReLUde operations again;

[0120] Step S305, perform spectral and spatial local feature modeling through the ECA attention module. Among them, the ECA module internally includes a Global Pooling module, a convolution module, a sigmoid module, and uses residual connection for output.

[0121] The ECA attention mechanism is expressed as:

[0122] ECA(X) = Sig(Cov(M(Gap(X)))) × X

[0123] Step S306, repeat Step S302 - Step S304 to extract the local feature matrix.

[0124] In this embodiment, Figure 4 shows the architecture diagram of the CNN branch in the lightweight algorithm model with a dual - branch structure. By integrating atrous convolution to expand the receptive field and combining depth - separable convolution to achieve efficient feature extraction, it is specifically optimized for the complexity and high - dimensionality of hyperspectral data. In addition, the ECA attention mechanism is innovatively incorporated to adaptively strengthen key spectral features, further enhancing the model's ability to understand the fine structure of hyperspectral images.

[0125] Furthermore, the effectiveness of the method of the present invention is verified on a public dataset. Under the condition of 20 training samples for each category, the method of the present invention is used to classify two public hyperspectral image datasets, Salinas and Indian Pines, and is compared with 7 current representative methods (in the same hardware configuration environment, according to the optimal parameter configuration of each method). The quantitative comparison results are shown in Table 1 below, and the qualitative comparison results are as Figure 5 and Figure 6 shown. Among them, Figure 5 shows the classification comparison result diagram of the lightweight algorithm model with a dual - branch structure of the present invention and 7 other methods on the Salinas dataset. Among them, Figure 5 in, (a) is the classification result diagram of the HybridSN method, (b) is the classification result diagram of the SSFTT method, (c) is the classification result diagram of the MASSFormer method, (d) is the classification result diagram of the LiT method, (e) is the classification result diagram of the ELS2T method, (f) is the classification result diagram of the DCTN method, (g) is the classification result diagram of the DBSSAN method, and (h) is the classification result diagram of the proposedmethod method. And the 7 current representative methods are the HybridSN method, the SSFTT method, the MASSFormer method, the LiT method, the ELS2T method, the DCTN method, and the DBSSAN method respectively. Figure 6 shows the classification comparison result diagram of the lightweight algorithm model with a dual - branch structure of the present invention and 7 other methods on the Indian Pines dataset. Among them, Figure 6Among them, (a) is the classification result diagram of the HybridSN method, (b) is the classification result diagram of the SSFTT method, (c) is the classification result diagram of the MASSFormer method, (d) is the classification result diagram of the LiT method, (e) is the classification result diagram of the ELS2T method, (f) is the classification result diagram of the DCTN method, (g) is the classification result diagram of the DBSSAN method, and (h) is the classification result diagram of the proposedmethod method. The classification results can prove that the network model of the present invention has strong robustness and strong generalization ability, and achieves high classification accuracy under limited training sample conditions.

[0126] Table 1

[0127]

[0128] Furthermore, the parameter quantity and computational complexity of the method of the present invention are verified on the public dataset. As shown in Table 2. Compared with the other seven methods, the total parameter quantity and computational complexity of the algorithm model of the present invention are significantly reduced, fully demonstrating the advantages of its lightweight and high efficiency.

[0129] Table 2

[0130]

[0131] Furthermore, the functionality and effectiveness of each module in the self-supervised algorithm model of the present invention are verified through ablation experiments, and the experimental results are shown in Table 3 below.

[0132] Table 3

[0133]

[0134] On the other hand, the embodiment of the present invention also provides a hyperspectral image classification system based on a dual-branch lightweight algorithm. The hyperspectral image classification system based on the dual-branch lightweight algorithm specifically includes:

[0135] An image acquisition module, configured to acquire a hyperspectral image and preprocess the hyperspectral image;

[0136] A sample extraction module, configured to load the preprocessed hyperspectral image, take the sample pixels in the hyperspectral image as the center points, extract samples from the hyperspectral image to obtain a modeling sample set. Among them, when extracting samples from the hyperspectral image, with the window step S as the side length unit, an s×s×L sample cube is extracted as a sample;

[0137] A model construction module, which is used to obtain a modeling sample set, divide the modeling sample set into a training set, a validation set and a test set according to a ratio, pre-construct a lightweight algorithm model with a dual-branch structure, iteratively train the dual-branch lightweight algorithm model based on the training set, calculate a loss function, optimize the parameters of the dual-branch lightweight algorithm model, and evaluate and optimize the dual-branch lightweight algorithm model based on the validation set, and output a converged dual-branch lightweight algorithm model;

[0138] An image classification module, which is used to take the test set as input, execute the dual-branch lightweight algorithm model, and the dual-branch lightweight algorithm model classifies the test set.

[0139] In this embodiment, the image acquisition module includes:

[0140] An image calibration unit, which uses a hyperspectral camera to collect hyperspectral images, calibrates the hyperspectral images, and eliminates the differences in hyperspectral images obtained at different time periods and with different sensor parameters;

[0141] An image correction unit, which eliminates the influence of emissivity caused by different factors based on a linear regression empirical method to achieve the correction of hyperspectral images;

[0142] An image enhancement unit, which is used to load the corrected hyperspectral images, denoise the hyperspectral images using wavelet transform, and simultaneously perform enhancement processing on the hyperspectral images using image enhancement techniques;

[0143] An image segmentation unit, which is used to segment and label the hyperspectral images, segment the hyperspectral images into several sub-regions, and label the pixels.

[0144] On the other hand, an embodiment of the present invention also provides an electronic device, including a processor and a memory. The processor can be a computer, an embedded device, an FPGA processor, a mobile terminal device, etc., which can deploy and execute the methods described in the above invention content; the memory can be used in cooperation with the above-mentioned processor to provide a storage space for executable instructions.

[0145] In still another aspect of the present invention, a computer-readable storage medium is further provided. Instructions executable by a computer are stored in the medium. When these instructions are called and executed by a processor, they will guide the processor to implement the method steps described in any one of the invention contents.

[0146] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions / modules corresponding to the hyperspectral image classification method based on the dual-branch lightweight algorithm in the embodiments of the present application. The memory may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created for the use of the hyperspectral image classification method based on the dual-branch lightweight algorithm, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely provided relative to the processor, and these remote memories can be connected to the local module through a network. Examples of the above networks include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] Finally, it should be noted that the computer-readable storage medium herein (e.g., the memory) can be volatile memory or non-volatile memory, or can include both volatile memory and non-volatile memory. By way of example and not limitation, non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), and this RAM can serve as an external cache memory. By way of example and not limitation, RAM can be obtained in various forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to include but are not limited to these and other suitable types of memory.

[0148] In summary, the present invention provides a hyperspectral image classification method and system based on a dual-branch lightweight algorithm. The present invention not only greatly reduces the algorithm complexity, but also ensures the high accuracy of hyperspectral image classification, and has good portability, facilitating deployment and application on embedded devices or mobile devices. Moreover, the lightweight algorithm model of the dual-branch introduces a lightweight spatial-spectral Transformer (LSS Transformer). Through the optimized design of the entire structure and the lightweight and efficient Attention-Aware mechanism, the model can better model the feature relationship between the spatial and spectral sequences while reducing the complexity.

[0149] It should be noted that for the foregoing embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, some steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting the protection scope of the invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art can still, without conflict, combine, add, delete or make other adjustments to the features in the embodiments of the present invention according to the circumstances without creative efforts, so as to obtain different technical solutions that do not essentially depart from the concept of the present invention, and these technical solutions also belong to the scope of protection of the present invention.

Claims

1. A hyperspectral image classification method based on a dual-branch lightweight algorithm, characterized in that The hyperspectral image classification method based on the dual-branch lightweight algorithm includes: Collect hyperspectral images and preprocess the hyperspectral images; Load the preprocessed hyperspectral images, take the sample pixels in the hyperspectral images as the center points, extract samples from the hyperspectral images to obtain a modeling sample set. When extracting samples from the hyperspectral images, use the window step size S as the side length unit, and extract an s×s×L sample cube as one sample; Obtain the modeling sample set, and divide the modeling sample set into a training set, a validation set, and a test set according to a ratio; Pre-construct a dual-branch lightweight algorithm model, iteratively train the dual-branch lightweight algorithm model based on the training set, calculate the loss function, optimize the parameters of the dual-branch lightweight algorithm model, and evaluate and optimize the dual-branch lightweight algorithm model based on the validation set, and output a converged dual-branch lightweight algorithm model; Use the test set as the input, execute the dual-branch lightweight algorithm model, and the dual-branch lightweight algorithm model classifies the test set.

2. The hyperspectral image classification method based on a dual-branch lightweight algorithm according to claim 1, wherein: The method for preprocessing the hyperspectral images includes: Collect hyperspectral images using a hyperspectral camera, calibrate the hyperspectral images to eliminate the differences in hyperspectral images obtained at different times and with different sensor parameters; Based on the linear regression empirical method, eliminate the influence of emissivity caused by different factors to achieve the correction of hyperspectral images; Load the corrected hyperspectral images, use wavelet transform to denoise the hyperspectral images, and at the same time use image enhancement technology to enhance the hyperspectral images; Segment and label the hyperspectral images, segment the hyperspectral images into several sub-regions, and label the pixels.

3. The hyperspectral image classification method based on a dual-branch lightweight algorithm according to claim 1, wherein: The dual-branch lightweight algorithm model is based on a lightweight dual-branch network architecture, integrates the convolutional neural network CNN and the Transformer model, improves the Transformer model, introduces the Residual Bottleneck Block to replace the multi-layer perceptron, and introduces a lightweight self-attention mechanism. The lightweight self-attention mechanism is used to efficiently and accurately capture the intricate associations and subtle differences between spectra, realize the deep and refined modeling of the high-dimensional spectral space features of HSI data, and fuse the gated recurrent unit Gate Recurrent Unit with the encoder module of the LSS Transformer to obtain the CNN branch and the Transformer branch.

4. The hyperspectral image classification method based on the dual-branch lightweight algorithm according to claim 3, wherein: The method for the dual-branch lightweight algorithm model to classify the test set specifically includes: Obtain the test set, use a sliding window of size s×s to split the original HSI data X in the test set to obtain N small blocks of size s×s×L; Perform an equal division of the sample tensor P in the channel dimension to generate two complementary sub-tensors X′ and X″, and respectively input them into the Transformer branch and the CNN branch. The Transformer branch and the CNN branch respectively perform deep and detailed feature extraction on the tensors X′ and X″; Load the sub-tensor X′, and the Transformer branch extracts features from the sub-tensor X′ to obtain the feature matrix Y Trfm ; Load the sub-tensor X″, and the CNN branch extracts features from the sub-tensor X″ to obtain the feature matrix Y cnn ; Obtain the feature matrix Y cnn and the feature matrix Y Trfm to merge the feature matrix Y cnn and the feature matrix Y Trfm Then, input the merged feature matrix Y cnn and the feature matrix Y Trfm into the multi-layer perceptron MLP of the double-branch lightweight algorithm model. The multi-layer perceptron MLP classifies the feature matrix Y cnn and the feature matrix Y Trfm and outputs the image classification result.

5. The hyperspectral image classification method based on the dual-branch lightweight algorithm according to claim 4, characterized in that: The method for the Transformer branch to extract features from the sub-tensor X′ specifically includes: Obtain the sub-tensor X′, and the Dimension Reduction Block in the Transformer branch compresses the channel dimension of the sub-tensor X′, and the compressed sub-tensor X′ is output as X DR ; Use the Patch Embedding Block to spatially flatten the feature tensor X after dimensionality reduction DR and convert it into a sequence of vectors through a linear transformation operation Loading vector sequence For the vector sequence Add learnable position embeddings P pos to obtain the embedded sequence X Tin The embedded sequence X Tin is expressed as: Obtain the embedded sequence X Tin , the LSS Transformer encoder performs long-distance spectral-spatial modeling on the embedded sequence X Tin to obtain the feature matrix Y Trfm .

6. The hyperspectral image classification method based on a dual-branch lightweight algorithm according to claim 5, characterized in that: The LSS Transformer encoder performs long-distance spectral-spatial modeling on the embedded sequence X Tin The method for performing long-distance spectral-spatial modeling specifically includes: Load the embedded sequence X Tin , the embedded sequence X Tin is input into the LayerNorm layer. After being processed by the LayerNorm layer, the embedded sequence X Tin is transmitted to the Multi-head self-attention unit using a residual connection; Among them, after the LayerNorm layer processes, the embedded sequence X is transmitted to the Multi-head self-attention unit using a residual connection Tin including: For the embedded sequence X Tin Perform linear mapping processing. Through linear mapping, Q, A, K, and V are obtained, where A represents the proxy token obtained through pooling. First, A is used as a query to perform attention calculation with the key K and the value V to aggregate the proxy feature V A ; Subsequently, using A as the key, perform attention calculation again to transfer the global information of the proxy feature to each query token, thereby obtaining the output O of the first stage sa1 ; Concatenate the input feature O sa1 with the query vector Q; then perform a channel shuffle operation and split the result into two parts, X1 and X2, and process the two parts through linear layers respectively. On the main branch, use the sigmoid function to evaluate the correlation of the hyperspectral sequence features; X1 and X2 are fed into the Residual Bottleneck Block after passing through the LayerNorm layer.

7. The hyperspectral image classification method based on a dual-branch lightweight algorithm according to claim 5, wherein: The method for the CNN branch to extract features from the sub-tensor X″ specifically includes: Obtain the sub-tensor X″, perform a channel shuffle operation on the sub-tensor X″, and divide the sub-tensor X″ into three parts: X″1, X″2, and X3″′; Perform 1×1 convolution calculation on X″1, and perform batch normalization and PReLUde operations; Then perform a channel shuffle operation on X1″, split it into two paths, perform atrous convolution and depthwise separable convolution respectively, and then sum them element-wise to obtain Y1.

8. The hyperspectral image classification method based on the dual-branch lightweight algorithm according to claim 7, characterized in that: The method for the CNN branch to extract features from the sub-tensor X″ specifically further includes: Overlay Y1 with the original X2″, and then perform batch normalization and PReLUde operations again; Perform spectral and spatial local feature modeling through the ECA attention module. Among them, the ECA module internally includes a GlobalPooling module, a convolution module, a sigmoid module, and uses residual connection for output.

9. A hyperspectral image classification system based on a dual-branch lightweight algorithm for implementing the hyperspectral image classification method based on the dual-branch lightweight algorithm according to any one of claims 1-8, characterized in that: The hyperspectral image classification system based on the dual-branch lightweight algorithm specifically includes: An image acquisition module for acquiring hyperspectral images and preprocessing the hyperspectral images; A sample extraction module for loading the preprocessed hyperspectral images, taking the sample pixels in the hyperspectral images as the center points, and extracting samples from the hyperspectral images to obtain a modeling sample set. Among them, when extracting samples from the hyperspectral images, taking the window step size S as the side length unit, and extracting an s×s×L sample cube as a sample; A model construction module for obtaining the modeling sample set, dividing the modeling sample set into a training set, a validation set, and a test set according to a ratio, pre-constructing a dual-branch lightweight algorithm model, iteratively training the dual-branch lightweight algorithm model based on the training set, calculating the loss function, optimizing the parameters of the dual-branch lightweight algorithm model, and evaluating and optimizing the dual-branch lightweight algorithm model based on the validation set, and outputting a converged dual-branch lightweight algorithm model; An image classification module for taking the test set as the input, executing the dual-branch lightweight algorithm model, and the dual-branch lightweight algorithm model classifies the test set.

10. The hyperspectral image classification system based on the dual-branch lightweight algorithm according to claim 9, characterized in that: The image acquisition module includes: An image calibration unit that uses a hyperspectral camera to acquire hyperspectral images, calibrates the hyperspectral images, and eliminates the differences in hyperspectral images obtained with different time periods and different sensor parameters; An image correction unit that eliminates the influence of emissivity caused by different factors based on the linear regression empirical method to achieve the correction of hyperspectral images; An image enhancement unit for loading the corrected hyperspectral images, using wavelet transform to denoise the hyperspectral images, and at the same time using image enhancement technology to enhance the hyperspectral images; An image segmentation unit for segmenting and annotating the hyperspectral images, segmenting the hyperspectral images into several sub-regions, and annotating the pixels.

Images