Small sample hyperspectral image classification method and device based on spatial-spectral feature fusion

Abstract

The invention relates to a small sample hyperspectral image classification method and device based on spatial-spectral feature fusion. The method comprises the following steps: processing HSI input data to obtain a data feature map IDres and data features; performing two rounds of spatial-spectral feature extraction and fusion of different granularities on the data feature map IDres by adopting two spatial-spectral feature extraction modules to obtain fine fusion data; and carrying out weight compensation and strong screening on the fine fusion data by adopting two-way feature fusion Mamba to obtain a result feature map. And according to the result feature map, constructing a compact long sequence spatial spectrum dependency relationship by adopting convolution operation, and performing feature compensation by adopting data features Ires to obtain refined fusion data, and performing image classification according to the refined fusion data to obtain a hyperspectral image classification result. According to the method, a multi-stage feature fusion strategy of'coarse fusion-fine optimization-strong screening 'is adopted, so that efficient classification under the condition of few samples is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image classification, and relates to a small sample hyperspectral image classification method and device based on spectral feature fusion. BACKGROUND

[0002] As an advanced remote sensing method, hyperspectral imagery (HSI) can provide high-resolution spatial information and fine spectral characteristics simultaneously by means of hundreds of continuous narrow bands. This technology can reveal the essence of objects and their evolution trends by remotely acquiring electromagnetic wave information. In the fields of earth observation, urban remote sensing, agricultural remote sensing and environmental remote sensing, HSI has shown great application potential. At present, the research on ground feature classification based on spatial pixels and continuous narrow-band spectral features has become an important driving force for the development of hyperspectral remote sensing technology.

[0003] For the HSI image classification task, the core of the research lies in how to effectively extract and utilize the key information in the image, namely the spatial features and the spectral signals. Spatial features reflect the spatial distribution and texture structure of pixels in the image, while spectral features contain the spectral information of each pixel point, reflecting the physical and chemical properties of matter.

[0004] Since spatial features and spectral signals have different characteristics, simply fusing the two cannot fully utilize the advantages of spectral and spatial features. Therefore, how to effectively fuse spatial features and spectral features to improve classification accuracy is a hot and difficult point in current research. Existing research results have made certain achievements in this regard. However, due to the high cost of sample labeling and the complex calculation of CNN, if the sample data is insufficiently labeled, the classification performance will be affected. Meta-learning has better generalization in classifying learning based on known knowledge to increase unknown data. These learning methods focusing on a small number of samples focus on the transfer of information between different neighborhoods. These methods extract knowledge from a small number of source domain samples, so that these samples can be applied to the target domain with known class information, thereby assisting in identifying classes or classes with very few samples that have not been seen before.

[0005] Among many algorithms for few-shot learning, using neural networks to process high-latitude data of HSI is prone to overfitting. Although the attention mechanism can compensate for the lack of samples to some extent, when the HSI dataset has unbalanced training samples, the performance is not necessarily satisfactory. At the same time, although the HSI classification method of convolutional neural network greatly improves the classification performance, it cannot well utilize the sequential characteristics of spectral features, and faces the challenge of being difficult to capture dynamic long sequence relationships as the network depth increases. SUMMARY

[0006] In view of the problems in the above-mentioned traditional method, the application provides a small sample hyperspectral image classification method and device based on spectral feature fusion, which can solve the problem of poor spectral feature fusion performance in the existing small sample situation, improve the spectral feature fusion performance, and improve the classification accuracy.

[0007] To achieve the above object, the embodiments of the application adopt the following technical solutions: On the one hand, a small sample hyperspectral image classification method based on spectral feature fusion is provided, which comprises: The high-dimensional convolution diffusion and patch expansion of the hyperspectral image input data are performed to obtain a data feature map IDres; and the operation of expanding the patch size is performed on the hyperspectral image input data without changing the number of wavebands to obtain a data feature Ires.

[0008] Two rounds of spectral feature extraction and fusion of different granularities are performed on the data feature map IDres by using two spectral feature extraction modules to obtain fine fusion data; wherein the first spectral feature extraction module is used to extract spatial multi-scale low-granularity features and spectral multi-scale high-granularity features from the data feature map IDres and perform feature fusion to obtain first round fusion features; and the second spectral feature extraction module is used to extract low-granularity features in the spectrum and spatial multi-scale high-granularity features from the first round fusion features and perform feature fusion to obtain fine fusion data.

[0009] The double-channel feature fusion Mamba is used to perform weight compensation and strong screening on the fine fusion data to obtain a result feature map.

[0010] The convolution operation is used to construct a close long sequence spectral dependence relationship according to the result feature map, and the feature Ires is used for feature compensation to obtain fine fusion data.

[0011] Image classification is performed according to the fine fusion data to obtain a hyperspectral image classification result.

[0012] On the other hand, a small sample hyperspectral image classification device based on spectral feature fusion is also provided, which comprises: A preprocessing unit is configured to perform high-dimensional convolution diffusion and patch expansion on the hyperspectral image input data to obtain a data feature map IDres; and perform the operation of expanding the patch size on the hyperspectral image input data without changing the number of wavebands to obtain a data feature Ires. The two-stage spectral-spatial feature extraction unit is configured to perform two rounds of spectral-spatial feature extraction and fusion of different granularities on the data feature map IDres by using two spectral-spatial feature extraction modules, to obtain fine fusion data. The first spectral-spatial feature extraction module is configured to extract spatial multi-scale low-granularity features and spectral multi-scale high-granularity features from the data feature map IDres, and perform feature fusion to obtain first round fusion features. The second spectral-spatial feature extraction module is configured to extract low-granularity features in the spectrum and spatial multi-scale high-granularity features from the first round fusion features, and perform feature fusion to obtain fine fusion data. The feature compensation and screening unit is configured to perform weight compensation and strong screening on the fine fusion data by using a double-path feature fusion Mamba, to obtain a result feature map. The feature fine fusion unit is configured to construct a close long sequence spectral-spatial dependence relationship by using a convolution operation according to the result feature map, and perform feature compensation by using the data feature Ires, to obtain fine fusion data. The hyperspectral image classification unit is configured to perform image classification according to the fine fusion data, to obtain a hyperspectral image classification result.

[0013] One of the above technical solutions has the following advantages and beneficial effects: The above small sample hyperspectral image classification method and device based on spectral-spatial feature fusion, the method comprises: processing HSI input data to obtain a data feature map IDres and data features; performing two rounds of spectral-spatial feature extraction and fusion of different granularities on the data feature map IDres by using two spectral-spatial feature extraction modules, to obtain fine fusion data; performing weight compensation and strong screening on the fine fusion data by using a double-path feature fusion Mamba, to obtain a result feature map. A close long sequence spectral-spatial dependence relationship is constructed by using a convolution operation according to the result feature map, and feature compensation is performed by using the data feature Ires, to obtain fine fusion data. Image classification is performed according to the fine fusion data, to obtain a hyperspectral image classification result. The method adopts a "coarse fusion-precise optimization-strong screening" multi-stage feature fusion strategy, to realize efficient classification under a small sample.

[0014] Description Figure 1 A flowchart of a small sample hyperspectral image classification method based on spectral-spatial feature fusion in one embodiment; Figure 2 A work flowchart of a small sample hyperspectral image classification method based on spectral-spatial feature fusion in one embodiment; Figure 3Figures showing the structure of SPLG-FEB, SPHG-FEB, SMHG-FEB and SMLG-FEB in an embodiment, wherein (a) is the structure of SMHG-FEB, (b) is the structure of SPHG-FEB, (c) is the structure of SPLG-FEB, and (d) is the structure of SMLG-FEB; Figure 4 Figure showing the structure of SSFAM in an embodiment; Figure 5 is Figure showing the structure of FFEVT in an embodiment; Figure 6 Figure showing the structure of DPFFM in an embodiment; Figure 7 Figure showing the classification results of all methods implemented on the LK dataset in an embodiment; Figure 8 Figure showing the classification results of all methods implemented on the HH dataset in an embodiment; Figure 9 Figures showing the performance bar charts of the methods using different neighborhood sizes for classification on each dataset in an embodiment, wherein (a) is the OA performance bar chart, (b) is the AA performance bar chart, and (c) is the Kappa performance bar chart; Figure 10 Figures showing the impact of sample size changes on the LK dataset in an embodiment, wherein (a) is the OA performance bar chart, and (b) is the Kappa performance bar chart; Figure 11 Figures showing the impact of sample size changes on the HH dataset in an embodiment, wherein (a) is the OA performance bar chart, and (b) is the Kappa performance bar chart; Figure 12 Figures showing the performance line charts of sample size changes on two datasets in an embodiment, wherein (a) is the performance line chart of sample size changes on the LK dataset, and (b) is the performance line chart of sample size changes on the HH dataset. DETAILED DESCRIPTION

[0015] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0016] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0017] In an embodiment, as shown in Figure 1 Fig. 1, a small sample hyperspectral image classification method based on spectral feature fusion is provided, which can include the following processing steps 100-108: Step 100: high-dimensional convolution diffusion and patch expansion are performed on the hyperspectral image input data to obtain data feature map IDres; without changing the number of bands, the input data is operated to expand the patch size to obtain data feature Ires.

[0018] Specifically, the hyperspectral image (HSI) is an advanced remote sensing method. First, the HSI input data is subjected to high-dimensional convolution diffusion and patch expansion to obtain data feature map IDres Meanwhile, without changing the number of bands, the input data is operated to expand the patch size to alleviate information loss, and the data feature . Figure 2 The convolution block in is mainly for type size normalization. Then, two rounds of different granularity spatial-spectral feature extraction and fusion are performed on the HSI data.

[0019] Step 102: two rounds of different granularity spatial-spectral feature extraction and fusion are performed on the data feature map IDres using two spatial-spectral feature extraction modules to obtain fine fusion data; wherein the first spatial-spectral feature extraction module is used to extract spatial multi-scale low-granularity features and spectral multi-scale high-granularity features from the data feature map IDres and perform feature fusion to obtain first round fusion features; the second spatial-spectral feature extraction module is used to extract low-granularity features in the spectrum and spatial multi-scale high-granularity features from the first round fusion features and perform feature fusion to obtain fine fusion data.

[0020] Specifically, the first spatial-spectral feature extraction module is referred to as ALEHEF module; and the second spatial-spectral feature extraction module is referred to as AHELEF module.

[0021] In ALEHEF, SPLG-FEB and SMHG-FEB are used to extract spatial multi-scale low-granularity features and spectral multi-scale high-granularity features. In AHELEF, SMLG-FEB and SPHG-FEB are used to extract low-granularity features in the spectrum and spatial multi-scale high-granularity features. SSFAM preliminarily fuses the obtained spatial-spectral granularity features. After the primary fusion data and residual feature map are input to FFEVT, the spatial-spectral features are further fine fused.

[0022] Through two feature fusion modules, the extracted spatial-spectral features are subjected to progressive feature fusion, and two rounds of feature fusion with different emphases are performed, which effectively improves the classification accuracy in the case of small samples.

[0023] Step 104: the fine fusion data is subjected to weight compensation and strong screening using a double-path feature fusion Mamba to obtain a result feature map.​

[0024] Specifically, after two rounds of spatial-spectral feature extraction with different granularities and comprehensive primary fusion and overall optimal fusion of spatial-spectral features via two spatial-spectral feature extraction modules, high and low granularities in spatial features and spectral signals are well fused. However, due to different focuses of the two rounds of fusion, the fused features may have weight deviation. The present application proposes a dual-path feature fusion Mamba (DualPath Feature Fusion Mamba, referred to as DPFFM) to realize deep optimal fusion of features, fine retention of core data and key boundary information, and also focus on building closer long-sequence spatial-spectral dependence to improve the fusion effect while maintaining feature diversity.

[0025] The dual-path structure in DPFFM can more effectively fuse multi-scale, direction and channel granular features, and realize the complementarity of spatial features and spectral signals. Meanwhile, DPFFM also builds a smoother long-sequence spatial-spectral dependence, and has better pixel representation and abstract semantic expression.

[0026] Step 106: According to the result feature map, a convolution operation is used to build a close long-sequence spatial-spectral dependence, and data features Ires are used for feature compensation to obtain fine fusion data.

[0027] Step 108: According to the fine fusion data, image classification is performed to obtain a hyperspectral image classification result.

[0028] Specifically, the fine fusion data is input into a classifier to obtain a hyperspectral image classification result.

[0029] The classifier can be fully connected + CrossEntropyLoss, fully connected + ArcFace additive angle interval + NLLLoss.

[0030] The above small sample hyperspectral image classification method based on spatial-spectral feature fusion processes the HSI input data to obtain data feature map IDres and data features; two rounds of spatial-spectral feature extraction with different granularities are performed on the data feature map IDres using two spatial-spectral feature extraction modules, and fusion is performed to obtain fine fusion data; weight compensation and strong screening are performed on the fine fusion data using a dual-path feature fusion Mamba to obtain a result feature map. According to the result feature map, a convolution operation is used to build a close long-sequence spatial-spectral dependence, and data features Ires are used for feature compensation to obtain fine fusion data, and according to the fine fusion data, image classification is performed to obtain a hyperspectral image classification result. A coarse fusion fine optimization strong screening multi-stage feature fusion strategy is adopted to realize efficient classification under few samples.

[0031] In an embodiment, the first hyperspectral feature extraction module in step 102 includes: an SPLG-FEB module for extracting low-granularity features in space, an SMHG-FEB module for extracting high-granularity features in spectrum, a hyperspectral fusion attention module for fusing features of different spatial and spectral granularities and dynamically weighting hyperspectral features, and a feature fusion enhanced visual transformer for refining and fusing global and local features of different dimensions; in the first hyperspectral feature extraction module: after reshaping the data feature map IDres, two-dimensional convolution and one-dimensional convolution modules are used for convolution processing respectively to obtain spaie1 features and speie1 features; the spaie1 features are input into the SPLG-FEB module to obtain spaoe1 features and spakf features; the speie1 features are input into the SMHG-FEB module to obtain speoe1 features and spekf features; after reshaping and splicing the spakf features and the spekf features, convolution processing is performed to obtain res1 features; the spaoe1 features after convolution processing are spliced with the speoe1 features after convolution and reshaping to obtain splicing features; the splicing features after convolution processing are input into the hyperspectral fusion attention module to obtain preliminary fusion features; after splicing the preliminary fusion features with the data feature map IDres, convolution processing is performed to obtain xf1 features; the xf1 features and the res1 features are input into the feature fusion enhanced visual transformer to obtain refined fusion features.

[0032] In an embodiment, the second hyperspectral feature extraction module in step 102 is obtained by replacing the SPLG-FEB module with an SMLG-FEB module for extracting low-granularity features in spectrum and replacing the SMHG-FEB module with an SPHG-FEB module for extracting spatial multi-scale high-granularity features; the two input features of the second hyperspectral feature extraction module are: the weighted fusion result of the xf1 features and the refined fusion features, and the refined fusion features.

[0033] In an embodiment, as Figure 3As shown in (a), the SMHG-FEB module includes two one-dimensional multi-scale convolution branches; the one-dimensional multi-scale convolution branch includes four one-dimensional convolution modules; the speie1 feature is input into the SMHG-FEB module to obtain the speoe1 feature and the spekf feature, including: the speie1 feature is input into the first one-dimensional multi-scale convolution branch respectively, after being processed by the first convolution module, the convolution feature is obtained, after being processed by the second one-dimensional convolution module, the second convolution feature is obtained, after the convolution feature and the second convolution feature are added, the third one-dimensional convolution module is used for processing to obtain the third convolution feature; after the convolution feature and the third convolution feature are added, the fourth convolution module is used for processing to obtain the fourth convolution feature, the convolution feature and the fourth convolution feature are multiplied and fused to obtain the first fusion feature, the result of batch standard processing of the first fusion feature and the fourth convolution feature added to the first convolution feature and the second convolution feature added is spliced to obtain the second fusion feature; the speie1 feature is input into the second one-dimensional multi-scale convolution branch respectively to obtain the third fusion feature and the fourth fusion feature; the first fusion feature and the third fusion feature are spliced to obtain the spekf feature; the spekf feature, the third fusion feature and the fourth fusion feature are spliced to obtain the speoe1 feature.

[0034] In one embodiment, as Figure 3 As shown in (a) to (d), the SMLG-FEB module and the SMHG-FEB module have the same structure, and the convolution kernel scale of the convolution layer in the one-dimensional convolution module in the two modules and the data type processed in the module are different; the SPLG-FEB module is obtained by replacing the one-dimensional convolution module in the SMLG-FEB module with a two-dimensional convolution module, and the SPLG-FEB module and the SPHG-FEB module have the same structure, and the convolution kernel scale of the convolution layer in the two-dimensional convolution module in the two modules and the data type processed in the module are different.

[0035] Specifically, the SPLG-FEB and the SPHG-FEB are respectively used to extract low-granularity features and high-granularity features in space, and 2DCNN is used for extraction. The feature maps in the SPLG-FEB are indicated by tail number 1, and the feature maps in the SPHG-FEB are indicated by tail number 2. The specific operation can be represented as follows.

[0036] (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) wherein, are input feature maps, weights, and biases, respectively In SPLG-FEB, the number of layers in SPALC1 is In order, they are . are all The number of layers in SPASC1 is In order, they are . are all 64. In SPHG-FEB, the number of layers in SPALC2 is In order, they are . are all 32. The number of layers in SPASC2 is In order, they are . are all 16. It is worth noting that the number of layers in SPASC1 is mirrored with that of SPALC1, and the number of layers in SPASC2 is mirrored with that of SPALC2. The spakf features and spaoe features are specifically (11) (12) wherein, and are feature maps from SPALC and SPASC, respectively. In SPHG-FEB, are .

[0037] SMHG-FEB and SMLG-FEB are used to extract high-granularity features and low-granularity features in the spectrum, respectively, and 1D CNN is used for extraction. Similar to the operation of SPLG-FEB and SPHG-FEB which are used to extract features in space. The feature maps in SMHG-FEB are denoted by a suffix of 1, while the feature maps in SMLG-FEB are denoted by a suffix of 2. The specific operation can be represented as follows: (13) (14) (15) (16) (17) (18) (19) (20) (21) wherein, are input feature maps and weights respectively ; in SMHG-FEB, the are in sequence . are all 32. The are in sequence . are all 64. In SMLG-FEB, the are in sequence . are all 32. The are in sequence . are all 64. It is worth noting that the number of layers in SPELC1 is the mirror image of SPESC1, and the number of layers in SPELC2 is the mirror image of SPESC2. The spekf feature and the speoe feature are: (22) (23) wherein, and are feature maps from SPESC and SPELC respectively. In SMLG-FEB, are .

[0038] In an embodiment, the spliced features are input into the spectral fusion attention module after convolution processing to obtain preliminary fusion features, including: the spliced features are subjected to convolution processing to obtain module input features; the module input features are first subjected to Squeeze and then reshaped to obtain reshaped Q, K and V; attention calculation is performed between each position of the transposed data of K and the reshaped Q to obtain attention scores: (24) wherein, is the attention score, and are the reshaped Q and K respectively, B is a batch matrix multiplication, the superscript T is a transpose, is a scaling factor.

[0039] The K after remodeling is weighted by a learnable mixing coefficient and added to the attention score to obtain a fusion attention feature; the V after remodeling and the attention score are processed to obtain an intermediate feature as: (25) wherein, is the intermediate feature, is a point-by-point convolution followed by batch normalization, is a view operation, is the V after remodeling.

[0040] The fusion attention feature and the intermediate feature are fused to obtain a preliminary fusion feature as: (26) wherein, is the preliminary fusion feature, is the fusion attention feature, .

[0041] Specifically, in order to effectively fuse the granularity features of different spaces and spectrums, and dynamically weight the space-spectrum features, obtain the space-spectrum feature association, the present application proposes a space-spectrum fusion attention module (SSFAM) to enhance the pixel-level representation ability, realize cross-channel attention interaction while maintaining the spatial structure, so as to realize comprehensive understanding of the HSI image. As shown in Figure 4 .

[0042] The input feature map is obtained, which is squeezed and reshaped into . Then, the transpose performs attention calculation between each position and to obtain an attention score , wherein a scaling factor is used to prevent gradient explosion. After the attention is obtained, a learnable mixing coefficient (initially 0) is used to strengthen the relevance of spatial features and spectral signals, and realize preliminary fusion of space-spectrum features. is the output result of the SSFAM, which is consistent with the shape and size of the input feature map. The specific operation is shown in formulas (24) to (26).

[0043] The space-spectrum fusion attention module (SSFAM) performs primary fusion on the extracted space-spectrum features. The SSFAM adaptively weights the space-spectrum features, realizes cross-channel feature interaction under the premise of maintaining the stability of the spatial structure, enhances the connection between spatial features and spectral signals, enhances the connection between pixels, and obtains better expression ability.

[0044] In one embodiment, the feature blending enhanced visual converter comprises a local stream and a global stream; the feature is input into the feature blending enhanced visual converter to obtain a refined fusion feature, comprising: The feature is input into the feature blending enhanced visual converter to obtain a refined fusion feature, comprising: The feature is input into the feature blending enhanced visual converter to obtain a refined fusion feature, comprising: (31) (32) (33) wherein, is the local feature, is the flattened and transposed xf1 feature, P() is the Permute operation, is the Reshape operation, is the 3x3 convolution kernel and the two-dimensional convolution of the c channel number, is the activation function, is the two-dimensional batch normalization processing operation, is the refined attention, is the query vector, the key vector, and the value vector, is the bias term, is the matrix multiplication, is the scaling factor, is the full connection operation, and the superscript T is the transposition operation; In the global stream, the local feature and the deformed res1 feature are data blended to obtain the initial interaction information of the local stream and the global stream: (34) (35) wherein, is the initial interaction information, is the intermediate information, is the adaptive weight, L is the sequence length, and Rp is the Repeat operation; The local feature and the initial interaction information are finely aggregated to obtain a refined fusion feature: (36) wherein, is the refined fusion feature, , is the concatenation operation.

[0045] ​Specifically, after the initial fusion via the SSFAM, to further improve the understanding of spatial knowledge and spectral signals, the present application proposes a feature fusion enhanced vision transformer (FFEVT). The FFEVT connects the fused feature maps more smoothly, strengthens the refined fusion of global and local features of different dimensions, and contains more detailed information and more abstract semantic expressions. The structure of the FFEVT is shown in Figure 5 is .

[0046] The FFEVT has two input feature maps, which are local streams and global streams, respectively. In the local stream, the inputted spatial-spectral features are refined and fused. First, is flattened and transposed to . is adopted, transposed and reshaped to , and then convolved to obtain the feature map . is divided into and , is type-remodeled and permuted to be divided into and . After that, , and rows are fine-optimized and fused to obtain attention to make the connection between features smoother. It also serves as a residual feature to compensate for the local feature . and are specifically operated as follows.

[0047] (37) (38) (39) wherein, is used for scaling to prevent data instability.

[0048] In the global stream, is data-fused with the deformed to obtain the initial interaction information of the local stream and the global stream to increase the connection between the local stream and the global stream. The specific operation is shown in equation (34) and equation (35). is a factor for stabilizing data.

[0049] with Fine-grained aggregation to enhance the control of local and global, increase the degree of integration between features. The resulting feature map Keep consistent with the shape of the input feature map. The operation is shown in formula (36).

[0050] Feature fusion enhanced visual transformer (FFEVT). FFEVT optimizes the fusion of primary fused spectral features. FFEVT has two feature inputs and is divided into local stream and global stream two ways to refine the fusion of spectral features to obtain more natural fusion features.

[0051] In one embodiment, the dual-path feature fusion Mamba includes a Mamba mainstream branch and a bypass branch; step 104 includes: weighting and dividing the fine-grained fusion data and the xf2 features of the second spectral feature extraction module into a plurality of non-overlapping local small blocks; performing regular vector mapping, region division, normalization and other regular processing on the local small blocks to obtain processed feature maps; inputting the processed feature maps into the Mamba mainstream branch to obtain Mamba stream feature maps as follows: (40) (41) (42) wherein, and is the processed feature map, and is the forward scanning feature and the backward scanning feature, is the Mamba stream feature map, is the SiLu activation function, SSM is the SSM operation; linear is the linearization operation, Re is the Rearrange operation, is a learnable parameter.

[0052] The Mamba stream feature map is strongly screened with the bypass feature to obtain a resulting feature map as follows: (43) wherein, is the resulting feature map, is the processed shunt feature, WR is the Window Reverse operation, is a learnable parameter, and MLP is a multi-layer perceptron, is a 3x3 convolution kernel and a 42-channel two-dimensional convolution.

[0053] Specifically, DPFFM realizes feature depth optimization fusion, fine preservation of core data and key boundary information, and construction of closer long sequence spatial spectrum dependency, to improve the fusion effect while maintaining feature diversity. The structure of DPFFM is shown in Figure 6 .

[0054] and . and . as the feature map input of DPFFM. are divided into non-overlapping local small blocks. To make the semantic connection between features smoother, DPFFM is divided into two directions, one is the long sequence spatial spectrum dependency expression of the Mamba mainstream branch, and the other is the feature compensation of the semantic expression of the bypass branch. The specific operation of the Mamba mainstream branch is as follows.

[0055] (44) (45) (46) wherein, and the number of channels are consistent.

[0056] The Mamba stream feature map calculation is shown in formula (40) to formula (42).

[0057] The SSM operation realizes data conversion through a learnable hidden state. The specific operation of SSM is as follows: (47) (48) (49) (50) (51) wherein, the parameter . The type of is consistent with the type of . It is worth noting that is a continuous parameter, while is a discrete parameter. The continuous parameter is converted into a discrete parameter to improve the calculation efficiency. At the same time, the time scale is represented by to use the zero-order hold rule. is a hidden state to realize the conversion from the input to the output

[0058] If the input sequence length is , the output of the discrete-time equation (57) can be calculated by defining a global convolution kernel . Specifically: , (52) The Mamba stream feature map is strongly screened with the bypass feature, and the effective features are fine-tuned and smoothed to obtain the result feature map as shown in equation (43) .

[0059] The network extracts the spectral features of different particle sizes through SPLG-FEB and SMLG-FEB, SPHG-FEB and SMHG-FEB, and then adaptively weights and enhances the spectral correlation through spectral fusion attention (SSFAM). The feature enhancement visual converter (FFEVT) combines local and global flow to optimize feature fusion, and finally uses the dual-path Mamba fusion module (DPFFM) to establish long sequence dependence.

[0060] In one embodiment, step 106 includes: using a hybrid convolution module to extract features from the result feature map, and splicing the extracted features and data features to obtain first spliced features; the hybrid convolution module includes a two-dimensional maximum pooling layer, a two-dimensional convolution layer, a two-dimensional batch normalization layer, a one-dimensional convolution layer, a Transpose2D, a one-dimensional batch normalization layer, and a ReLU activation function; after the first spliced features are subjected to two-dimensional convolution processing, fine-grained fusion data is obtained.

[0061] In some embodiments, experimental examples are also provided, WHU_HI_LongKou (LK) and WHU_HI_HongHu (HH) two public HSI datasets, the first is a satellite remote sensing dataset, and the last two are unmanned aerial vehicle remote sensing datasets. Each class in the two datasets is randomly selected 5 as training samples, and the size of the neighborhood is 9x9 pixels. The remaining pixels are used as test samples. Such an experimental setup is relatively fair.

[0062] (1) LK dataset The LK dataset is collected by the Headwall NanoHyperspec imaging spectrometer with an 8mm focal length, mounted on the DJI Matrice 600 Pro unmanned aerial vehicle platform, in Honghu City, Hubei Province, in 2018, provided by the RSIDEA research team of Wuhan University. The dataset contains 550x400 image pixels, after processing, there are 270 effective bands, the spectral range is 400-1000 nm, and the spatial resolution is 0.463 m. The LK dataset has 9 mutually exclusive land cover classes, as shown in Table 1. ​

[0063] Table 1. Land cover classes and sample distribution of LK dataset

[0064] (2) HH dataset The HH dataset was collected in Honghu City, Hubei Province in 2017 by a 17mm focal length Headwall NanoHyperspec imaging spectrometer mounted on a DJI Matrice 600 Pro UAV platform. The dataset contains 940x475 image pixels, after processing, there are 270 valid bands, with a spectral range of 400-1000 nm and a spatial resolution of 0.043m. The HH dataset has 22 mutually exclusive land cover classes, as shown in Table 2.

[0065] Table 2. Land cover classes and sample distribution of HH dataset

[0066] (3) Experimental setup This example is carried out on the Ubuntu 18.04.5 LTS platform, the GPU is NVIDIA GeForce RTX 3090, the memory is 24GB, and the software environment is python3.10.13 and pytorch2.1.1+cu118.

[0067] In order to verify the effectiveness of the method, a series of classification experiments were carried out using two famous hyperspectral datasets, including traditional machine learning algorithms such as extreme learning machine (ELM), DRIN, SSFTT, SSTN, SSConvNeXt, SF, GSP, DS, morph and gscvit, and nine kinds of CNN algorithms, as well as MambaHSI and MHSSMamba two kinds of Mamba classification model.

[0068] The input size of the patch of the method is 9x9, and the number of channels depends on the dataset. The number of channels of LK and HH is 270. For the LK dataset, the learning rate of the model is 0.0001, the epoches is 200, and the batch size is 192. For the HH dataset, the learning rate is 5e-4, the epoches is 500, and the batch size is 128. When training all datasets, the optimizer and loss function are consistent, using Adam optimizer and cross-entropy loss function respectively.

[0069] For the traditional machine learning method ELM, it is implemented using the scikit-elm library in Python with default parameters. For the deep learning methods, their network structures all follow the design of the original paper. The parameters used by DRIN, SSFTT, SSTN, SSConvNeXt, DS, gscvit, MambaHSI, and MHSSMamba are consistent with the proposed method. Due to the characteristics of the network structure, the parameters of SF, GSP, and morph basically follow the original paper. The input data size is 9x9. The classification performance is evaluated by three indicators: overall accuracy (OA), average accuracy (AA), and kappa coefficient (Kappa).

[0070] (4) Comparative test 1) Results of LK dataset The classification performance of different methods on the LK dataset is shown in Figure 7 . All methods are clear in identifying the water class, with only a small amount of impurities appearing in the SSTN method. In classifying corn and broadleaf soybeans, the proposed method performs better, with ELM showing a large number of noise points, and other methods showing more or less chaotic pixel points. In summary, the proposed method performs better, with a relatively clearer picture.

[0071] The class classification accuracy of different classification methods on the LK dataset is shown in Table 3. From Table 3, it can be seen that all methods perform well in classifying water, with OA values higher than 95%, and the lowest OA value of the SSTN method is 95.17%. Different methods for classifying corn and broadleaf soybeans, ELM has a lower OA value, and the proposed method has a higher OA value. In summary, deep learning-based methods perform better than traditional machine learning methods, with higher OA, AA, and Kappa values. The proposed method performs the best, with an OA value about 2% higher than MambaHSI and about 20% higher than ELM. Therefore, the classification performance of the proposed method on the LK dataset is good.

[0072] Table 3 Classification results of LK dataset

[0073] 3) Results of HH dataset The classification map of each method on the HH dataset is shown in Figure 8 . All methods have different degrees of misclassification pixels when classifying red roofs and cotton. Among them, ELM, SF, and GSP perform poorly, with a large number of mottling points, and the visual performance is poor. Among them, DS and the proposed method are relatively clear in distinguishing the two classes, with the red roof map relatively clear. In summary, the proposed method shows a relatively clean thematic map.

[0074] Table 4 shows that ELM, SF, and GSP have the lowest classification accuracy for red roofs and cotton, consistent with visual perception. DS and our proposed method, however, perform better than other methods in these two categories, achieving classification accuracies of 94.58% and 95.86% for red roofs, respectively. Overall, our proposed method performs best, with an overall accuracy (OA) approximately 1% higher than MambaHSI and approximately 43% higher than the worst-performing ELM method. Furthermore, our evaluation metrics OA and Kappa are the highest, at 76.33% and 70.46%, respectively. Therefore, our proposed method performs well on the HH dataset and exhibits good generalization ability.

[0075] Table 4 Classification results of the HH dataset

[0076] (5) Discussion of experimental results 1) The influence of neighborhood size For HSI classification tasks, many factors influence classification performance. Among them, the size of the input network's neighborhood has a significant impact on the final classification result. This is because neighborhoods of different sizes have different pixel features, and the number of pixel features is closely related to the neighborhood size. The richer the features required by the classification network, the larger the input neighborhood tends to be. Therefore, most existing HSI classification methods choose larger neighborhoods as inputs.

[0077] To further explore the impact of neighborhood size on classification performance, four neighborhood sizes—5×5, 7×7, 9×9, and 11×11—were compared. Figure 9 As shown, where Figure 9 (a) is a bar chart of OA performance. Figure 9 (b) is a bar chart showing the performance of AA. Figure 9 (c) is a bar chart showing the performance of Kappa, from... Figure 9 It can be seen that larger neighborhoods result in better classification performance. However, the performance improvement from 9×9 to 11×11 is significantly lower than that from 5×5 to 7×7, indicating that the performance improvement with increasing neighborhood size tends to plateau. Furthermore, larger neighborhoods mean larger input data volumes, increasing the computational cost of the network structure and naturally reducing efficiency. Therefore, after balancing various factors, a neighborhood size of 9×9 was set for the experiments.

[0078] 2) The impact of the number of training samples The number of training samples also has a significant impact on the classification performance of HSI classification tasks. The more training samples input, the more data the network model can learn from, and the higher its classification performance. Therefore, most existing HSI classification methods use a large number of samples for training.

[0079] To further evaluate the impact of the number of training samples on classification performance, the classification accuracy of three sample sizes (1, 5, and 10) was compared. Figure 10 This is a diagram illustrating the impact of changes in the sample size of the LK dataset. Figure 10 (a) is a bar chart of OA performance. Figure 10 (b) is a bar chart showing the performance of Kappa. Figure 11 A bar chart showing the impact of changes in sample size on the HH dataset. Figure 11 (a) is a bar chart of OA performance. Figure 11 (b) is a performance bar chart.

[0080] from Figure 10 and Figure 11 It can be seen that the number of training samples has a positive effect on the network's classification performance. However, when the sample size reaches a certain level, the OA coefficient also tends to stabilize. For example, the accuracy improvement from 5 samples to 10 samples is less than the accuracy improvement from 1 sample to 5 samples. This indicates that the network's classification performance improves with increasing sample size, but the rate of improvement gradually decreases and plateaus after reaching a certain threshold.

[0081] 3) The impact of the number of training samples for different categories A key factor affecting the performance of HSI classification tasks is the number of training samples. Generally, classification accuracy improves with increasing sample size. The total number of training samples consists of the sample size for each class. Furthermore, the actual HSI datasets available are often limited, making it essential to expand the existing dataset for small-sample classification tasks. This paper employs a spatial shuffling technique to expand the sample size based on an initial sample size of 5 classes obtained from two datasets.

[0082] To further analyze the impact of sample size on classification accuracy, the results of implementing four different sample sizes—1000, 5000, 10000, and 50000—were compared. Figure 12 (a) is a line graph showing the performance of the LK dataset under varying sample size. Figure 12The middle (b) is the performance fold line chart of the sample size change on the HH dataset. The two datasets perform better when the sample size is expanded to 50,000. The OA value and Kapp value from 5,000 to 50,000 show an overall upward trend. However, compared with the trend from 1,000 to 5,000 and from 5,000 to 50,000, the improvement is obviously better. This phenomenon also shows that after the sample size reaches a certain threshold, the evaluation index tends to be stable, and the classification effect is difficult to further improve. Therefore, after comprehensively balancing various factors, the sample size of each class of the LK dataset is set to 5,000, and the sample size of each class of the HH dataset is set to 10,000.

[0083] It should be understood that, although the above process Figure 1 shows the steps in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps of the above process Figure 1 may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0084] In one embodiment, a small sample hyperspectral image classification device based on spectral feature fusion is also provided, which comprises: a preprocessing unit for high-dimensional convolution diffusion and patch expansion on the hyperspectral image input data to obtain data feature map IDres; and an operation of expanding the patch size on the hyperspectral image input data without changing the number of wavebands to obtain data feature Ires.

[0085] A two-stage spectral feature extraction unit is configured to perform two rounds of spectral feature extraction and fusion with different granularities on the data feature map IDres by using two spectral feature extraction modules to obtain fine fusion data; wherein the first spectral feature extraction module is configured to extract spatial multi-scale low-granularity features and spectral multi-scale high-granularity features from the data feature map IDres, and perform feature fusion to obtain first round fusion features; and the second spectral feature extraction module is configured to extract low-granularity features in the spectrum and spatial multi-scale high-granularity features from the first round fusion features, and perform feature fusion to obtain fine fusion data.

[0086] A feature compensation and screening unit is configured to perform weight compensation and strong screening on the fine fusion data by using a double-path feature fusion Mamba to obtain a result feature map.

[0087] The feature fine fusion unit is configured to construct a close long sequence space spectrum dependence relationship by using a convolution operation according to the result feature map, and to perform feature compensation by using the data feature Ires to obtain fine fusion data. The hyperspectral image classification unit is configured to perform image classification according to the fine fusion data to obtain a hyperspectral image classification result.

[0088] In one embodiment, the first space spectrum feature extraction module in the two-stage space spectrum feature extraction unit includes: an SPLG-FEB module configured to extract low-granularity features in space, an SMHG-FEB module configured to extract high-granularity features in spectrum, a space spectrum intermingling attention module configured to fuse features of different spatial and spectral granularities and dynamically weight the space spectrum features, and a feature intermingling enhanced visual converter configured to refine and fuse global and local features of different dimensions; in the first space spectrum feature extraction module: after the data feature map IDres is reshaped, two-dimensional and one-dimensional convolution modules are used for convolution processing to obtain spaie1 features and speie1 features; the spaie1 features are input into the SPLG-FEB module to obtain spaoe1 features and spakf features; the speie1 features are input into the SMHG-FEB module to obtain speoe1 features and spekf features; after the spakf features and the spekf features are reshaped and spliced, convolution processing is performed to obtain res1 features; the results of convolution processing of the spaoe1 features and the results of convolution and reshaping of the speoe1 features are spliced to obtain spliced features; after the spliced features are subjected to convolution processing, the preliminary fusion features are obtained by inputting the spliced features into the space spectrum intermingling attention module; after the preliminary fusion features are spliced with the data feature map IDres and subjected to convolution processing, the xf1 features are obtained; the xf1 features and the res1 features are input into the feature intermingling enhanced visual converter to obtain the refined fusion features.

[0089] In one embodiment, the second space spectrum feature extraction module in the two-stage space spectrum feature extraction unit is obtained by replacing the SPLG-FEB module with an SMLG-FEB module configured to extract low-granularity features in spectrum and replacing the SMHG-FEB module with an SPHG-FEB module configured to extract multi-scale high-granularity features in space; the two input features of the second space spectrum feature extraction module are: a weighted fusion result of the xf1 features and the refined fusion features, and the refined fusion features.

[0090] In an embodiment, the SMHG-FEB module includes two one-dimensional multi-scale convolution branches; the one-dimensional multi-scale convolution branch includes four one-dimensional convolution modules; the two-stage space-spectrum feature extraction unit is further configured to input the speie1 feature into the first one-dimensional multi-scale convolution branch, obtain a convolution feature after processing the speie1 feature by the first convolution module, obtain a second convolution feature after processing the convolution feature by the second one-dimensional convolution module, obtain a third convolution feature after processing the convolution feature and the second convolution feature by the third one-dimensional convolution module, obtain a fourth convolution feature after processing the convolution feature and the third convolution feature by the fourth convolution module, obtain a first fusion feature by fusing the convolution feature and the fourth convolution feature, and obtain a second fusion feature by splicing a result of batch standard processing of a sum of the first fusion feature and the fourth convolution feature and a result of batch standard processing of a sum of the first convolution feature and the second convolution feature; input the speie1 feature into the second one-dimensional multi-scale convolution branch to obtain a third fusion feature and a fourth fusion feature; splice the first fusion feature and the third fusion feature to obtain the spekf feature; and splice the spekf feature, the third fusion feature, and the fourth fusion feature to obtain the speoe1 feature.

[0091] In an embodiment, the SMLG-FEB module and the SMHG-FEB module in the two-stage space-spectrum feature extraction unit have the same structure, and the convolution kernel scale of the convolution layer in the one-dimensional convolution module and the data type processed in the module are different between the two modules; the SPLG-FEB module is obtained by replacing the one-dimensional convolution module in the SMLG-FEB module with a two-dimensional convolution module, and the SPLG-FEB module and the SPHG-FEB module have the same structure, and the convolution kernel scale of the convolution layer in the two-dimensional convolution module and the data type processed in the module are different between the two modules.

[0092] In an embodiment, the two-stage space-spectrum feature extraction unit is further configured to perform convolution processing on the spliced feature to obtain a module input feature; perform Squeeze on the module input feature, and then reshape to obtain a reshaped Q, K, and V; perform attention calculation between each position on transposed data of the reshaped K and the reshaped Q to obtain an attention score as shown in formula (24); add the reshaped K weighted by a learnable mixing coefficient to the attention score to obtain a fusion attention feature; process the reshaped V and the attention score to obtain an intermediate feature as shown in formula (25); and fuse the fusion attention feature and the intermediate feature to obtain a preliminary fusion feature as shown in formula (26).

[0093] In one embodiment, the two-stage spectral feature extraction unit is further configured to input the xf1 feature as a local stream, and adopt the local stream to refine and fuse the spectral feature to obtain a local feature as shown in equations (31) to (33); in a global stream, the local feature is mixed with the deformed res1 feature to obtain initial interaction information of the local stream and the global stream as shown in equations (34) and (35); and the local feature and the initial interaction information are finely aggregated to obtain a refined fusion feature as shown in equation (36).

[0094] In one embodiment, the two-stage spectral feature extraction unit is further configured to input the xf1 feature as a local stream, and adopt the local stream to refine and fuse the spectral feature to obtain a local feature as shown in equations (31) to (33); in a global stream, the local feature is mixed with the deformed res1 feature to obtain initial interaction information of the local stream and the global stream as shown in equations (34) and (35); and the local feature and the initial interaction information are finely aggregated to obtain a refined fusion feature as shown in equation (36).

[0095] In one embodiment, the feature fine fusion unit is further configured to extract features from the result feature map by using a hybrid convolution module, and splice the extracted features and data features to obtain a first spliced feature; the hybrid convolution module includes a two-dimensional maximum pooling layer, a two-dimensional convolution layer, a two-dimensional batch normalization layer, a one-dimensional convolution layer, a Transpose2D, a one-dimensional batch normalization layer, and a ReLU activation function; and after the first spliced feature is processed by two-dimensional convolution, fine fusion data is obtained.

[0096] It can be understood that the specific explanations and descriptions of the small sample hyperspectral image classification device based on spectral feature fusion can refer to the corresponding explanations and descriptions of the small sample hyperspectral image classification method based on spectral feature fusion in the above embodiments, which will not be repeated here. Each module in the small sample hyperspectral image classification device based on spectral feature fusion described above can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of a device with data processing function in hardware form, or can be stored in the memory of the aforementioned device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices in the prior art.

[0097] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of the technical features is considered to be within the scope of the present disclosure.

[0098] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the protection scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, some modifications and improvements can be made without departing from the concept of the present application, and all of them belong to the protection scope of the present application.

Claims

1. A small sample hyperspectral image classification method based on spectral feature fusion, characterized in that, The method comprises: high-dimensional convolution diffusion and patch expansion are performed on the hyperspectral image input data to obtain data feature map IDres; the size of the patch of the hyperspectral image input data is expanded without changing the number of bands, to obtain data feature Ires; two rounds of different granularity spatial-spectral feature extraction and fusion are performed on the data feature map IDres by using two spatial-spectral feature extraction modules to obtain fine fusion data; the first spatial-spectral feature extraction module is used to extract spatial multi-scale low-granularity features and spectral multi-scale high-granularity features from the data feature map IDres, and perform feature fusion to obtain first round fusion features; the second spatial-spectral feature extraction module is used to extract low-granularity features in the spectrum and spatial multi-scale high-granularity features from the first round fusion features, and perform feature fusion to obtain fine fusion data; weight compensation and strong screening are performed on the fine fusion data by using a double-path feature fusion Mamba to obtain a result feature map; a close long sequence spatial-spectral dependence relationship is constructed by using a convolution operation according to the result feature map, and feature compensation is performed by using the data feature Ires to obtain refined fusion data; image classification is performed according to the refined fusion data to obtain a hyperspectral image classification result.

2. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 1, characterized in that, The first spatial-spectral feature extraction module comprises an SPLG-FEB module for extracting low-granularity features in the space, an SMHG-FEB module for extracting high-granularity features in the spectrum, a spatial-spectral intermingling attention module for fusing different spatial and spectral granularity features and dynamically weighting spatial-spectral features, and a feature intermingling enhanced visual converter for refining and fusing different dimensional global and local features; In the first spatial-spectral feature extraction module: the data feature map IDres is reshaped and then subjected to convolution processing by using a two-dimensional convolution module and a one-dimensional convolution module to obtain spaie1 features and speie1 features; the spaie1 features are input into the SPLG-FEB module to obtain spaoe1 features and spakf features; the speie1 features are input into the SMHG-FEB module to obtain speoe1 features and spekf features; the spakf features and the spekf features are reshaped, spliced and then subjected to convolution processing to obtain res1 features; the results of convolution processing of the spaoe1 features and the results of convolution and reshaping of the speoe1 features are spliced to obtain splicing features; the splicing features are input into the spatial-spectral intermingling attention module after convolution processing to obtain preliminary fusion features; the preliminary fusion features are spliced with the data feature map IDres and then subjected to convolution processing to obtain xf1 features; the xf1 features and the res1 features are input into the feature intermingling enhanced visual converter to obtain refined fusion features.

3. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 2, characterized in that, The second spectral feature extraction module is obtained by replacing the SPLG-FEB module with an SMLG-FEB module for extracting low-granularity features in the spectrum, and replacing the SMHG-FEB module with an SPHG-FEB module for extracting spatial multi-scale high-granularity features. Two input features of the second spectral feature extraction module are: a result of weighted fusion of the xf1 feature and the refined fusion feature, and the refined fusion feature.

4. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 2, characterized in that, The SMHG-FEB module comprises: two one-dimensional multi-scale convolution branches; and each one-dimensional multi-scale convolution branch comprises four one-dimensional convolution modules. The speie1 feature is input into the SMHG-FEB module to obtain speoe1 feature and spekf feature, including: The speie1 feature is input into the first one-dimensional multi-scale convolution branch, and after being processed by the first convolution module, a convolution feature is obtained. After the convolution feature is processed by the second one-dimensional convolution module, a second convolution feature is obtained. After the convolution feature and the second convolution feature are added, the third one-dimensional convolution module is used for processing to obtain a third convolution feature. After the convolution feature and the third convolution feature are added, the fourth convolution module is used for processing to obtain a fourth convolution feature. The convolution feature and the fourth convolution feature are multiplied and fused to obtain a first fusion feature. The result of batch standard processing of the first fusion feature and the fourth convolution feature added to the first convolution feature and the second convolution feature is spliced to obtain a second fusion feature. The speie1 feature is input into the second one-dimensional multi-scale convolution branch to obtain a third fusion feature and a fourth fusion feature. The first fusion feature and the third fusion feature are spliced to obtain the spekf feature. The spekf feature, the third fusion feature, and the fourth fusion feature are spliced to obtain the speoe1 feature.

5. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 2, characterized in that, The SMLG-FEB module and the SMHG-FEB module have the same structure, and the convolution kernel scale of the convolution layer in the one-dimensional convolution module and the data type processed in the module are different in the two modules. The SPLG-FEB module is obtained by replacing the one-dimensional convolution module in the SMLG-FEB module with a two-dimensional convolution module. The SPLG-FEB module and the SPHG-FEB module have the same structure, and the convolution kernel scale of the convolution layer in the two-dimensional convolution module and the data type processed in the module are different in the two modules.

6. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 2, characterized in that, After the spliced feature is convoluted, the initial fusion feature is obtained by inputting the spliced feature into the spectral cross-fusion attention module, including: The spliced feature is convoluted to obtain a module input feature; The module input feature is first squeezed and then reshaped to obtain reshaped Q, K, and V; The attention score is obtained by performing attention calculation between each position of the transposed data of the reshaped K and the reshaped Q. wherein, is an attention score, and are reshaped Q and K, respectively, is a batch matrix multiplication, and the superscript T is the transpose, is a scaling factor; The fusion attention feature is obtained by adding the reshaped K weighted by the learnable mixing coefficient to the attention score. The remolded V and the attention score are processed to obtain an intermediate feature as: wherein, is an intermediate feature, is a point-wise convolution followed by batch normalization, is a view operation, is the reshaped V; The fusion attention feature and the intermediate feature are fused to obtain a preliminary fusion feature as: wherein, is the preliminary fused feature, is the fused attention feature.

7. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 2, characterized in that, The feature intermingling enhanced visual converter includes a local stream and a global stream; The xf1 feature and the res1 feature are input into the feature intermingling enhanced visual converter to obtain a refined fusion feature, including: The xf1 feature is taken as an input of the local stream, and the local stream is used to refine and fuse the empty spectrum feature to obtain a local feature as: wherein, is a local feature, is the flattened and transposed xfl feature, P() is a Permute operation, is a Reshape operation, is a 3x3 convolution kernel and c channel number two-dimensional convolution, is an activation function, is a two-dimensional batch normalization processing operation, is a refined attention, is a query vector, a key vector, a value vector, is a bias term, is a matrix multiplication, is a scaling factor, is a full connection operation, and a superscript T is a transposition operation. In the global stream, the local feature and the deformed res1 feature are data-intermingled to obtain initial interaction information of the local stream and the global stream as: wherein, is the initial mutual information, is the intermediate information, is the adaptive weight, L is the sequence length, and Rp is the Repeat operation; The local feature and the initial interaction information are finely aggregated to obtain a refined fusion feature as: wherein, to refine the fused features, to point convolution, to stitching operation.

8. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 1, characterized in that, The dual-path feature fusion Mamba includes a Mamba mainstream branch and a bypass branch, The fine fusion data is subjected to weight compensation and strong screening by the dual-path feature fusion Mamba to obtain a result feature map, including: The fine fusion data and the xf2 feature of the second empty spectrum feature extraction module are weighted and divided into a plurality of local small blocks that do not overlap; The local small blocks are subjected to regular vector mapping, region division, normalization and other regular processing to obtain a processed feature map; The processed feature map is input into the Mamba mainstream branch to obtain a Mamba stream feature map as: wherein, and are processed feature maps, and are forward and backward scan features, is a Mamba stream feature map, is a SiLu activation function, SSM is an SSM operation; linear is a linearization operation, Re is a Rearrange operation, are learnable parameters; The Mamba stream feature map and the bypass feature are strongly screened to obtain a result feature map as: wherein, is the result feature map, is the processed split feature, WR is a Window Reverse operation, is a learnable parameter, MLP is a multi-layer perceptron, is a 3x3 kernel and a 42-channel two-dimensional convolution.

9. The small sample size hyperspectral image classification method based on the fusion of spectral and spatial features according to claim 1, characterized in that, According to the result feature map, a tight long sequence empty spectrum dependency relationship is constructed by a convolution operation, and the data feature Ires is used for feature compensation to obtain fine fusion data, including: The result feature map is subjected to feature extraction by a hybrid convolution module, and the extracted feature and the data feature Ires are spliced to obtain a first spliced feature; the hybrid convolution module includes a two-dimensional maximum value pooling layer, a two-dimensional convolution layer, a two-dimensional batch normalization layer, a one-dimensional convolution layer, a Transpose2D, a one-dimensional batch normalization layer and a ReLU activation function; After the first spliced feature is subjected to two-dimensional convolution processing, fine fusion data is obtained.

10. A small sample hyperspectral image classification device based on spectral feature fusion, characterized in that, Including: A preprocessing unit is configured to perform high-dimensional convolution diffusion and patch expansion on the hyperspectral image input data to obtain a data feature map IDres; Without changing the number of bands, the hyperspectral image input data is operated to expand the patch size to obtain a data feature Ires; A two-stage empty spectrum feature extraction unit is configured to perform two rounds of empty spectrum feature extraction and fusion with different granularities on the data feature map IDres by two empty spectrum feature extraction modules to obtain fine fusion data; wherein, a first empty spectrum feature extraction module is configured to extract spatial multi-scale low-granularity features and spectral multi-scale high-granularity features from the data feature map IDres and perform feature fusion to obtain a first round of fusion features; a second empty spectrum feature extraction module is configured to extract low-granularity features in the spectrum and spatial multi-scale high-granularity features from the first round of fusion features and perform feature fusion to obtain fine fusion data; The feature compensation screening unit is configured to perform weight compensation and strong screening on the fine fusion data by using a two-way feature fusion Mamba, to obtain a result feature map; The feature fine fusion unit is configured to construct a close long-sequence space spectrum dependence relationship by using a convolution operation according to the result feature map, and to perform feature compensation by using the data feature Ires, to obtain fine fusion data; The hyperspectral image classification unit is configured to perform image classification according to the fine fusion data, to obtain a hyperspectral image classification result.

Images