Hierarchical progressive hyperspectral image classification method based on FKAN

Abstract

The present invention belongs to the technical field of remote sensing image classification, and in particular, relates to a hierarchical progressive hyperspectral image classification method based on FKAN. S1: After dividing the labeled hyperspectral image data set into a training set and a test set, the training set is processed to obtain global-level samples, local-level samples, and pixel-level scale samples; S2: A hierarchical progressive fusion network model is constructed, and the hierarchical progressive fusion network model includes a global information branch, a local feature extraction branch, and a pixel-level feature extraction branch; S3: The hierarchical progressive fusion network model is trained using a total loss function to obtain a trained hierarchical progressive fusion network model; S4: The test set is input into the trained hierarchical progressive fusion network model to obtain the corresponding prediction classification results. The present invention retains the artificial weight allocation of the multi-branch network to flexibly adapt to actual classification scenarios with different requirements for spectral information and spatial information.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image classification, and in particular relates to a hierarchical progressive hyperspectral image classification method based on FKAN. Background Art

[0002] As a commonly used method of earth observation, hyperspectral remote sensing imaging technology has extremely high spectral resolution. By capturing hundreds or thousands of continuous spectral bands, hyperspectral remote sensing images can characterize the complex optical characteristics of objects in the spectral range of visible and non-visible light. Therefore, hyperspectral images have been widely used in ecological monitoring, urban planning, smart agriculture, mineral exploration and other fields, especially in the task of object classification, showing unique advantages.

[0003] In the early research work on hyperspectral image classification, researchers focused on the mining of pixel-level features. However, using only pixel-level features for hyperspectral image classification will lead to salt and pepper noise and low classification accuracy. Existing hyperspectral image classification methods introduce spatial information. While achieving high-precision classification, their reliance on spatial information brings new problems such as over-smoothing and high computational costs. Since existing models mostly use cross-entropy loss functions for supervised learning, the models pay more attention to the classification accuracy of labeled target samples and ignore the ability to distinguish different categories of samples in the background. This further exacerbates the transition smoothing phenomenon. In order to balance the spatial-spectral information, a variety of models that adaptively explore the weights of the two types of information have been proposed. However, the clarity of the background boundary is often not measured by the classification accuracy of the target. Adaptive algorithms that rely only on category labels as supervisory signals are difficult to cope with different practical application requirements.

[0004] In addition, more complete information representation and adding an independent spectral information extraction module to the model can make the boundaries of objects more accurate. Extracting spatial spectrum information separately in the model means that independent learning is required at more scales, which increases the computational cost of the model itself. Moreover, features between different scales have potential information redundancy and interference, and direct fusion will destroy the features extracted by each branch to a certain extent. To this end, researchers have proposed a variety of improvement methods. Researchers have tried to optimize the classification results through adaptive weight adjustment and late fusion, but they still face mutually restrictive problems and challenges in terms of complete information representation, computational cost, cross-scale fusion and boundary clarity. Summary of the invention

[0005] In view of this, the present invention aims to provide a hierarchical progressive hyperspectral image classification method based on FKAN to solve the mutual constraints faced by the prior art in terms of complete information representation, computational cost, cross-scale fusion and boundary clarity. The present invention proposes a technical solution for feature interaction and fusion from pixel level to local level and then to global level. Through precise pairing and guided fusion, the information conflict and information redundancy in traditional cross-scale fusion are reduced, and the artificial weight allocation of multi-branch networks is retained to flexibly adapt to actual classification scenarios with different requirements for spectral information and spatial information.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0007] A hierarchical progressive hyperspectral image classification method based on FKAN specifically includes the following steps:

[0008] S1: After dividing the labeled hyperspectral image dataset into a training set and a test set, the training set is processed to obtain global level samples, local level samples, and pixel level scale samples;

[0009] S2: construct a hierarchical progressive fusion network model, which includes a global information branch, a local feature extraction branch and a pixel-level feature extraction branch;

[0010] S3: inputting the global level samples into the global information branch, inputting the local level samples into the local feature extraction branch, inputting the pixel level scale samples into the pixel level feature extraction branch, and using the total loss function to train the hierarchical progressive fusion network model to obtain a trained hierarchical progressive fusion network model;

[0011] S4: Input the test set into the trained hierarchical progressive fusion network model to obtain the corresponding prediction classification results.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] S11: In a labeled hyperspectral image dataset, randomly select a pixels from the pixel set of each category; and randomly select one-half of the total number of pixels from the pixel set of the category whose pixel number is less than a;

[0014] S12: taking all pixels selected in step S11 as a training set, and taking all pixels in the hyperspectral image dataset except the training set as a test set;

[0015] S13: The hyperspectral image based on which the hyperspectral dataset is made is used as a global level sample;

[0016] The set of spectral vectors of each pixel is taken as pixel-level scale samples;

[0017] The target pixel and the neighboring pixels are used to form an image block to obtain local level samples.

[0018] Furthermore, in step S2, the hierarchical progressive fusion network model further includes a first FKAN module, which converts the global level features output by the global information branch into , local level features output by the local feature extraction branch And the pixel-level features output by the pixel-level feature extraction branch After weighting, the weighted results are stacked according to the dimension of the band to obtain the fusion sequence features. , the fusion sequence features Input to the first FKAN module for processing to obtain the predicted classification result;

[0019] The mathematical expression for obtaining the predicted classification result is:

[0020] ;

[0021] in, is the vector of predicted classification results, is a single-layer FKAN with cn nodes, c n is the number of categories, In order to stack the multi-channel feature maps into a single-channel feature map according to the dimension of the band, For pixel-level features The corresponding weights, For local level features The corresponding weights, For global level features The corresponding weight.

[0022] Furthermore, the global information branch includes a first-level encoder, a first-level decoder, a second-level encoder, a second-level decoder, a first 3D convolutional layer, a second 3D convolutional layer, a third 3D convolutional layer and a first softmax function. After the global level sample is encoded by the first-level encoder, a feature map is obtained. and feature map , the feature map Input to the second-stage decoder which is jump-connected to the first-stage encoder, and transform the feature map Input to the second-level encoder for encoding processing to obtain the feature map and feature map , the feature map Input to the first-stage decoder which is jump-connected to the second-stage encoder, and transform the feature map Input to the first 3D convolutional layer for processing to obtain the feature map , the feature map Input to the second 3D convolutional layer for processing to obtain the feature map , the feature map Perform dimension stacking to obtain feature maps , the feature map Input to the first level decoder for processing to obtain the feature map , the feature map Input to the second level encoder for processing to obtain the feature map , feature map The global level features are obtained by processing the third 3D convolution layer and the first softmax function. .

[0023] Furthermore, the mathematical expression of the processing process of the first-stage encoder and the second-stage encoder is:

[0024] ;

[0025] ;

[0026] in, is the i-th level encoder, is the input sample, W, H and B represent the length, width and number of bands of the input sample respectively, DT is the dimension conversion between the input features of two convolutional layers or hidden layers of different dimensions, is the feature map output by the i-th level encoder for jump connection to the corresponding decoder, is the feature map output by the i-th level encoder, is the d-dimensional convolution sum of i channels, For the maximum pooling process, is the activation function;

[0027] The mathematical expression of the processing process of the first-level decoder is:

[0028]

[0029] ;

[0030] in, is the first-level decoder, is bicubic upsampling, To stack the input features into a multi-channel feature map by channel, is the output feature of the first-level decoder, is the d-dimensional convolution sum of i channels, DT is the dimension conversion between the input features of two convolutional layers or hidden layers with different dimensions, The feature map ;

[0031] The processing of the second-stage decoder is the same as that of the first-stage decoder. The input of the second-stage decoder is the feature map and feature map .

[0032] Furthermore, the hierarchical progressive fusion network model also includes an ECM module, which combines global level samples and global level features Input to the ECM module for similarity constraint to obtain the LI loss function;

[0033] The global level features After standard Gaussian blur processing, the feature map is obtained ; Perform principal component analysis on the global level samples, retain the first principal component, and perform standard Gaussian blur processing on the first principal component to obtain the feature map ; The feature map and feature map Convolution operations are performed with the 0°sobel convolution operator, the 45°sobel convolution operator, the 90°sobel convolution operator, and the 135°sobel convolution operator in turn to obtain the corresponding feature maps and feature map , the feature map and feature map Input into the bandpass filter for processing, and obtain the corresponding feature map and feature map , the feature map and feature map The mean absolute error is used as the LI loss function;

[0034] The mathematical calculation process of the ECM module is:

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] in, is the k°sobel convolution operator, is the convolution operation, and They are used to control the bandpass range in the bandpass filter constructed by the activation function, ReLU() is the linear rectification function, NL() is the normalization function, is the loss function, The feature map or feature map The total number of elements of and Represents the feature maps and feature map The ith element of .

[0041] Furthermore, the pixel-level feature extraction branch includes a random dropout module, a first FKAN 1D convolution module, a second FKAN 1D convolution module, a flattening module, a second FKAN module, a third FKAN 1D convolution module, a fourth FKAN 1D convolution module and a second Softmax function. The pixel-level scale samples are processed by the random dropout module, the first FKAN 1D convolution module and the second FKAN 2D convolution module in sequence to obtain a feature map. , feature map After being processed by the flattening module and the second FKAN module, the sequence features are obtained , sequence features After being processed by the third FKAN 1D convolution module and the fourth FKAN 1D convolution module, the sequence features are obtained. , the sequence features and sequence characteristics Perform element-level addition operations to obtain sequence features , the sequence features Input to the second Softmax function for processing to obtain pixel-level features ;

[0042] The local feature extraction branch includes the first FKAN 3D convolution module, the second FKAN 3D convolution module, the FKAN-based spectral attention module, the FKAN-based spatial attention module, the third FKAN 3D convolution module, the FKAN 2D convolution module, the CSGFM module and the third Softmax function, which converts the local level samples The images are input into the first FKAN 3D convolution module and the second FKAN 3D convolution module for processing, and the corresponding feature maps are obtained. and feature map , the feature map Input to the FKAN-based spectral attention module for processing to obtain the feature map , the feature map Input to the FKAN-based spatial attention module for processing to obtain the feature map , the feature map and feature map Cascade, and input the feature map obtained after cascading into the third FKAN 3D convolution module for processing to obtain sequence features , sequence features and sequence characteristics , the feature map , feature map and feature map Correspondence and sequence characteristics , sequence features and sequence characteristics After cascading, it is input into the CSGFM module for processing to obtain the fused feature map K and the fused feature map V. The channel is stacked three times to obtain the fused feature map G, and the fused feature map G is input into the CSGFM module. The fused feature map G is multiplied by the transposed matrix of the fused feature map K to obtain the weight feature. The weight feature is multiplied by the fused feature map V to obtain the fused feature F output by the CSGFM module. The fused feature F is processed by the FKAN 2D convolution module and the third Softmax function in turn to obtain the local level feature. ;

[0043] Sequence features and sequence characteristics Perform element-level addition operations to obtain sequence features The mathematical expression is:

[0044] ;

[0045] in, is the d-dimensional convolution operation based on c channels of the FKAN neural network layer, and DT is the dimensionality conversion between the input features of two convolutional layers or hidden layers of different dimensions.

[0046] Furthermore, the mathematical expression of the FKAN-based spectral attention module is:

[0047] ;

[0048] ;

[0049] in, is the maximum pooling operation, is the average pooling operation, is the d-dimensional convolution operation based on the c channels of the FKAN neural network layer, DT is the dimensional conversion between the input features of two convolutional layers or hidden layers of different dimensions, To stack the input features into a multi-channel feature map by channel, The feature map , The feature map , It is the intermediate feature map, which is composed of the cascade of the feature map after the maximum pooling of the spectral attention module based on FKAN and the feature map after the average pooling;

[0050] The mathematical expression of the spatial attention module based on FKAN is:

[0051] ;

[0052] ;

[0053] in, The feature map , The feature map , It is an intermediate feature map, which is composed of the cascade of the feature map after the maximum pooling and the feature map after the average pooling based on the spatial attention module of FKAN.

[0054] Furthermore, the feature map , feature map and feature map Correspondence and sequence characteristics , sequence features and sequence characteristics The mathematical expression for processing after cascading is:

[0055] ;

[0056] ;

[0057] in, represents a function that expands the input features into one dimension, is a single-layer FKAN neural network layer with 64 nodes. is the feature after the i-th cascade, Because of the characteristics The generated i-th feature vector Key, Reason The generated ith eigenvector Value, is the d-dimensional convolution operation based on the c channels of the FKAN neural network layer, and DT is the dimensionality conversion between the input features of two convolutional layers or hidden layers of different dimensions;

[0058] The three sets of feature vectors Key and the three sets of feature vectors Value are cascaded by channel to obtain the fused feature maps K and V.

[0059] Furthermore, the total loss function for:

[0060] ;

[0061] in ， m is the number of samples to be classified, for The corresponding true label, is the L1 loss function, For control range, This is the classification result predicted by the hierarchical progressive fusion network model.

[0062] Compared with the prior art, the invention can achieve the following beneficial effects:

[0063] The present invention creates the FKAN-based hierarchical progressive hyperspectral image classification method. First, a hierarchical progressive fusion network model is constructed, and by introducing the FKAN neural network layer (Fast Kolmogorov-ArnoldNetworks) into the model, the model's fitting ability for nonlinear features is enhanced, complete information representation is fully considered, and the model's lightweight is taken into account; second, the present invention designs a global feature extraction network (i.e., a global information branch) based on Sobel operator edge constraints, which captures high-level semantic features through global long-distance dependencies and strengthens edge information. In the ECM module, an edge constraint algorithm is first constructed, and only one principal component is retained after principal component analysis of the original hyperspectral image, and then the image is fed into the edge constraint algorithm to obtain an edge label. In addition, the classification result graph of the global information branch is fed into the algorithm to obtain predicted edge data, and similarity constraints are performed based on the predicted edge data and the constructed edge labels, thereby alleviating the potential risk of over-smoothing while maintaining the classification accuracy; Fourth, the present invention proposes a technical solution for feature interaction and fusion from pixel level to local level and then to global level. Through precise pairing and guided fusion, the information conflict and information redundancy in traditional cross-scale fusion are alleviated, and the artificial weight distribution of multi-branch networks is retained to flexibly adapt to actual classification scenarios with different requirements for spectral information and spatial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0065] Figure 1 A schematic diagram of a process flow of a hierarchical progressive hyperspectral image classification method based on FKAN according to an embodiment of the present invention;

[0066] Figure 2A schematic diagram of the structure of a hierarchical progressive fusion network model according to an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of the structure of the global information branch described in the embodiment of the present invention;

[0068] Figure 4 A schematic diagram of an edge constraint algorithm based on a Sobel operator according to an embodiment of the present invention;

[0069] Figure 5 A schematic diagram of the structure of the FKAN-based spectral attention module described in the embodiment of the present invention;

[0070] Figure 6 A schematic diagram of the structure of the spatial attention module based on FKAN according to an embodiment of the present invention;

[0071] Figure 7 A schematic diagram of the structure of the CSGFM module described in the embodiment of the present invention;

[0072] Figure 8 This is a schematic diagram comparing the classification results of the embodiments of the present invention on the UP dataset and the IP dataset with other cutting-edge methods. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0074] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0075] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0076] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0077] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0078] like Figure 1 As shown, the hierarchical progressive hyperspectral image classification method based on FKAN proposed in the present invention comprises the following steps:

[0079] S1: After dividing the labeled hyperspectral image dataset (which can be a UP dataset or an IP dataset) into a training set and a test set, the training set is processed to obtain global level samples, local level samples, and pixel level scale samples;

[0080] S11: In a labeled hyperspectral image dataset, randomly select a pixels from the pixel set of each category; and randomly select one-half of the total number of pixels from the pixel set of the category whose pixel number is less than a;

[0081] S12: all pixels selected in step S11 are used as training sets, and all pixels in the hyperspectral image dataset except the training set are used as test sets, where the test sets are used to evaluate the classification accuracy of the hierarchical progressive fusion network model;

[0082] S13: using the hyperspectral image based on which the hyperspectral dataset is prepared as a global-level sample and using a set of spatial position indexes of each pixel of the global-level sample to train and test the hierarchical progressive fusion network model;

[0083] The set of spectral vectors of each pixel is taken as pixel-level scale samples;

[0084] The target pixels and neighboring pixels are used to form an image block. The length and width of the image block are 7×7, and the height of the image block is the number of 16 bands retained after dimensionality reduction. Local-level samples are obtained. The principal component analysis method is used for dimensionality reduction, and data enhancement is performed through random vertical and horizontal flipping.

[0085] S2: Construct a hierarchical progressive fusion network model, which includes a global information branch, a local feature extraction branch and a pixel-level feature extraction branch.

[0086] S3: Input the global level samples into the global information branch, input the local level samples into the local feature extraction branch, input the pixel level scale samples into the pixel level feature extraction branch, and use the total loss function to train the hierarchical progressive fusion network model to obtain a trained hierarchical progressive fusion network model.

[0087] S4: Input the test set into the trained hierarchical progressive fusion network model to obtain the corresponding prediction classification results.

[0088] The test set is processed to obtain global-level samples, local-level samples, and pixel-level scale samples corresponding to the test set, which are then fed into the trained hierarchical progressive fusion network model to obtain the corresponding prediction classification results.

[0089] like Figure 2 As shown in the figure, the hierarchical progressive fusion network model includes a global information branch, a local feature extraction branch, and a pixel-level feature extraction branch. The training set samples of each scale are fed into different network feature extraction branches respectively, and the cross-scale information interaction and feature fusion are completed through the CSGFM module. Finally, the output results of each network branch are stacked according to different weights and fed into a single-layer FKAN network to obtain the final classification result. These branches are introduced in detail below.

[0090] For the global information branch, we construct a U-Net network based on the edge constraint algorithm of the Sobel operator to avoid over-smoothing. The U-Net network is a common full-batch input and output model with good spatial feature extraction capabilities, but it is often accompanied by a large amount of computation. This amount of computation is directly related to the number of channels and model depth of the U-Net network. In order to balance the lightweight of the model and the over-smoothing problem that is prone to occur in global information dependence, we use a two-layer, two-channel 3D convolution method to construct the U-Net network with a small number of learnable parameters, and add an edge constraint algorithm to enhance the edge information in a supervised manner.

[0091] The global information branch first encodes the input features twice, and after conversion, it is fed into the decoder twice, and finally the classifier outputs the feature map of this branch. In addition, this network branch also includes an edge constraint algorithm based on the Sobel operator, which extracts edges from the label map and the prediction map respectively and constructs the L1 loss to alleviate overfitting. The feature map output by the global information branch is Gaussian blurred and the edges are extracted in four directions (0°, 45°, 90°, 135°). The same operation is performed on the hyperspectral image that retains one principal component to obtain the edge label. The L1 loss is used to minimize the difference between the label edge and the prediction result edge, thereby strengthening the edge information of the classification result. Figure 3 As shown in the figure, the structures of the two encoders are the same, both of which contain a single-channel 3D convolution layer, a dual-channel 3D convolution layer and a maximum pooling layer. Each encoder outputs two feature maps, one of which passes information to the corresponding decoder in a skip connection manner to reduce information loss, and the other performs maximum pooling as the input of the next level encoder.

[0092] Specifically, the global information branch includes a first-level encoder, a first-level decoder, a second-level encoder, a second-level decoder, a first 3D convolution layer, a second 3D convolution layer, a third 3D convolution layer and a first softmax function. After the global level sample is encoded by the first-level encoder, a feature map is obtained. and feature map , the feature map Input to the second-stage decoder which is jump-connected to the first-stage encoder, and transform the feature map Input to the second-level encoder for encoding processing to obtain the feature map and feature map , the feature map Input to the first-stage decoder which is jump-connected to the second-stage encoder, and transform the feature map Input to the first 3D convolutional layer for processing to obtain the feature map , the feature map Input to the second 3D convolutional layer for processing to obtain the feature map , the feature map Perform dimension stacking to obtain feature maps , the feature map Input to the first level decoder for processing to obtain the feature map , the feature map Input to the second level encoder for processing to obtain the feature map , feature map The global level features are obtained by processing the third 3D convolution layer and the first softmax function. .

[0093] The mathematical expression of the processing process of the first-stage encoder and the second-stage encoder is:

[0094] ;

[0095] ;

[0096] in, is the i-th level encoder, is the input sample, W, H and B represent the length, width and number of bands of the input sample respectively, DT is the dimension conversion between the input features of two convolutional layers or hidden layers of different dimensions, is the feature map output by the i-th level encoder for jump connection to the corresponding decoder, is the feature map output by the i-th level encoder, is the d-dimensional convolution sum of i channels, For the maximum pooling process, is the activation function, The input features are stacked according to the dimension where the bands are located.

[0097] After the input sample is encoded twice, it is fed into the converter (including the first 3D convolution layer and the second 3D convolution layer). The input and output sizes of the converter are consistent, which is used to smooth the transition process between the encoder and the decoder and perform deeper feature extraction. Similar to the encoder, the middle layer also contains two convolution layers with the same number of channels as the encoder, and the results are stacked by channel. The difference is that the middle layer adds padding operations so as not to change the size of the features. The mathematical process is as follows:

[0098] ;

[0099] in, is the output feature map of the converter, which is used for subsequent decoding and CSGFM modules. In order to stack the multi-channel feature maps into a single-channel feature map according to the dimension of the band, is the d-dimensional volume sum of i channels, Represents a converter.

[0100] The decoder includes a bicubic upsampling operation and three single-channel 3D convolutional layers. In order to alleviate the potential risk of information loss in lightweight networks, the decoder refers to the skip connection structure, and its input consists of part of the output feature map of the corresponding encoder and the feature map output by the previous feature extraction unit. The mathematical expression of the processing process of the first-level decoder is:

[0101]

[0102] ;

[0103] in, is the first-level decoder, is bicubic upsampling, To stack the input features into a multi-channel feature map by channel, is the output feature of the first-level decoder, is the d-dimensional convolution layer of the i-th channel, and DT is the dimension conversion between the input features of two convolution layers or hidden layers with different dimensions;

[0104] The processing of the second-stage decoder is the same as that of the first-stage decoder. The input of the second-stage decoder is the feature map and feature map The output is fed into the classifier (a third 3D convolution layer with a single channel on one side and a first softmax function cascaded) to obtain global level features .

[0105] The extraction of global features can easily cause the model to be overly dependent on spatial information, resulting in over-smoothing. As the number of learnable parameters decreases, the network's ability to distinguish background pixels further decreases, which may aggravate the smoothing of the classification map. In order to alleviate this large-scale boundary coverage problem, the ECM module (edge ​​enhancement module based on multi-directional Sobel) is designed here and integrated into the propagation process and loss function to impose secondary constraints on the output image of the U-Net network. Specifically: the global level samples and global level features are combined into a single Input to the ECM module for similarity constraint, obtain the LI loss function, and improve the model's classification effect on boundaries. Figure 4 As shown in the figure, the ECM module is composed of a combination of multiple custom convolution kernels and a specific activation function. It builds labels with edge features based on the data itself, and realizes fine edge detection and enhancement of hyperspectral images. The specific process is as follows: the module first uses multiple predefined convolution kernels, including standard Sobel kernels (vertical and horizontal directions) and 45° and 135° rotated Sobel convolution kernels. After standard Gaussian blur processing, the feature map is obtained ; Perform principal component analysis on the global level samples, retain the first principal component, and perform standard Gaussian blur processing on the first principal component to obtain the feature map ; The feature map and feature map Convolution operations are performed with the 0°sobel convolution operator, the 45°sobel convolution operator, the 90°sobel convolution operator, and the 135°sobel convolution operator in turn to obtain the corresponding feature maps and feature map , the feature map and feature map Input into the bandpass filter for processing, and obtain the corresponding feature map and feature map , the feature map and feature map The mean absolute error of is used as the LI loss function. The algorithm can capture edge information in more directions, thereby improving the comprehensiveness and accuracy of the model edge detection. In addition, the feature data range is controlled by the normalization function, and a series of line activation functions are used for translation and nonlinear transformation to construct a bandpass filter to enhance the edge saliency and contrast and obtain the final result. The mathematical calculation process of the ECM module is:

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] in, is the k°sobel convolution operator, is the convolution operation, and They are used to control the bandpass range in the bandpass filter constructed by the activation function, ReLU() is the linear rectification function, NL() is the normalization function, is the loss function, The feature map or feature map The total number of elements of and Represents the feature maps and feature map The ith element of .

[0112] Finally, the output of the network consists of three parts. The first is the global information branch prediction classification result, which is used in the final fusion of multiple branches. The second is that some feature data of the encoder and converter are used to form part of the input of the CSGFM module to achieve a precise pairing fusion method. The third is the result of optimizing the global information branch prediction through the L1 loss function. Compared with data-extracted labels (feature maps and feature map ) to constrain the edge of the prediction result and alleviate the potential over-smoothing phenomenon in the U-Net network.

[0113] The pixel-level feature extraction branch includes a random dropout module, a first FKAN 1D convolution module, a second FKAN 2D convolution module, a flattening module, a second FKAN module, a third FKAN 1D convolution module, a fourth FKAN 1D convolution module, and a second Softmax function. The pixel-level scale samples are processed by the random dropout module, the first FKAN 1D convolution module, and the second FKAN 2D convolution module in sequence to obtain a feature map. , feature map After being processed by the flattening module and the second FKAN module, the sequence features are obtained , sequence features After being processed by the third FKAN 1D convolution module and the fourth FKAN 1D convolution module, the sequence features are obtained. , the sequence features and sequence characteristics Perform element-level addition operations to obtain sequence features , the sequence features Input to the second Softmax function for processing to obtain pixel-level features .

[0114] A pixel-level feature extraction network branch based on FKAN convolution is constructed and used as the pixel-level feature extraction branch. The input of this network branch is pixel-level scale samples. First, the spectral sequence of pixel-level scale samples is randomly inactivated to alleviate the potential overfitting risk of FKAN, and then it is fed into two one-dimensional convolutional layers based on FKAN to obtain the feature map. The propagation process from layer l to layer L+1 in FKAN convolution is described as:

[0115] ;

[0116] in, = is the feature sequence of the l-th layer FKAN, is the feature vector of the lth layer, is the number of n nodes in the lth layer, for The pth row of are the n residual activation functions corresponding to the p-th node in the l-th layer for the l+1-th layer.

[0117] Activation Function The calculation process is as follows:

[0118] ;

[0119] in, is the sigmoid activation function. is the radial basis function, is the number of the i-th grid point, and They are and The learnable weights, is the learnable bias and NL() is the normalization function.

[0120] The L-layer FKAN network can be expressed as:

[0121] ;

[0122] in, Represents nested operators for inner and outer functions.

[0123] On this basis, the convolution process based on FKAN (taking three-dimensional convolution as an example) is as follows:

[0124] ;

[0125] in, Representation function matrix The index in is ( ), M, N, and T are used to represent the size of the convolution kernel. Represents the three-dimensional index of the input three-dimensional feature ( ) corresponding data, Represents the convolution kernel of FKAN convolution.

[0126] Will After expansion, it is fed into a single-layer FKAN network to obtain sequence features , and then Feed two layers of one-dimensional FKAN convolutional layers to obtain sequence features . Further, the sequence features and sequence characteristics Perform element-level addition operations to obtain sequence features The mathematical expression is:

[0127] ;

[0128] in, is the d-dimensional convolution operation based on c channels of the FKAN neural network layer, and DT is the dimensionality conversion between the input features of two convolutional layers or hidden layers of different dimensions.

[0129] Sequence characteristics The second softmax output is the pixel-level features of the same number of nodes as the number of categories in the dataset , and fuse the outputs of other branches according to the weights for the final classification prediction. In addition, the sequence features It is also used to construct the input of the subsequent CSGFM module.

[0130] The input samples of the local feature extraction branch are patch-level samples. , After two FKAN-based three-dimensional convolutions, feature maps are generated and . Spectral attention module and spatial attention module are constructed to and Strengthen information.

[0131] The local feature extraction branch includes the first FKAN 3D convolution module, the second FKAN 3D convolution module, the spectral attention module, the spatial attention module, the third FKAN 3D convolution module, the FKAN 2D convolution module, the CSGFM module and the third Softmax function, which converts the local level samples The images are input into the first FKAN 3D convolution module and the second FKAN 3D convolution module for processing, and the corresponding feature maps are obtained. and feature map , the feature map Input to the spectral attention module for processing to obtain the feature map , the feature map Input to the spatial attention module for processing to obtain the feature map , the feature map and feature map Cascade, and input the feature map obtained after cascading into the third FKAN 3D convolution module for processing to obtain sequence features , sequence features and sequence characteristics , the feature map , feature map and feature map Correspondence and sequence characteristics , sequence features and sequence characteristics After cascading, it is input into the CSGFM module for processing to obtain the fused feature map K and the fused feature map V. The channel is stacked three times to obtain the fused feature map G, and the fused feature map G is input into the CSGFM module. The fused feature map G is multiplied by the transposed matrix of the fused feature map K to obtain the weight feature. The weight feature is multiplied by the fused feature map V to obtain the fused feature F output by the CSGFM module. The fused feature F is processed by the FKAN 2D convolution module and the third Softmax function in turn to obtain the local level feature. .

[0132] The FKAN convolution operation required by the local feature extraction branch and the pixel-level feature extraction branch is similar to the convolution operation of CNN, but its convolution kernel is composed of spline functions, and the parameters learned in the convolution process are the coefficients of the spline basis function, the coefficients and bias of the activation function, and the convolution operation on the feature map is completed in the form of a sliding window.

[0133] like Figure 5 As shown in the FKAN-based spectral attention module, the feature map Perform three-dimensional FKAN convolution operation to obtain sequence features , respectively, for sequence features After calculating the maximum pooling and average pooling, the obtained feature maps are cascaded. The cascade result is Feed two layers of one-dimensional FKAN convolution to the weight feature map, which is Multiplying together finally gives the output of the spectral attention module , the mathematical expression of the FKAN-based spectral attention module is:

[0134] ;

[0135] ;

[0136] in, is the maximum pooling operation, is the average pooling operation, is the d-dimensional convolution operation based on the c channels of the FKAN neural network layer, DT is the dimensional conversion between the input features of two convolutional layers or hidden layers of different dimensions, To stack the input features into a multi-channel feature map by channel, The feature map , The feature map , is the cascade result;

[0137] like Figure 6 As shown, for the feature map Perform a three-dimensional FKAN convolution operation to obtain the feature map , respectively, for the feature maps After calculating the maximum pooling and average pooling, the obtained feature maps are cascaded. The cascade result is Feed two layers of one-dimensional FKAN convolution to the weight feature map, which is Multiplying together finally gives the output of the spectral attention module , the mathematical expression of the spatial attention module based on FKAN is:

[0138] ;

[0139] ;

[0140] in, The feature map , The feature map , Cascade result.

[0141] What you get and Cascade and input into a three-channel three-dimensional FKAN convolution layer to obtain three sequence features , , Used in subsequent CSGFM modules.

[0142] like Figure 7 As shown in the figure, the input of the CSGFM module includes three parts, one of which is the output features of each level of encoder in the global information branch and the intermediate characteristics of the converter , and the second is the three feature sequences output in the local feature extraction branch , , , and the third is the output feature map in the pixel-level feature extraction branch For a single propagation, the spatial coordinates corresponding to the pixel-level scale samples are retained and the original image size is The size scaling relationship is determined by the spatial coordinates. Each coordinate position on the feature map corresponds to a row of feature vectors, which are recorded as .

[0143] Secondly, and , , Cascading enables accurate pairing of local-level features, global-level features, and pixel-level features under encoding and transformation at different scales. Afterwards, the cascaded features are subjected to a series of one-dimensional convolution operations based on FKAN convolution in the CSGFM module to further mine information and generate feature vectors Key and Value. The mathematical expression of this process is as follows:

[0144] ;

[0145] ;

[0146] in, represents a function that expands the input features into one dimension, is a single-layer FKAN neural network layer with 64 nodes. is the feature after the i-th cascade, Because of the characteristics The generated i-th feature vector Key, Reason The generated ith eigenvector Value, is the d-dimensional convolution operation based on c channels of the FKAN neural network layer, and DT is the dimensionality conversion between the input features of two convolutional layers or hidden layers of different dimensions.

[0147] The three sets of feature vectors Key and the three sets of feature vectors Value are cascaded by channel to obtain the fused feature map K and the fused feature map V.

[0148] The feature map The channel is stacked three times to obtain the fused feature map G, and the fused feature map G is input into the CSGFM module. The fused feature map G is multiplied by the transposed matrix of the fused feature map K to obtain the weight feature. The weight feature is multiplied by the fused feature map V to obtain the fused feature F output by the CSGFM module. The fused feature F is processed by the FKAN 2D convolution module and the third Softmax function in turn to obtain the local level feature. , its mathematical expression is:

[0149]

[0150] Among them, cn represents the number of object categories in the dataset. , K, and V are all fusion feature maps.

[0151] The hierarchical progressive fusion network model also includes a first FKAN module, which converts the global level features output by the global information branch into , local level features output by the local feature extraction branch And the pixel-level features output by the pixel-level feature extraction branch After weighting, the weighted results are concatenated to obtain the fusion sequence features. , the fusion sequence features Input to the first FKAN module for processing to obtain the predicted classification result;

[0152] The mathematical expression for obtaining the predicted classification result is:

[0153] ;

[0154] in, is the vector of predicted classification results, is a single-layer FKAN with cn nodes, c n is the number of categories, In order to stack the multi-channel feature maps into a single-channel feature map according to the dimension of the band, For pixel-level features The corresponding weights, For local level features The corresponding weights, For global level features The corresponding weight.

[0155] The cross entropy loss function is constructed based on the prediction results, and the cross entropy loss function is combined with the L1 loss function for boundary constraints to construct the total loss function. for:

[0156] ;

[0157] Among them, m is the number of samples to be classified, for The corresponding true label, is the L1 loss function, For control range.

[0158] In addition, considering that too large L1 loss may affect classification accuracy, The value of will automatically decay, that is, every time the cross entropy loss increases three times, will decay to its original times, Different settings are used in different datasets. The default setting is 0.8.

[0159] After constructing the total loss function, the prediction results of the forward propagation are used to optimize the total loss function, thereby forming a hierarchical progressive fusion network model.

[0160] The training set is fed into the hierarchical progressive fusion network model and the total loss function is optimized through iteration. The number of training iterations is 150, and the optimizer is Adam ( =0.9, ), batch size is 256, and initial learning rate is 0.005. In the process of optimizing the total loss function, the learnable parameters of the model are updated so that the classification accuracy of the model's training set is gradually improved. In each generation of training, the validation set samples are fed into the network to verify its classification performance for untrained samples, and the hyperparameters are adjusted based on the results. After the classification accuracy of the validation set reaches the highest, training is stopped. The test set samples are fed into the trained hierarchical progressive fusion network model, and the output is the final predicted classification result.

[0161] The three sequence features output from the local feature extraction branch are cascaded with the features of the corresponding spatial positions in the encoder and converter in the global information branch, and are fed into two FKAN-based convolutional neural networks to obtain the feature Key and feature Value. The output feature sequence of the pixel-level feature extraction branch is multiplied by the Key and Value to obtain the fusion result of the three branches, which is fed into a single-layer FKAN network as the output of the local feature extraction network.

[0162] like Figure 8 As shown, the upper two rows are UP data sets, and the lower two rows are IP data sets. In the figure, (a)-(j) are: (a) SVM, (b) MVAHN, (c) MRViT, (d) DSGSF, (e) FM, (f) Mamba, (g) HKAN, (h) MLP, (i) HPFN, (j) Ground truth, the hierarchical progressive fusion network model of the present invention is abbreviated as HPFN, and the other existing uncommon networks are: MVANH (Multiple vision architectures-based hybrid network), MRViT (Mixed residual convolutions with vision transformer, mixed residual convolution ViT), DSGSF (Dualview spectral and global spatial feature fusion network, dual view spectral and global spatial feature fusion network), FM (full model, complete information model), HKAN (hybrid KAN, hybrid KAN network).

[0163] The present invention can achieve good classification results on both data sets.

[0164] The following table shows the comparison of the classification indicators of the present invention (HPFN) with other cutting-edge methods on the UP dataset and the IP dataset, including the average accuracy (AA), the overall accuracy (OA), and kappa, which are indicators for evaluating the classification performance of the model. As can be seen from the table below, the present method has the best average accuracy.

[0165] Table 1

[0166]

[0167] Table 2

[0168]

[0169] The following table shows the comparison results of the efficiency indicators of the present invention (HPFN) and other cutting-edge methods on the UP data set and the IP data set. It can be seen from the table that the present method has lower computational complexity and fewer parameters than other models. Flops in the table is the number of floating-point operations per second.

[0170] Table 3

[0171]

[0172] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0173] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A hierarchical progressive hyperspectral image classification method based on FKAN, characterized by: The specific steps include: S1: After dividing the labeled hyperspectral image dataset into a training set and a test set, the training set is processed to obtain global level samples, local level samples, and pixel level scale samples; S2: constructing a hierarchical progressive fusion network model, wherein the hierarchical progressive fusion network model includes a global information branch, a local feature extraction branch, and a pixel-level feature extraction branch; The hierarchical progressive fusion network model also includes a first FKAN module, which converts the global level features output by the global information branch into , local level features output by the local feature extraction branch And the pixel-level features output by the pixel-level feature extraction branch After weighting, the weighted results are stacked according to the dimension of the band to obtain the fusion sequence features. , the fusion sequence features Input to the first FKAN module for processing to obtain the predicted classification result; The mathematical expression for obtaining the predicted classification result is: ； in, is the vector of predicted classification results, is a single-layer FKAN with cn nodes, c n is the number of categories, In order to stack the multi-channel feature maps into a single-channel feature map according to the dimension of the band, For pixel-level features The corresponding weights, For local level features The corresponding weights, For global level features The corresponding weights; S3: inputting the global level samples into the global information branch, inputting the local level samples into the local feature extraction branch, inputting the pixel level scale samples into the pixel level feature extraction branch, and using the total loss function to train the hierarchical progressive fusion network model to obtain a trained hierarchical progressive fusion network model; S4: Input the test set into the trained hierarchical progressive fusion network model to obtain the corresponding prediction classification results.

2. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: In a labeled hyperspectral image dataset, randomly select a pixels from the pixel set of each category; and randomly select one-half of the total number of pixels from the pixel set of the category whose pixel number is less than a; S12: taking all pixels selected in step S11 as a training set, and taking all pixels in the hyperspectral image dataset except the training set as a test set; S13: The hyperspectral image based on which the hyperspectral dataset is made is used as a global level sample; The set of spectral vectors of each pixel is taken as pixel-level scale samples; The target pixel and the neighboring pixels are used to form an image block to obtain local level samples.

3. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 1, characterized in that: The global information branch includes a first-level encoder, a first-level decoder, a second-level encoder, a second-level decoder, a first 3D convolution layer, a second 3D convolution layer, a third 3D convolution layer and a first softmax function. After the global level sample is encoded by the first-level encoder, a feature map is obtained. and feature map , the feature map Input to the second stage decoder which is jump-connected with the first stage encoder, and convert the feature map Input to the second-level encoder for encoding processing to obtain the feature map and feature map , the feature map Input to the first stage decoder which is jump-connected with the second stage encoder, and convert the feature map Input to the first 3D convolutional layer for processing to obtain the feature map , the feature map Input to the second 3D convolutional layer for processing to obtain the feature map , the feature map Perform dimension stacking to obtain feature maps , the feature map Input to the first level decoder for processing to obtain the feature map , the feature map Input to the second level encoder for processing to obtain the feature map , feature map The global level features are obtained by processing the third 3D convolution layer and the first softmax function. .

4. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 3 is characterized by: The mathematical expression of the processing process of the first-stage encoder and the second-stage encoder is: ； ； in, is the i-th level encoder, is the input sample, W, H and B represent the length, width and number of bands of the input sample respectively, DT is the dimension conversion between the input features of two convolutional layers or hidden layers of different dimensions, is the feature map output by the i-th level encoder for jump connection to the corresponding decoder, is the feature map output by the i-th level encoder, is the d-dimensional convolution sum of i channels, For the maximum pooling process, is the activation function; The mathematical expression of the processing process of the first-level decoder is: ； in, is the first-level decoder, is bicubic upsampling, To stack the input features into a multi-channel feature map by channel, is the output feature of the first-level decoder, is the d-dimensional convolution sum of i channels, DT is the dimension conversion between the input features of two convolutional layers or hidden layers with different dimensions, The feature map ; The processing of the second-stage decoder is the same as that of the first-stage decoder. The input of the second-stage decoder is the feature map and feature map .

5. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 3 is characterized in that: The hierarchical progressive fusion network model also includes an ECM module, which combines global level samples and global level features. Input to the ECM module for similarity constraint to obtain the LI loss function; The global level features After standard Gaussian blur processing, the feature map is obtained ; Perform principal component analysis on the global level samples, retain the first principal component, and perform standard Gaussian blur processing on the first principal component to obtain the feature map ; The feature map and feature map Convolution operations are performed with the 0°sobel convolution operator, the 45°sobel convolution operator, the 90°sobel convolution operator, and the 135°sobel convolution operator in turn to obtain the corresponding feature maps and feature map , the feature map and feature map Input into the bandpass filter for processing, and obtain the corresponding feature map and feature map , the feature map and feature map The mean absolute error is used as the LI loss function; The mathematical calculation process of the ECM module is: ； ； ； ； ； in, is the k°sobel convolution operator, is the convolution operation, and They are used to control the bandpass range in the bandpass filter constructed by the activation function, ReLU() is the linear rectification function, NL() is the normalization function, is the loss function, The feature map or feature map The total number of elements of and Represents the feature maps and feature map The ith element of .

6. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 1, characterized in that: The pixel-level feature extraction branch includes a random dropout module, a first FKAN 1D convolution module, a second FKAN 1D convolution module, a flattening module, a second FKAN module, a third FKAN 1D convolution module, a fourth FKAN 1D convolution module and a second Softmax function. The pixel-level scale sample is processed by the random dropout module, the first FKAN 1D convolution module and the second FKAN 2D convolution module in sequence to obtain a feature map. , the feature map After being processed by the flattening module and the second FKAN module, the sequence features are obtained , sequence features After being processed by the third FKAN 1D convolution module and the fourth FKAN 1D convolution module, the sequence features are obtained. , the sequence features and sequence characteristics Perform element-level addition operations to obtain sequence features , the sequence features Input to the second Softmax function for processing to obtain pixel-level features ; The local feature extraction branch includes a first FKAN 3D convolution module, a second FKAN 3D convolution module, a FKAN-based spectral attention module, a FKAN-based spatial attention module, a third FKAN 3D convolution module, a FKAN 2D convolution module, a CSGFM module and a third Softmax function, which converts the local level samples The images are input into the first FKAN 3D convolution module and the second FKAN 3D convolution module for processing, and the corresponding feature maps are obtained. and feature map , the feature map Input to the FKAN-based spectral attention module for processing to obtain the feature map , the feature map Input to the FKAN-based spatial attention module for processing to obtain the feature map , the feature map and feature map Cascade, and input the feature map obtained after cascading into the third FKAN 3D convolution module for processing to obtain sequence features , sequence features and sequence characteristics , the feature map , feature map and feature map Correspondence and sequence characteristics , sequence features and sequence characteristics After cascading, it is input into the CSGFM module for processing to obtain the fused feature map K and the fused feature map V. The channel is stacked three times to obtain the fused feature map G, and the fused feature map G is input into the CSGFM module. The fused feature map G is multiplied by the transposed matrix of the fused feature map K to obtain the weight feature. The weight feature is multiplied by the fused feature map V to obtain the fused feature F output by the CSGFM module. The fused feature F is processed by the FKAN 2D convolution module and the third Softmax function in turn to obtain the local level feature. ; Sequence features and sequence characteristics Perform element-level addition operations to obtain sequence features The mathematical expression is: ； in, is the d-dimensional convolution operation based on c channels of the FKAN neural network layer, and DT is the dimensionality conversion between the input features of two convolutional layers or hidden layers of different dimensions.

7. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 6, characterized in that: The mathematical expression of the FKAN-based spectral attention module is: ； ； in, is the maximum pooling operation, is the average pooling operation, is the d-dimensional convolution operation based on the c channels of the FKAN neural network layer, DT is the dimensional conversion between the input features of two convolutional layers or hidden layers of different dimensions, To stack the input features into a multi-channel feature map by channel, The feature map , The feature map , It is the intermediate feature map, which is composed of the cascade of the feature map after the maximum pooling and the feature map after the average pooling of the spectral attention module based on FKAN; The mathematical expression of the spatial attention module based on FKAN is: ； ； in, The feature map , The feature map , It is an intermediate feature map, which is composed of the cascade of the feature map after the maximum pooling and the feature map after the average pooling based on the spatial attention module of FKAN.

8. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 4, characterized in that: The feature map , feature map and feature map Correspondence and sequence characteristics , sequence features and sequence characteristics The mathematical expression for processing after cascading is: ； ； in, represents a function that expands the input features into one dimension, is a single-layer FKAN neural network layer with 64 nodes. is the feature after the i-th cascade, Because of the characteristics The generated i-th feature vector Key, Reason The generated ith eigenvector Value, is the d-dimensional convolution operation based on the c channels of the FKAN neural network layer, and DT is the dimensionality conversion between the input features of two convolutional layers or hidden layers of different dimensions; The three sets of feature vectors Key and the three sets of feature vectors Value are cascaded by channel to obtain the fused feature maps K and V.

9. The hierarchical progressive hyperspectral image classification method based on FKAN according to claim 1, characterized in that: Total loss function for: ； in ， m is the number of samples to be classified, for The corresponding true label, is the L1 loss function, For control range, This is the classification result predicted by the hierarchical progressive fusion network model.

Images