Lightweight cross-domain small sample learning hyperspectral image classification method

Abstract

The invention provides a lightweight cross-domain small sample learning hyperspectral image classification method. The method comprises the following steps: extracting image features from a hyperspectral image to be classified; inputting the image features into a trained first neural network model to obtain a classification result of the hyperspectral image; the first neural network model is obtained by the following steps: inputting image features corresponding to a training sample into a three-dimensional depth separable convolution module and a spatial spectrum coordinate attention module which are connected in sequence to obtain first image features corresponding to the training sample; training a neural network model based on the first image features to obtain a trained neural network model; and performing M times of pruning on the trained neural network model based on a block pruning module, and selecting the pruned neural network model with the highest pruning score from the M pruned neural network models as a trained first neural network model.

Description

Technical Field

[0001] The present invention relates to the field of image detection, and in particular to a lightweight cross-domain small sample learning hyperspectral image classification method, which is a cross-domain small sample learning hyperspectral image classification method. Background Art

[0002] In recent years, neural networks have achieved remarkable results in real-world applications of hyperspectral image classification tasks, but they always face the problem of high computational cost. In order to improve the speed of these networks, researchers have proposed a variety of network compression methods, such as network pruning, network decoupling, and network quantization. However, most early methods rely on the original training set to restore the accuracy of the model. But in general, in order to protect data privacy and achieve rapid deployment of classification systems, there may be only a few hyperspectral images available as training data, and the current cross-domain small sample hyperspectral image classification algorithms are too complex. In most cross-domain small sample hyperspectral image classification methods, the network parameters are complex and the network has the problem of parameter redundancy. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a lightweight cross-domain small sample learning hyperspectral image classification method, which is used to solve the problems of complex network parameters and parameter redundancy in most current cross-domain small sample hyperspectral image classification methods.

[0004] According to a first aspect of the present invention, a cross-domain small sample learning hyperspectral image classification method is provided. The method of the present invention is a lightweight hyperspectral image classification method based on cross-domain small sample learning of block pruning, and the method comprises the following steps:

[0005] Step S1: extracting image features from the hyperspectral image to be classified;

[0006] Step S2: inputting the image features into the trained first neural network model to obtain the classification result of the hyperspectral image;

[0007] The first neural network model is obtained by:

[0008] Inputting the image feature corresponding to the training sample into the three-dimensional depth separable convolution module and the empty spectral coordinate attention module connected in sequence to obtain the first image feature corresponding to the training sample;

[0009] Training a neural network model based on the first image feature to obtain a trained neural network model;

[0010] The trained neural network model is pruned M times based on the block pruning module, and a pruned neural network model is obtained each time. The pruned neural network model with the highest pruning score is selected from the M pruned neural network models as the first trained neural network model.

[0011] Preferably, the three-dimensional depth-separable convolution module includes three three-dimensional convolution sub-modules connected in sequence, and the image features corresponding to the training samples are used as input data and input into the first three-dimensional convolution sub-module; each three-dimensional convolution sub-module performs three-dimensional channel-by-channel convolution and three-dimensional point-by-point convolution on its own input data to obtain its own corresponding output data, and the output data includes the spatial features and spectral features corresponding to the training samples; the output data of the first three-dimensional convolution sub-module is used as the input data of the second three-dimensional convolution sub-module, and the output data of the second three-dimensional convolution sub-module is used as the input data of the third three-dimensional convolution sub-module, and the output data of the third three-dimensional convolution sub-module is connected to the image features corresponding to the training samples, and a RELU operation is performed to obtain the output vector of the three-dimensional depth-separable convolution module.

[0012] Preferably, the three-dimensional convolution submodule includes a three-dimensional channel-by-channel convolution submodule and a three-dimensional point-by-point convolution submodule connected in sequence; the three-dimensional channel-by-channel convolution submodule includes a convolution layer having three first convolution kernels, each of which is used to perform a convolution operation on the data of a channel of the input data of the three-dimensional channel-by-channel convolution submodule to obtain the convolution results of each channel of the input data of the three-dimensional channel-by-channel convolution submodule; the three-dimensional point-by-point convolution submodule includes four second convolution kernels of size 1*1*d, where d is the depth value of the convolution result; each second convolution kernel performs a convolution operation on the convolution results of all channels, and the convolution operation of the second convolution kernel is a convolution operation on each position of the three-dimensional data under each channel.

[0013] Preferably, the spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and based on the output vector of the three-dimensional depth separable convolution module, obtains the attention vector in the horizontal direction and the attention vector in the vertical direction of the training sample, and obtains the channel attention vector of the training sample; based on the attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector of the training sample, obtains the fused attention vector of the training sample, and based on the fused attention vector, generates the first image feature corresponding to the training sample.

[0014] Preferably, the spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and obtains the attention vector of the training sample in the horizontal direction and the attention vector in the vertical direction based on the output vector of the three-dimensional depth separable convolution module, including:

[0015] The output vector X of the three-dimensional depth-separable convolution module is [x1, x2, ..., x C ]∈R C×H×W Perform global pooling, the global pooling formula is:

[0016]

[0017] where x1, x2, ..., x c are the features of the output vector of the three-dimensional depth-separable convolution module on each channel;

[0018] Two pooling kernels (H, 1) and (1, W) are used to encode the features on each channel along the horizontal and vertical coordinates, respectively, where H and W are the sizes of the hyperspectral image in the horizontal and vertical directions, respectively, and x c (i,j) represents the input with coordinates (i,j);

[0019] Therefore, the output of the cth channel at height h is:

[0020]

[0021] The output of the cth channel at width w is:

[0022]

[0023] Based on the output of each channel at each position of the training sample, obtaining the output of the training sample in the horizontal direction and the output of the training sample in the vertical direction;

[0024] The output of the training sample in the horizontal direction and the output in the vertical direction are concatenated and convolved to generate a first feature map; the first feature map is then activated, and the activated first feature map is split in the horizontal direction and the vertical direction to obtain the attention vector of the training sample in the horizontal direction and the attention vector in the vertical direction.

[0025] Preferably, the empty spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and obtains the channel attention vector of the training sample based on the output vector of the three-dimensional depth separable convolution module, including:

[0026] The output vector X of the three-dimensional depth-separable convolution module is [x1, x2, ..., xC ]∈R C×H×W Performing global average pooling and global maximum pooling to obtain a global average pooled vector and a global maximum pooled vector respectively, taking the global average pooled vector as a global feature, taking the global maximum pooled vector as a salient feature, subjecting the global average pooled vector to convolution, ReLU, and convolution, and obtaining a result as a first global feature; subjecting the global maximum pooled vector to convolution, ReLU, and convolution, and obtaining a result as a first salient feature; splicing the first global feature and the first salient feature, activating the spliced ​​result by a Sigmoid function, and obtaining a channel attention vector of the training sample;

[0027] The step of obtaining a fused attention vector of the training sample based on the attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector, and generating a first image feature corresponding to the training sample based on the fused attention vector includes:

[0028] The attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector of the training sample are weighted to obtain a fused attention vector of the training sample; the fused attention vector is multiplied by the image feature corresponding to the training sample to obtain a first image feature corresponding to the training sample.

[0029] Preferably, the block pruning module is used to remove a network block B of the trained neural network model. i , the pruned neural network model is recorded as The network block corresponds to a network layer of the neural network model; In the network block B in the trained neural network model, i Each of the previous network blocks is used as a front-end network block. For each front-end network block, an adapter is added at the output end of the network layer corresponding to the front-end network block as a new front-end network block. In the network block B in the trained neural network model, i Each subsequent network block is used as a successor network block. For each successor network block, an adapter is added to the input end of the network layer corresponding to the successor network block as a new successor network block; the adapter is a convolution layer with a convolution kernel of 1*1; establish a distance network block B i The output of the nearest new front network block is the distance network block B i The connection of the input of the nearest new successor network block and the calculation of the pruned neural network model The pruning score of

[0030] The block-based pruning module prunes the trained neural network model M times, each pruning obtains a pruned neural network model, and selects a pruned neural network model with the highest pruning score from the M pruned neural network models as the first trained neural network model, including:

[0031] Step S21: Prune the trained neural network model M times based on the block pruning module, and prune each pruned neural network model Do the following:

[0032] Calculate the recoverability R(B i ):

[0033]

[0034] Where θ is the parameter in the trained neural network model, x is the image feature, p(x) is the distribution of x, and E x～p(x) is the expected value of x when it is distributed according to the probability density function p(x), M o (x; θ) is a trained neural network model with parameters. is the square of the Frobenius norm, \bi is the value after removing the network block bi , α is the parameter in the adapter; A pruned neural network model with parameters and adapter parameters

[0035] Compute network block B i Delay time:

[0036]

[0037] Among them, latM o is the delay time of the trained neural network model, The neural network model after pruning Delay time;

[0038] Calculate the pruning score:

[0039]

[0040] Step S22: Selecting a pruned neural network model with the highest pruning score from the M pruned neural network models as the first trained neural network model.

[0041] According to a second aspect of the present invention, a cross-domain small sample learning hyperspectral image classification device is provided, the device comprising:

[0042] Feature extraction module: configured to extract image features from the hyperspectral image to be classified;

[0043] Classification module: configured to input the image features into the trained first neural network model to obtain the classification result of the hyperspectral image;

[0044] The first neural network model is obtained by:

[0045] Inputting the image feature corresponding to the training sample into the three-dimensional depth separable convolution module and the empty spectral coordinate attention module connected in sequence to obtain the first image feature corresponding to the training sample;

[0046] Training a neural network model based on the first image feature to obtain a trained neural network model;

[0047] The trained neural network model is pruned M times based on the block pruning module, and a pruned neural network model is obtained each time. The pruned neural network model with the highest pruning score is selected from the M pruned neural network models as the first trained neural network model.

[0048] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0049] A processor, which is used to execute multiple instructions;

[0050] A memory for storing a plurality of instructions;

[0051] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0052] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the aforementioned method.

[0053] In order to more effectively mine deep information in images, the present invention discloses a feature extraction module based on three-dimensional deep separable convolution. The module uses channel-by-channel and point-by-point convolution to replace traditional three-dimensional convolution, thereby reducing the number of model parameters and computational complexity. This method can capture the spatial-spectral information in cross-domain small sample hyperspectral image data without introducing a large number of parameters, achieve efficient spatial-spectral feature extraction, and make the network model more lightweight.

[0054] Hyperspectral images contain rich information, but when cross-domain small sample hyperspectral image algorithms describe images, there is a problem of insufficient extraction and description of spatial spectral features. The present invention discloses a spatial spectral coordinate attention mechanism module, which can learn the importance weights of cross-domain small sample hyperspectral image data in different bands and different spatial positions, focus on the bands and spatial positions that contribute to the task, allow the network to focus more on important features or areas, and can extract more important or more effective spatial spectral features under limited labeled samples, thereby reducing the number of redundant parameters in the network to a certain extent, helping to reduce the complexity and computational burden of the network, and further achieving network lightweighting.

[0055] In order to further solve the problem of high complexity of cross-domain small sample hyperspectral image classification models, the present invention discloses a block pruning module, which scores the pruned model according to the recoverability and delay time of the pruned network, and obtains the optimal block pruning model according to the score. Compared with the original network model, the pruned model reduces the number of algorithm parameters while ensuring the accuracy, thereby achieving a lightweight network structure.

[0056] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the present invention is described by providing the following accompanying drawings. In the accompanying drawings:

[0058] Figure 1 A schematic diagram of a cross-domain small sample learning hyperspectral image classification method according to an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a three-dimensional depth-separable convolution module according to an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a three-dimensional convolution submodule according to an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of data flow of a spatial spectral coordinate attention module according to an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the operation of a block pruning module according to an embodiment of the present invention;

[0063] Figure 6 A schematic diagram of pruning a block pruning module according to an embodiment of the present invention;

[0064] Figure 7 This is a structural block diagram of a cross-domain small sample learning hyperspectral image classification device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] First combine Figure 1 The present invention describes a cross-domain small sample learning hyperspectral image classification method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0066] Step S1: extracting image features from the hyperspectral image to be classified;

[0067] Step S2: inputting the image features into the trained first neural network model to obtain the classification result of the hyperspectral image;

[0068] The first neural network model is obtained by:

[0069] Inputting the image feature corresponding to the training sample into the three-dimensional depth separable convolution module and the empty spectral coordinate attention module connected in sequence to obtain the first image feature corresponding to the training sample;

[0070] Training a neural network model based on the first image feature to obtain a trained neural network model;

[0071] The trained neural network model is pruned M times based on the block pruning module, and a pruned neural network model is obtained each time. The pruned neural network model with the highest pruning score is selected from the M pruned neural network models as the first trained neural network model.

[0072] Furthermore, the three-dimensional depth-separable convolution module includes three three-dimensional convolution sub-modules connected in sequence, and the image features corresponding to the training samples are used as input data and input into the first three-dimensional convolution sub-module; each three-dimensional convolution sub-module performs three-dimensional channel-by-channel convolution and three-dimensional point-by-point convolution on its own input data to obtain its own corresponding output data, and the output data includes the spatial features and spectral features corresponding to the training samples; the output data of the first three-dimensional convolution sub-module is used as the input data of the second three-dimensional convolution sub-module, and the output data of the second three-dimensional convolution sub-module is used as the input data of the third three-dimensional convolution sub-module, and the output data of the third three-dimensional convolution sub-module is connected to the image features corresponding to the training samples, and a RELU operation is performed to obtain the output vector of the three-dimensional depth-separable convolution module.

[0073] The three-dimensional convolution submodule includes a three-dimensional channel-by-channel convolution submodule and a three-dimensional point-by-point convolution submodule connected in sequence; the three-dimensional channel-by-channel convolution submodule includes a convolution layer with three first convolution kernels, each of which is used to perform a convolution operation on the data of a channel of the input data of the three-dimensional channel-by-channel convolution submodule to obtain the convolution results of each channel of the input data of the three-dimensional channel-by-channel convolution submodule; the three-dimensional point-by-point convolution submodule includes four second convolution kernels with a size of 1*1*d, d is the depth value of the convolution result; each second convolution kernel performs a convolution operation on the convolution results of all channels, and the convolution operation of the second convolution kernel is a convolution operation on each position of the three-dimensional data under each channel.

[0074] In the present invention, the output data of each three-dimensional convolution submodule, the output data of the third three-dimensional convolution submodule and the image features corresponding to the training sample are connected by the following connection method:

[0075] X l+1 =Concat([X l,1 ,X l,2 ],axis=-1)

[0076] Among them, X l+1 represents the output of the l+1th layer network, X l,1 and X l,2 They represent the input tensors of the lth layer network, with shapes of (b, H, W, c1) and (b, H, W, c2), where c1 and c2 are X l,1 and X l,2 The size in the channel dimension, H, W are the sizes of the hyperspectral image in the horizontal and vertical directions respectively, b is the batch size, axis=-1 means splicing in the last dimension, that is, splicing in the channel dimension, the spliced ​​tensor X l+1 The shape of is (b,H,W,c1+c2).

[0077] Three-dimensional convolution can simultaneously extract the spatial and spectral features of hyperspectral images. Using a small-scale convolution kernel can reduce the complexity of the algorithm. However, using three-dimensional convolution to extract features from hyperspectral images means more parameters and higher complexity, which reduces the processing efficiency of hyperspectral images. To solve this problem, the three-dimensional deep separable convolution module of the present invention uses a multi-branch structure to extract multi-scale spatial features, and uses three-dimensional deep separable convolution to fuse the branch features, thereby efficiently extracting multi-scale spatial features and spectral features.

[0078] The three-dimensional convolution submodule designed in the present invention performs three-dimensional depth-separable convolution, and the three-dimensional convolution submodule solves multiple key problems in hyperspectral image processing. First, it significantly reduces the computational complexity and the amount of model parameters, which helps to efficiently process large-scale hyperspectral data. Secondly, the three-dimensional convolution submodule improves the generalization ability of the model, making it perform better on different data sets. In addition, it can adapt to features of different scales in hyperspectral images and help capture spatial and spectral information. And the number of parameters of the model is reduced, making it capable of being applied to resource-constrained environments, such as embedded devices and mobile applications, thereby widely improving the practicality and efficiency of hyperspectral image processing technology.

[0079] The 3D channel-by-channel convolution submodule is the first step of 3D depth-wise separable convolution. It is similar to the traditional convolution operation, but has independent convolution kernels when processing features of different channels. This means that each input channel is convolved with a separate convolution kernel, rather than mixing features of all channels together as in traditional convolution. 3D channel-by-channel convolution helps the network better capture and distinguish feature information in different channels, which can increase the network's representation ability and improve the processing efficiency of multi-channel input data.

[0080] 3D point-by-point convolution is the second step of 3D depth-wise separable convolution. In this step, the network uses a small convolution kernel (usually a kernel of size 1×1×d, d is the depth of the 3D data block of each channel after channel-by-channel convolution) to process the output of the previous 3D channel-by-channel convolution. The small convolution kernel performs convolution operations on each position (each pixel) of the 3D data under each channel, but only focuses on the information of one point, not the neighborhood information. 3D point-by-point convolution helps reduce the dimensionality of the data and the amount of computation, and is usually used to reduce the number of channels of the network or the depth of the feature map to reduce the complexity of the network.

[0081] The three-dimensional depth separable convolution module of the present invention uses a three-dimensional convolution submodule (Three-dimensionalDepth-separable Convolution, 3d-DSConv) to extract the spatial and spectral features of hyperspectral images. The input data will enter three consecutive three-dimensional convolution submodules, each of which is composed of a three-dimensional channel-by-channel convolution submodule and a three-dimensional point-by-point convolution submodule. In view of the characteristics of high-dimensional three-dimensional features of hyperspectral images, the three-dimensional spatial spectrum features of hyperspectral images can be deeply extracted. At the same time, jump connections are added to the features after deep extraction to avoid overfitting. The present invention uses the Concat operation to merge the two branches, which can avoid the loss of spatial and spectral feature information of hyperspectral images and improve the expression ability of the algorithm. Finally, ReLU is used to introduce nonlinear transformation. The nonlinear characteristics of ReLU can help the network better fit the training data, reduce the risk of overfitting, and still maintain the expression ability of the network.

[0082] Furthermore, the spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and based on the output vector of the three-dimensional depth separable convolution module, obtains the attention vector in the horizontal direction and the attention vector in the vertical direction of the training sample, and obtains the channel attention vector of the training sample; based on the attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector of the training sample, obtains the fused attention vector of the training sample, and based on the fused attention vector, generates the first image feature corresponding to the training sample.

[0083] The spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and obtains the attention vector of the training sample in the horizontal direction and the attention vector in the vertical direction based on the output vector of the three-dimensional depth separable convolution module, including:

[0084] The output vector X of the three-dimensional depth-separable convolution module is [x1, x2, ..., x C ]∈R C×H×W Perform global pooling, the global pooling formula is:

[0085]

[0086] where x1, x2, ..., x c are the features of the output vector of the three-dimensional depth-separable convolution module on each channel;

[0087] Two pooling kernels (H, 1) and (1, W) are used to encode the features on each channel along the horizontal and vertical coordinates, respectively, where H and W are the sizes of the hyperspectral image in the horizontal and vertical directions, respectively, and x c (i,j) represents the input with coordinates (i,j);

[0088] Therefore, the output of the cth channel at height h is:

[0089]

[0090] The output of the cth channel at width w is:

[0091]

[0092] Based on the output of each channel at each position of the training sample, obtaining the output of the training sample in the horizontal direction and the output of the training sample in the vertical direction;

[0093] The output of the training sample in the horizontal direction and the output in the vertical direction are concatenated and convolved to generate a first feature map; the first feature map is then activated, and the activated first feature map is split in the horizontal direction and the vertical direction to obtain the attention vector of the training sample in the horizontal direction and the attention vector in the vertical direction.

[0094] The spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and obtains the channel attention vector of the training sample based on the output vector of the three-dimensional depth separable convolution module, including:

[0095] The output vector X of the three-dimensional depth-separable convolution module is [x1, x2, ..., x c ]∈R C×H×W Perform global average pooling and global maximum pooling to obtain a global average pooled vector and a global maximum pooled vector respectively, use the global average pooled vector as a global feature, use the global maximum pooled vector as a salient feature, perform convolution processing, ReLU processing, and convolution processing on the global average pooled vector, and obtain a result as a first global feature; perform convolution processing, ReLU processing, and convolution processing on the global maximum pooled vector, and obtain a result as a first salient feature; splice the first global feature and the first salient feature, activate the spliced ​​result by a Sigmoid function, and obtain a channel attention vector of the training sample.

[0096] The step of obtaining a fused attention vector of the training sample based on the attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector of the training sample, and generating a first image feature corresponding to the training sample based on the fused attention vector includes:

[0097] The attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector of the training sample are weighted to obtain a fused attention vector of the training sample; the fused attention vector is multiplied by the image feature corresponding to the training sample to obtain a first image feature corresponding to the training sample.

[0098] In the design of lightweight algorithms, the attention mechanism has multiple advantages, which helps to improve the performance and efficiency of the model. First, the attention mechanism can improve parameter efficiency. By reducing the number of parameters in the model, the computational and storage burdens are reduced, making the model more lightweight. Secondly, it helps to improve computational efficiency. By reducing the complexity of the calculation, the reasoning speed of the model is accelerated, which is particularly suitable for scenarios with limited computing resources. In addition, the attention mechanism is adaptive and can automatically adjust the area or features it focuses on according to different input data conditions, thereby enhancing the generalization ability of the model. Finally, the performance of the model can also be improved because the attention mechanism helps to extract important task-related information, reduce the risk of overfitting, and improve the interpretability of the model. The spatial spectral coordinate attention module of the present invention can effectively solve the problem that hyperspectral images contain rich information, but when the cross-domain small sample hyperspectral image algorithm describes the image, there is insufficient extraction and description of spatial spectral features.

[0099] In hyperspectral image processing, spatial attention mechanism and channel attention mechanism each have their own advantages. The spatial attention mechanism can capture the spatial relationship between pixels, emphasize the local information in the image, and help improve the performance of tasks such as object positioning and edge detection. However, the spatial attention mechanism ignores the correlation between different channels, resulting in the failure to fully utilize the spectral information when processing hyperspectral data. In contrast, the channel attention mechanism focuses on the relationship between channels, can better capture spectral information, and is suitable for the selection and enhancement of spectral features. However, the channel attention mechanism ignores the spatial layout between pixels and may have limited effect in pixel-level tasks. Therefore, in order to make full use of the spatial and spectral information of hyperspectral images, the present invention combines the spatial and channel attention mechanisms and introduces the spatial spectral coordinate attention mechanism. This mechanism effectively captures the correlation between channels while retaining the spatial relationship between pixels, thereby improving the performance of various tasks of hyperspectral images. By jointly modeling the attention of space and channels, the model is made more adaptable and robust.

[0100] In the present invention, the spatial spectral coordinate attention module takes the intermediate feature tensor as input, which is expressed as X = [x1, x2, ..., x C ]∈R C×H×W , and generates a transformed tensor Y = [y1, y2, ..., y C ]. This enhances the feature extraction of important parts of the original sample.

[0101] In the spatial domain, the present invention designs a spatial attention mechanism that can fully extract the spatial information of hyperspectral images. First, the input features are globally pooled. Global pooling is usually used in channel attention to encode spatial information in a global manner, but global pooling compresses global spatial information into a channel descriptor, so it is difficult to retain position information, which is crucial for capturing spatial structures in visual tasks. In order to motivate the attention block to capture long-range interactions in space with precise position information, global pooling is formulated as a pair of 1D feature encoding operations, as shown below.

[0102]

[0103] Specifically, given an input X, two spatial range pooling kernels (H, 1) or (1, W) are used to encode each channel along the horizontal and vertical coordinates, respectively. Where H and W represent the size in the horizontal and vertical directions, respectively, and x c (i,j) represents the input x with coordinates (i,j). Therefore, the output of the cth channel at height h can be expressed as follows.

[0104]

[0105] Similarly, the output of the cth channel at width w can be expressed as follows.

[0106]

[0107] After global pooling based on coordinate information, Concat and Conv2d are used to aggregate features along two spatial directions to generate a pair of direction-aware feature maps, and then BatchNorm and Non-linear are used to generate activated feature maps. The activated feature maps are split along the spatial dimension, and the dimensionality is increased by 1×1 convolution combined with Sigmod to obtain the attention vectors in the horizontal and vertical directions, thus achieving weighting of important information in the spatial domain of the hyperspectral image.

[0108] The present invention uses coordinate information to enable the attention block to capture long-distance dependencies along one spatial direction and maintain accurate position information in another spatial direction, thereby helping the network to more accurately locate the object of interest, achieving a global receptive field and encoding accurate position information. By weighting the important information in the spatial domain of the hyperspectral image through position information, the more important part of the rich information in the hyperspectral image can be screened out.

[0109] In the spectral domain, in order to screen channels with more important features and remove redundant parts in the spectral domain, this paper proposes a channel attention mechanism that can cope with high-latitude hyperspectral images. First, the input feature map is subjected to global average pooling and global maximum pooling to obtain two one-dimensional vectors as the global features and salient features of the image. The global features and salient features are then passed through Conv1d, ReLU, and Conv1d to obtain a new feature map. Using two convolutional layers can reduce the number of parameters and calculations while making full use of the prior knowledge that adjacent channels have relevant information, which is more conducive to the channel's attention to discover important feature information. Finally, the processed global features and salient features are concatenated and the Sigmod activation function is used to obtain the final one-dimensional weight map.

[0110] The weight map obtained in the empty spectrum domain is used to weight the input features, and the model is multiplied with the feature map of the original input, and the original feature map X is emphasized and eliminated in the empty spectrum domain. In summary, the empty spectrum coordinate attention mechanism module can better utilize the spatial and spectral information in the hyperspectral image, thereby improving the accuracy and robustness of feature extraction.

[0111] In the present invention, a neural network model is trained based on the first image feature to obtain a trained neural network model, wherein the neural network model is any conventional neural network model in the prior art.

[0112] The block pruning module is used to remove a network block B of the trained neural network model. i , the pruned neural network model is recorded as The network block corresponds to a network layer of the neural network model; In the network block B in the trained neural network model, i Each of the previous network blocks is used as a front-end network block. For each front-end network block, an adapter is added at the output end of the network layer corresponding to the front-end network block as a new front-end network block. In the network block B in the trained neural network model, iEach subsequent network block is used as a successor network block. For each successor network block, an adapter is added to the input end of the network layer corresponding to the successor network block as a new successor network block; the adapter is a convolution layer with a convolution kernel of 1*1; establish a distance network block B i The output of the nearest new front network block is the distance network block B i The connection of the input of the nearest new successor network block and the calculation of the pruned neural network model The pruning score of .

[0113] The block-based pruning module prunes the trained neural network model M times, each pruning obtains a pruned neural network model, and selects a pruned neural network model with the highest pruning score from the M pruned neural network models as the first trained neural network model, including:

[0114] Step S21: Prune the trained neural network model M times based on the block pruning module, and prune each pruned neural network model Do the following:

[0115] Calculate the recoverability R(B i ):

[0116]

[0117] Where θ is the parameter in the trained neural network model, x is the image feature, p(x) is the distribution of x, and E x～p(x) is the expected value of x when it is distributed according to the probability density function p(x), M o (x; θ) is a trained neural network model with parameters. is the square of the Frobenius norm, \bi is the value after removing the network block bi , α is the parameter in the adapter; A pruned neural network model with parameters and adapter parameters

[0118] Compute network block B i Delay time:

[0119]

[0120] Among them, latM o is the delay time of the trained neural network model, The neural network model after pruning Delay time;

[0121] Calculate the pruning score:

[0122]

[0123] Step S22: Selecting a pruned neural network model with the highest pruning score from the M pruned neural network models as the first trained neural network model.

[0124] The block pruning module can be optimized in the following ways:

[0125]

[0126] Among them, θ o is the frozen parameter of the trained neural network model, θ p is the trainable parameter of the first neural network model after training, M o (x;θ o ) is a trained neural network model with parameters, M P (x;θ p ) is a neural network model after pruning with parameters.

[0127] Model pruning is a technique used in the field of deep learning to reduce the size of neural network models. It aims to reduce the computational complexity and storage requirements of the model while maintaining or even improving the performance of the model. The basic idea of ​​this technique is to prune redundant parameters and structures in the network to obtain a lighter model while maintaining the predictive power of the original model. Model pruning is usually divided into the following two types: parameter pruning and structural pruning. In parameter pruning, the size of the model is reduced by setting or removing certain weight parameters in the model. Usually, parameter pruning selects the weights to be pruned based on some pruning strategies, such as L1 regularization, sensitivity analysis, or adaptive pruning. This can reduce the number of connections in the neural network, thereby reducing the amount of computation and memory usage. Unlike parameter pruning, structural pruning involves deleting certain layers or units in the entire neural network. This type of pruning usually requires some heuristic strategies, such as channel pruning or layer pruning, to decide which layers or units should be pruned. Structural pruning is usually more aggressive and can significantly reduce the size of the model. The advantage of model pruning is that it can significantly reduce the storage and computing resource requirements of deep neural networks, making the model more suitable for deployment in resource-constrained environments, such as mobile devices, embedded systems, or edge computing platforms. In addition, the pruned model is expected to achieve better generalization performance because some redundant parameters and structures are removed, reducing the risk of overfitting.

[0128] In the study of lightweight algorithms for small sample image classification, most previous studies have adopted filter-level pruning. However, on actual computing devices, such as GPUs, high acceleration ratios cannot be achieved. In order to make the compressed model actually run faster than the uncompressed model, it is necessary to reduce a large number of floating-point operations through filter-level pruning. At the same time, without sufficient training data sets, it is difficult to restore the accuracy of the compressed model. Therefore, previous small sample compression methods usually show a poor trade-off between latency and accuracy. In order to further improve the performance of cross-domain small sample hyperspectral image classification models in the case of insufficient data sets, the present invention proposes a block pruning module based on model pruning technology.

[0129] In the present invention, firstly, the data passes through a given original network M o The model is trained on the network, and then the model is block pruned to remove a block B in the model network. i , and get the pruned model In order to minimize the impact of removing this module, adapters are inserted in the network layers before and after connecting this block. Then the L2 distance between the original network and the pruned model is calculated as the network's recoverability. The recoverability is defined as follows:

[0130]

[0131] Among them, θ represents the parameters in the original model, \bi represents the parameters in the excluded block bi, and α represents the parameters in the adapter. All adapters are convolutional layers with a kernel size of 1×1 and are placed before / after the original convolutional layer according to different positions. For the blocks before the discarded blocks, the adapters are inserted after the convolutional layer. Conversely, the adapters are inserted before the convolutional layer. Because the convolution is linear, all adapters can be fused into adjacent convolutional layers while keeping the output unchanged. This means that these adapters will not become the overhead of the pruned model. Since the computational amount and parameters of the adapters are limited, calculating the recoverability requires only a few training samples and little training time. Calculating the recoverability of the pruned block is only one aspect. For the lightweight design of the network, the delay time of each block must also be calculated. The calculation formula is as follows:

[0132]

[0133] Among them, lat refers to the delay time, and finally a block pruning score is defined as shown in the following formula:

[0134]

[0135] In the present invention, the pruning score takes into account the recoverability and latency of each block. The lower the score, the higher the discard priority of the block. The pruning score of each block is calculated, and the first k blocks with the lowest scores are deleted, and k is adjusted according to the data set results.

[0136] After the structure of the pruned model is determined, the accuracy of the pruned model needs to be restored. The traditional method is to use a small training set to fine-tune the cross entropy loss, but the pruned model is prone to overfitting problems. The use of knowledge distillation, especially the feature-level distillation method, can alleviate the overfitting problem and obtain better accuracy. MiR (Mimicking thenReplacing) discards the posterior distribution alignment relied on in traditional knowledge distillation methods by using only the knowledge transfer of the Penultimate Layer. And by grafting the classification head / detection head in the original model to the compressed model, the best performance is achieved with fewer training samples. Therefore, the present invention follows MiR in this module and fine-tunes the pruned model by minimization. The process is shown in the following formula:

[0137]

[0138] In the formula, θ o is the frozen parameter of the original model, θ p are the trainable parameters of the pruned model.

[0139] The three-dimensional deep separable convolution module of the present invention is designed with three-dimensional deep separable convolution, and the module structure is divided into two parts. Among them, one part continuously uses multiple deep separable convolutions to extract feature information of the low receptive field spatial range, and the other part directly uses residual connection to maintain the original spatial feature information of the data. In this way, the module can extract high-scale information of hyperspectral image data and obtain global information, thereby improving the stability of the model. The use of deep separable convolution reduces the training parameters of the module, speeds up the calculation speed of the algorithm, and thus improves the operation efficiency of the algorithm.

[0140] The spatial-spectral coordinate attention module of the present invention is aimed at the attribute of hyperspectral images as three-dimensional data with high dimensions. By combining the spatial attention mechanism with the channel attention mechanism, it is possible to learn the importance weights of cross-domain small sample hyperspectral image data in different bands and different spatial positions, so that the network focuses on the bands and spatial positions with more important features. More important and effective spatial-spectral features can be extracted under limited labeled samples.

[0141] The block pruning module of the present invention performs block pruning on each block of the entire network to obtain a network that lacks a certain block compared to the original network. Then, the recoverability and delay time of the pruned network are calculated, and the score of the pruned network is calculated based on this. Finally, the scores of all block pruned networks are sorted to obtain the optimal block pruned network.

[0142] The three-dimensional depth separable convolution module disclosed in the present invention replaces the traditional three-dimensional convolution through three-dimensional channel-by-channel convolution and three-dimensional point-by-point convolution, thereby reducing the number of parameters and computational complexity of the model. At the same time, in order to extract more important or more effective spatial spectral features under limited labeled samples, the spatial spectral coordinate attention module of the present invention can learn the importance weights of hyperspectral image data in different bands and different spaces, focus on the bands and spaces that contribute to the task, and allow the network to focus more on important features or areas, thereby reducing the number of redundant parameters in the network to a certain extent, and helping to reduce the complexity and computational burden of the network. Finally, in order to further solve the problem of high complexity of the cross-domain small sample hyperspectral image classification model, the present invention constructs a block pruning module, which calculates the recoverability and delay time of the pruned network and uses this as an indicator to score the pruned model, and selects the optimal block pruning model according to the score from high to low. Compared with the original network model, the pruned model can effectively reduce the number of parameters of the algorithm while ensuring the accuracy, and successfully achieves the lightweight of the network. Experimental results show that the recognition performance of the present invention on three commonly used hyperspectral datasets is good and the complexity is effectively reduced compared with before.

[0143] Furthermore, the present invention preprocesses the original hyperspectral image, and the preprocessed hyperspectral image is used as the hyperspectral image to be classified. The preprocessing is an optional operation of the present invention, and is not a necessary operation.

[0144] The pre-processing comprises:

[0145] Step S11: Determine the main components of the hyperspectral image, including:

[0146] Acquire a hyperspectral image, and generate a data matrix X corresponding to the hyperspectral image, wherein the size of the data matrix X is n×m, wherein n is the number of pixels and m is the number of bands;

[0147] Compute the covariance matrix C of the data matrix X, where:

[0148]

[0149] N is the number of sample points in the data matrix X;

[0150] Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​λ1,λ2,…,λM and the corresponding eigenvectors v1,v2,…,v M , arrange the eigenvalues ​​in descending order, and select the features corresponding to the first K1 eigenvalues ​​as the principal components;

[0151] Step S12: obtaining an original hyperspectral image to be processed, and multiplying the original hyperspectral image by the principal component to obtain a hyperspectral image after dimensionality reduction;

[0152] Step S13: for the central pixel X1 of the original hyperspectral image to be processed, extract the central pixel and the pixels of size S×S in its neighborhood, and form them into a three-dimensional cuboid data XX, that is, XX∈R(S,S,D), where S represents the length and width of the cuboid, and D represents the depth;

[0153] Step S14, obtaining a preprocessed hyperspectral image based on the hyperspectral image after dimension reduction and the three-dimensional cuboid data XX corresponding to the original hyperspectral image to be processed.

[0154] In the present invention, hyperspectral image preprocessing can be divided into three steps: principal component analysis, neighborhood pixel extraction and data partitioning. Assume that the length of a hyperspectral image H is l, the width is w, and the depth is d, that is, H∈R(l,w,d), and all its label data are represented by T, then T∈R(l,w). Considering the high-dimensional characteristics of hyperspectral image data, there is a significant information correlation between adjacent dimensions in the original data, resulting in high data redundancy. In order to address this problem, a suitable dimensionality reduction method is needed. In this study, the PCA method was selected to achieve data dimensionality reduction, reducing the number of channels to k dimensions while keeping the spatial dimension unchanged. The steps of dimensionality reduction using the PCA method are described in detail below.

[0155] First, the hyperspectral image data is organized into a matrix X, where each row represents a sample (pixel) and each column represents a band (feature). Assuming there are n samples and m bands, an n×m data matrix X is obtained.

[0156] In order to ensure that each band is equally important, the data needs to be standardized, usually by zero-meaning each band (mean is 0) and unit variance (variance is 1). This step ensures that the contribution of each band is equal. The data matrix after centering is obtained:

[0157]

[0158] Secondly, calculate the covariance matrix C of the data matrix X. The covariance matrix C describes the correlation between the various bands in the data. The calculation formula of the covariance matrix is ​​as follows:

[0159]

[0160] Next, perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalues ​​and corresponding eigenvectors. The eigenvalues ​​represent the variance in the data, while the eigenvectors represent the projections of the data in different directions. Assume that the eigenvalues ​​obtained are λ1,λ2,…,λ M , the corresponding eigenvectors are v1,v2,…,v M , arranged in descending order of eigenvalues. The number of principal components to be retained is selected by analyzing the size of the eigenvalues, that is, the number of dimensions after dimensionality reduction. The number of principal components to be retained can be a subset of the original data dimensions, for example, the first K principal components can be retained.

[0161] Finally, project the data onto the selected principal components to obtain new data after dimensionality reduction. For each sample, multiply it with the selected principal component to obtain the sample after dimensionality reduction. Assuming that the first K principal components are selected, the data matrix after dimensionality reduction is Y, then: Y = XV k ,Here, Vk is the matrix containing the first K eigenvectors.

[0162] Although PCA dimension reduction has reduced the spectral dimension of hyperspectral images and reduced the redundant information of hyperspectral images, a single spectral curve cannot effectively utilize the rich spatial domain information of hyperspectral images. Therefore, the neighborhood pixel extraction method is used to reconstruct the unit sample in order to create a suitable sample set for the deep learning network. The neighborhood pixel extraction method is to transform the original one pixel point as a sample unit into a spectral curve block as a sample unit for training.

[0163] Specifically, for a given central pixel x, the pixel and its neighboring pixels of size S×S are extracted to form a three-dimensional cuboid data X, that is, X∈R(S,S,D), where S represents the length and width of the cuboid and D represents the depth. In this way, a dataset P consisting of sample data is obtained, where each sample consists of a three-dimensional cuboid data and a corresponding label value, expressed as P=(X a ,Y a ), X a is the three-dimensional cuboid data of the a-th sample, Y a Represents the label value of the a-th sample.

[0164] The data division is to divide the sample set consisting of the hyperspectral images as samples into a training set and a test set.

[0165] The preprocessing method includes principal component analysis and neighborhood pixel extraction. The principal component analysis method solves the feature redundancy problem of the original image on the channel by reducing the channel dimension of the hyperspectral image data. The neighborhood pixel extraction method divides the original data set into sample sets suitable for deep learning model network training and testing.

[0166] The present invention provides a specific embodiment of a cross-domain small sample learning hyperspectral image classification method, the method comprising:

[0167] Step 1. Input data, including hyperspectral data and corresponding labels.

[0168] Step 2. Perform preprocessing operations and use the principal component analysis method to reduce the dimension of the input hyperspectral data set.

[0169] Step 3. Use neighborhood pixel extraction to cut the data set and divide the data set to obtain a sample set for deep learning network training and testing.

[0170] Step 4. Use the feature extraction module based on three-dimensional deep separable convolution and spatial spectral coordinate attention mechanism to extract small sample hyperspectral image features.

[0171] Step 5. Send the features obtained through deep spatial spectrum feature extraction into the network model for training to obtain the trained original model Mo.

[0172] Step 6. Prune the trained initial network model, discard a certain block structure Bi in the network, and use the adapter to control the consistency of the network input and output structures between the discarded block structures. Obtain the pruned model MP(Bi) and calculate the pruning score.

[0173] Step 7. Based on the calculated pruning scores of different pruning blocks, discard the layer with the lowest score to obtain the network model with the best performance after pruning.

[0174] The experiments in the present invention are all completed by a server host with 32GB RAM and RTX2080Ti GPU. The programming language used is Python version 3.7, and the present invention is implemented using the Pytorch framework, version 1.2.1.

[0175] The present invention conducts experiments on four hyperspectral datasets, namely Chikusei, Indian Pines, Pavia University and Salinas Scene, wherein Chikusei is used as the source domain dataset, and the other three datasets are used as target domain datasets for three experiments. In the field of hyperspectral image classification, the size of the input sample needs to be adjusted for different datasets. The size of the input sample of the Chikusei dataset is set to 9×9×128, the size of the input sample of the Indian Pines dataset is set to 9×9×200, the size of the input sample of the Pavia University dataset is set to 9×9×103, and the input sample of the Salinas Scene dataset is set to 9×9×204. The optimizer used in this experiment is the Adam optimizer, and its learning rate is set to 0.001. All convolutional layers and linear layers are initialized using Xavier normalization, and the batch normalization layer uses the normal distribution N(1,0.02). The default value of the 2-norm regularization for feature extraction is 0, and the weight decay of other modules is set to 1e-4. The number of training iterations is set to 10,000 to prevent too few training epochs from fully learning the prototypes of each class.

[0176] In order to evaluate the performance of Gia-CFSL in the case of SD and TD from different sensors and containing different categories, the present invention uses related algorithms for comparison, including SVM, 3DCNN, PToP CNN, DAAN, DSAN, DPGN, DFSL, RN-FSC, DCFSL and 3D-GiaCFSL. The training sample settings are as follows: SVM, 3DCNN and PToP CNN are regarded as supervised methods, requiring training and test samples with the same dimension and the same category, so only five labeled samples of each category in TD are selected as training sets, and the rest are used as test sets. For the unsupervised domain adaptation methods DAAN and DSAN, the projection layer consistent with BPGia-CFSL is added, and all labeled SD data and unlabeled TD data are used for training. For 3D-GiaCFSL, DPGN, DFSL, RN-FSC and DCFSL developed based on FSL, the selection of training samples is the same as BPGia-CFSL. Specifically, DPGN only uses the SD construction meta-task for training, and does not involve TD, DFSL, and training samples of the same dimension and the same category. Therefore, only five labeled samples of each category in TD are selected as the training set, and the rest are used as the test set. For the unsupervised domain adaptation methods DAAN and DSAN, a projection layer consistent with BPGia-CFSL is added, and all labeled SD data and unlabeled TD data are used for training. The training sample selection of DPGN, DFSL, RN-FSC and DCFSL developed based on FSL is the same as that of BPGia-CFSL. Specifically, DPGN only uses the SD construction meta-task for training, and does not involve TD, DFSL, and training samples of the same dimension and the same category. Therefore, only five labeled samples of each category in TD are selected as the training set, and the rest are used as the test set.

[0177] The present invention randomly selects five labeled samples from each category as the supervision samples of the target domain, and conducts ten experiments to calculate the average accuracy and standard deviation. The accuracy evaluation uses three indicators, namely, the overall accuracy OA, the average accuracy AA and the Kappa coefficient. The computational performance evaluation uses the training time, the test time and the number of parameters as indicators to measure the complexity of the network.

[0178] In order to more comprehensively analyze the performance of the hyperspectral image classification algorithm, the present invention uses quantitative analysis, qualitative analysis and computational performance analysis methods to evaluate the algorithm performance. In the quantitative analysis, the present invention evaluates the classification accuracy of the algorithm by comparing the overall accuracy, average accuracy Kappa coefficient and. In the qualitative analysis, the present invention evaluates the performance of different algorithms by comparing the predicted classification graphs. At the same time, the algorithm is also subjected to computational performance analysis, and the complexity of the algorithm is evaluated by comparing the training time, test time and parameter amount of the algorithm. Through the combination of these two methods, the present invention can more comprehensively evaluate the accuracy performance of the cross-domain small sample hyperspectral image classification algorithm.

[0179] (1) Statistics of classification results of different algorithms under small sample conditions

[0180] In order to achieve quantitative analysis of the classification effect of the algorithm, Tables 1-3 show the classification results of different algorithms on the Pavia University, Indian Pines, and Salinas Scene datasets. For all classification methods, 5 labeled samples are randomly selected from each category as supervised samples in TD, and the experimental average accuracy and standard deviation of 10 groups of different random seeds for each algorithm are calculated. The following is an analysis of the statistical table of classification results of various algorithms on three different datasets:

[0181] Table 1

[0182]

[0183] 1) Compared with SVM, 3DCNN and PToP CNN which are trained with only a small number of target domain TD samples, DAAN, DSAN, DFSL, RN-FSC, DCFSL, 3DGia-CFSL which utilize source domain knowledge and design domain alignment module, and BPGia-CFL which adds 3D deep separable convolutional feature extraction module, spatial spectral coordinate attention mechanism module and block pruning module on the basis of 3DGia-CFSL, improve OA by 5%-10%, which indicates that when there are fewer samples in TD, transferring SD knowledge to TD helps to improve the classification accuracy of the model on TD. The cross-domain approach is a better way to solve the problem of insufficient samples.

[0184] 2) The performance of DFSL, RN-FSC, DCFSL, 3DGia-CFSL and BPGia-CFSL methods based on FSL is 4%-8% higher than that of unsupervised data processing methods such as DAAN and DSAN. This is because the learning method of constructing meta-tasks for scenario training in FSL has stronger generalization ability and learns more complete SD knowledge. This shows that the FSL method trained with meta-learning ideas can better solve the problem of fewer labeled samples, improve the robustness of the network, and increase the adaptability of the network.

[0185] Table 2

[0186]

[0187] 3) Due to the domain alignment module design, BPGia-CFSL, 3DGia-CFSL and DCFSL outperform DPGN, DFSL and RN-FSC which only focus on FSL. This shows that in the cross-domain case, aligning the source domain and the target domain can effectively improve the generalization performance of the network. Overall, the global accuracy OA of the experiments on these three datasets has increased by about 1.5%.

[0188] Table 3

[0189]

[0190]

[0191] 4) The 3DGia-CFSL network was improved by adding a 3D deep separable convolution feature extraction module, a spatial spectral coordinate attention mechanism, and a block pruning module to obtain the BPGia-CFSL network. According to the results on three different data sets, the lightweight network can maintain the classification accuracy of the model while reducing the model parameters. The accuracy of the lightweight network in the PaviaU and Indian Pines data sets was slightly improved, and there was only a slight difference between the accuracy of the lightweight network and the 3DGia-CFSL in the Salinas data center. This shows that lightweighting the network can reduce redundant modules in the original network, reduce the complexity of the network, and improve the performance of the network.

[0192] (2) Label diagram of classification results of different algorithms under small sample conditions

[0193] The SVM algorithm and the 3D-CNN algorithm have a large number of misclassifications. In contrast, the other algorithms that added small sample learning and domain alignment modules showed better classification results. In particular, the present invention showed more accurate classification results in very detailed places. The present invention reduces redundant parameters in the network and improves the generalization of the network, so it has a better classification effect for pixels at the junction of different types of objects.

[0194] Secondly, the classification accuracy of each classification algorithm on the Indian Pines dataset was qualitatively analyzed.

[0195] Finally, a qualitative analysis is conducted on the classification accuracy of each classification algorithm on the Salinas dataset.

[0196] Based on the above experiments, the experimental results of the three data sets are analyzed. It is found that the present invention performs well in classification accuracy, which indicates that the three-dimensional deep separable convolution module and the spatial spectral coordinate attention mechanism module constructed by the present invention can fully extract the spatial spectral features of the hyperspectral image. Therefore, the present invention has a strong feature extraction capability and can achieve correct classification.

[0197] (3) Impact of each module on the classification performance of the algorithm

[0198] In order to explore the impact of lightweighting of each module on the classification performance of the BPGia-CFSL algorithm compared with 3DGia-CFSL, four ablation experiments as shown in Table 4 were performed. 3DGia-CFSL: The present invention was trained on three data sets respectively, and each data set was trained ten times to calculate the experimental mean and standard deviation.

[0199] Table 4

[0200]

[0201] Obviously, the addition of each lightweight module in the BPGia-CFSL of the present invention maintains the accuracy of the original model on the basis of reducing the amount of parameters, and successfully realizes the lightweighting of the 3DGia-CFSL model. It can be found from the table that the experimental classification accuracy of 3DGia-CFSL is the highest on the Pavia University dataset and the Salinas dataset, and the BPGia-CFSL accuracy is the highest on the Indian Pines dataset. The same algorithm ranks differently on different datasets. This may be because the original network model structure is more suitable for the Pavia University and Salinas datasets and can achieve a higher accuracy. The lightweight network model has better generalization ability due to the reduction of the number of parameters and the complexity of the model, so it achieves the highest accuracy in Indian Pines. In summary, the present invention can maintain or even exceed the classification accuracy of the original model on the basis of effectively reducing the number of parameters of the system, which proves the effectiveness of the present invention.

[0202] (4) Computational performance analysis of different algorithms

[0203] In order to explore quantitative analysis, the number of parameters to be trained of different algorithm networks and the training time and test time on three types of data sets are statistically shown in Table 5.

[0204] Table 5

[0205]

[0206] Obviously, according to the data in the table, PToP CNN has the highest computational cost, which is caused by unsupervised pre-training in patch-to-patch mode. SVM and 3DCNN do not contain domain alignment or FSL, but directly train classifiers with a small number of samples, with low computational complexity. The complexity of 3DGia-CFSL is similar to that of cross-domain classification methods such as DAAN and DCFSL, indicating that the knowledge transfer in cross-domain classification increases the computational cost but guarantees the classification performance. Compared with the conventional three-dimensional convolution structure in 3DGia-CFSL, the use of three-dimensional depth-separable convolution brings fewer parameters (about 12.9% reduction in parameters), and the block pruning method is used to discard the network blocks with lower pruning scores in the network, reducing the delay time of training the network. Therefore, the training and testing time of BPGia-CFSL are effectively shortened (the training time is reduced by an average of 14.1%, and the test time is shortened by an average of 14.4%), while maintaining a good classification accuracy. This shows that the network is more lightweight and the performance of the network is improved.

[0207] The data preprocessing of the present invention preprocesses the input hyperspectral image data, which may include denoising, normalization, dimensionality reduction and other operations to prepare for subsequent feature extraction and classification. Feature extraction: extract features from the preprocessed hyperspectral image data, which can reflect the spectral information and spatial information of different bands in the image. Block pruning: select the extracted features based on block pruning to reduce the dimension of the feature space and retain the most representative feature blocks, thereby improving the classification efficiency. Cross-domain small sample learning: deal with cross-domain small sample learning problems, and use sample data from different fields to perform classification tasks. Transfer learning, meta-learning and other technologies may be used to transfer the learned knowledge to the classification task in the target field. Model training: use the selected learning algorithm (such as support vector machine, neural network, etc.) to train the processed features and establish a classification model. Model evaluation: evaluate the trained model, and evaluate the performance of the model through cross-validation, confusion matrix and other methods to determine the accuracy, recall and other indicators of the classifier. Classification prediction: use the trained model to classify and predict the new hyperspectral image data and classify it into the corresponding category. Hyperspectral image classification: It can accurately classify hyperspectral images and can be used to identify and classify land objects, vegetation, soil, etc. in agriculture, environmental monitoring, geological exploration and other fields.

[0208] The present invention can solve the problem of scarce or difficult data sets by using sample data from different fields to perform classification tasks, and improve the generalization ability and adaptability of the model. The feature selection method based on block pruning can effectively reduce the dimension of the feature space, improve classification efficiency and reduce computational costs. By optimizing feature selection and cross-domain learning methods, the accuracy and stability of the classifier can be improved, and the adaptability to complex scenes can be enhanced. The optimized classification method has high real-time performance and efficiency, can quickly process large-scale hyperspectral image data, and meet the real-time requirements in practical applications.

[0209] The present invention provides a cross-domain small sample learning hyperspectral image classification device, the device comprising:

[0210] Feature extraction module: configured to extract image features from the hyperspectral image to be classified;

[0211] Classification module: configured to input the image features into the trained first neural network model to obtain the classification result of the hyperspectral image;

[0212] The first neural network model is obtained by:

[0213] Inputting the image feature corresponding to the training sample into the three-dimensional depth separable convolution module and the empty spectral coordinate attention module connected in sequence to obtain the first image feature corresponding to the training sample;

[0214] Training a neural network model based on the first image feature to obtain a trained neural network model;

[0215] The trained neural network model is pruned M times based on the block pruning module, and a pruned neural network model is obtained each time. The pruned neural network model with the highest pruning score is selected from the M pruned neural network models as the first trained neural network model.

[0216] A specific embodiment of the present invention includes:

[0217] Hyperspectral image input module: used to receive and process input hyperspectral image data.

[0218] Feature extraction module: used to extract features from the input hyperspectral image, which may adopt convolutional neural network (CNN) or other feature extraction methods.

[0219] Block pruning module: implements feature selection based on block pruning to reduce the dimension of feature space and improve classification efficiency.

[0220] Cross-domain small sample learning module: used to handle cross-domain small sample learning problems. It may use transfer learning, meta-learning or other techniques to utilize sample data from different fields for classification tasks.

[0221] Classifier: used to classify the processed data, which may adopt machine learning algorithms such as support vector machine (SVM) and random forest.

[0222] Performance evaluation module: used to evaluate the performance of the classifier, such as accuracy, recall and other indicators.

[0223] User interface (optional): Provides a user interaction interface to facilitate user operation and monitoring of the classification process.

[0224] Storage module: used to store training data, model parameters and other information.

[0225] The embodiment of the present invention further provides an electronic device, including:

[0226] A processor, which is used to execute multiple instructions;

[0227] A memory for storing a plurality of instructions;

[0228] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0229] The embodiment of the present invention further provides a computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method as described above.

[0230] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0231] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0234] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a physical server, or a network cloud server, etc., and the Ubuntu operating system must be installed) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0235] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A cross-domain small sample learning hyperspectral image classification method, characterized in that: The method comprises the following steps: Step S1: extracting image features from the hyperspectral image to be classified; Step S2: inputting the image features into the trained first neural network model to obtain the classification result of the hyperspectral image; The first neural network model is obtained by: Inputting the image feature corresponding to the training sample into the three-dimensional depth separable convolution module and the empty spectral coordinate attention module connected in sequence to obtain the first image feature corresponding to the training sample; Training a neural network model based on the first image feature to obtain a trained neural network model; The trained neural network model is pruned M times based on the block pruning module, and a pruned neural network model is obtained each time. The pruned neural network model with the highest pruning score is selected from the M pruned neural network models as the first trained neural network model.

2. The method according to claim 1, characterized in that The three-dimensional depth-separable convolution module includes three three-dimensional convolution submodules connected in sequence, and the image features corresponding to the training samples are used as input data and input into the first three-dimensional convolution submodule; each three-dimensional convolution submodule performs three-dimensional channel-by-channel convolution and three-dimensional point-by-point convolution on its own input data to obtain its own corresponding output data, and the output data includes the spatial features and spectral features corresponding to the training samples; The output data of the first three-dimensional convolution submodule is used as the input data of the second three-dimensional convolution submodule, and the output data of the second three-dimensional convolution submodule is used as the input data of the third three-dimensional convolution submodule. The output data of the third three-dimensional convolution submodule is connected to the image features corresponding to the training sample, and a RELU operation is performed to obtain the output vector of the three-dimensional depth-separable convolution module.

3. The method according to claim 2, characterized in that The three-dimensional convolution submodule includes a three-dimensional channel-by-channel convolution submodule and a three-dimensional point-by-point convolution submodule connected in sequence; the three-dimensional channel-by-channel convolution submodule includes a convolution layer with three first convolution kernels, each of which is used to perform a convolution operation on the data of a channel of the input data of the three-dimensional channel-by-channel convolution submodule to obtain the convolution results of each channel of the input data of the three-dimensional channel-by-channel convolution submodule; the three-dimensional point-by-point convolution submodule includes four second convolution kernels with a size of 1*1*d, d is the depth value of the convolution result; each second convolution kernel performs a convolution operation on the convolution results of all channels, and the convolution operation of the second convolution kernel is a convolution operation on each position of the three-dimensional data under each channel.

4. The method according to claim 3, characterized in that The spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and based on the output vector of the three-dimensional depth separable convolution module, obtains the attention vector in the horizontal direction and the attention vector in the vertical direction of the training sample, and obtains the channel attention vector of the training sample; based on the attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector of the training sample, obtains the fused attention vector of the training sample, and based on the fused attention vector, generates the first image feature corresponding to the training sample.

5. The method according to claim 4, characterized in that The spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and obtains the attention vector of the training sample in the horizontal direction and the attention vector in the vertical direction based on the output vector of the three-dimensional depth separable convolution module, including: The output vector X of the three-dimensional depth-separable convolution module is [x1, x2, ..., x c ]∈R C×H×W Perform global pooling, the global pooling formula is: where x1, x2, ..., x c are the features of the output vector of the three-dimensional depth-separable convolution module on each channel; Two pooling kernels (H, 1) and (1, W) are used to encode the features on each channel along the horizontal coordinate and the vertical coordinate, respectively, where H and W are the sizes of the hyperspectral image in the horizontal and vertical directions, respectively, and x c (i, j) represents the input with coordinates (i, j); Therefore, the output of the cth channel at height h is: The output of the cth channel at width w is: Based on the output of each channel at each position of the training sample, obtaining the output of the training sample in the horizontal direction and the output of the training sample in the vertical direction; The output of the training sample in the horizontal direction and the output in the vertical direction are concatenated and convolved to generate a first feature map; the first feature map is then activated, and the activated first feature map is split in the horizontal direction and the vertical direction to obtain the attention vector of the training sample in the horizontal direction and the attention vector in the vertical direction.

6. The method according to claim 5, characterized in that The spatial spectral coordinate attention module receives the output vector of the three-dimensional depth separable convolution module, and obtains the channel attention vector of the training sample based on the output vector of the three-dimensional depth separable convolution module, including: The output vector X of the three-dimensional depth-separable convolution module is [x1, x2, ..., x c ]∈R C×H×W Performing global average pooling and global maximum pooling to obtain a global average pooled vector and a global maximum pooled vector respectively, taking the global average pooled vector as a global feature, taking the global maximum pooled vector as a salient feature, subjecting the global average pooled vector to convolution, ReLU, and convolution, and obtaining a result as a first global feature; subjecting the global maximum pooled vector to convolution, ReLU, and convolution, and obtaining a result as a first salient feature; splicing the first global feature and the first salient feature, activating the spliced ​​result by a Sigmoid function, and obtaining a channel attention vector of the training sample; The step of obtaining a fused attention vector of the training sample based on the attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector, and generating a first image feature corresponding to the training sample based on the fused attention vector includes: The attention vector in the horizontal direction, the attention vector in the vertical direction, and the channel attention vector of the training sample are weighted to obtain a fused attention vector of the training sample; the fused attention vector is multiplied by the image feature corresponding to the training sample to obtain a first image feature corresponding to the training sample.

7. The method according to claim 6, characterized in that The block pruning module is used to remove a network block B of the trained neural network model. i , the pruned neural network model is recorded as The network block corresponds to a network layer of the neural network model; In the network block B in the trained neural network model, i Each of the previous network blocks is used as a front-end network block. For each front-end network block, an adapter is added at the output end of the network layer corresponding to the front-end network block as a new front-end network block. In the network block B in the trained neural network model, i Each subsequent network block is used as a successor network block. For each successor network block, an adapter is added to the input end of the network layer corresponding to the successor network block as a new successor network block; the adapter is a convolution layer with a convolution kernel of 1*1; establish a distance network block B i The output of the nearest new front network block is the distance network block B i The connection of the input of the nearest new successor network block and the calculation of the pruned neural network model The pruning score of The block-based pruning module prunes the trained neural network model M times, each pruning obtains a pruned neural network model, and selects a pruned neural network model with the highest pruning score from the M pruned neural network models as the first trained neural network model, including: Step S21: Prune the trained neural network model M times based on the block pruning module, and prune each pruned neural network model Do the following: Calculate the recoverability R(B i ): Where θ is the parameter in the trained neural network model, x is the image feature, p(x) is the distribution of x, and E x～p(x) is the expected value of x when it is distributed according to the probability density function p(x), M o (x; θ) is a trained neural network model with parameters. is the square of the Frobenius norm, \bi is the value after removing the network block bi , α is the parameter in the adapter; A pruned neural network model with parameters and adapter parameters Compute network block B i Delay time: Among them, latM o is the delay time of the trained neural network model, The neural network model after pruning Delay time; Calculate the pruning score: Step S22: Selecting a pruned neural network model with the highest pruning score from the M pruned neural network models as the first trained neural network model.

8. A cross-domain small sample learning hyperspectral image classification device, characterized in that: The device comprises: Feature extraction module: configured to extract image features from the hyperspectral image to be classified; Classification module: configured to input the image features into the trained first neural network model to obtain the classification result of the hyperspectral image; The first neural network model is obtained by: Inputting the image feature corresponding to the training sample into the three-dimensional depth separable convolution module and the empty spectral coordinate attention module connected in sequence to obtain the first image feature corresponding to the training sample; Training a neural network model based on the first image feature to obtain a trained neural network model; The trained neural network model is pruned M times based on the block pruning module, and a pruned neural network model is obtained each time. The pruned neural network model with the highest pruning score is selected from the M pruned neural network models as the first trained neural network model.

9. An electronic device, comprising: A processor, which is used to execute multiple instructions; A memory for storing a plurality of instructions; The plurality of instructions are used to be stored in the memory and loaded and executed by the processor according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method as claimed in any one of claims 1 to 7.

Images