A hyperspectral image classification method based on deformable convolution

Abstract

The application discloses a hyperspectral image classification method based on deformable convolution, in the proposed spatial-spectral joint feature extraction model, first, multi-layer 1*1 convolution is used for spectral feature aggregation and extraction, then three-dimensional convolution layer is used to extract spatial-spectral joint features, finally, global average pooling is used to integrate global information while reducing parameters, and then the full connection layer is input to realize classification; the method extends the deformable convolution to the spectral dimension, selects the appropriate classification band for each surface feature through the offset on the spectral dimension, makes the extraction of the spectral feature concentrate on the effective band, improves the weight of the corresponding band value, and then extracts more discriminative features, and only needs to learn the offset on the spectral dimension, and the calculation complexity is only half of the spatial deformable convolution. The design method has very outstanding performances in classification accuracy, model complexity and sample generalization.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image classification technology, specifically to a hyperspectral image classification method based on deformable convolution. Background Technology

[0002] Hyperspectral Remote Sensing Image (HSI) consists of hundreds of consecutive bands with spectral resolution down to the nanometer scale, containing a wealth of spectral information. Furthermore, HSI integrates image and spectral characteristics, simultaneously utilizing rich spectral and spatial features to achieve precise classification of ground features. Currently, HSI is widely used in vegetation analysis, mineral resource development, and precision agriculture.

[0003] HSI (Hyperspective Sequence Injection) faces numerous challenges in classification and object recognition applications due to its large data volume, strong band correlation, redundant spectral features, and lack of labeled samples. Traditional HSI classification methods mainly include Support Vector Machines (SVMs), Multinomial Logistic Regression (MLR), and Random Forests. However, these methods rely on manually designed features, require rigorous model design and complex mathematical derivations, and cannot autonomously learn features based on data. Therefore, the robustness and applicability of these models need improvement.

[0004] In recent years, deep learning methods have greatly promoted the development of HSI classification. Deep learning is a neural network algorithm with a deep structure in machine learning. Its motivation is to build a neural network that simulates the human brain's analytical learning, combining low-level features in a hierarchical manner to autonomously learn deep features of images. Among them, Convolutional Neural Networks (CNNs), based on their unique convolutional operations, have a stronger feature extraction capability for spatial features and are widely used in HSI classification tasks. Methods such as 1D-CNN, 2D-CNN, and 3D-CNN have been developed. 1D-CNN compresses the image into a one-dimensional vector to extract spectral features; 2D-CNN uses two-dimensional convolutional kernels to extract features in the spatial dimension of the image, enabling the modeling of spatial information, but neglecting the extraction of spectral features; 3D-CNN extracts feature information in both the spectral and spatial dimensions simultaneously, fully utilizing the unified image-spectral characteristics of HSI, and achieving better classification results. How to use convolutional neural networks to fully extract the rich spatial-spectral features of HSI still requires further research.

[0005] Secondly, in image classification tasks, the receptive field is a crucial issue. Models typically require multi-scale receptive fields to capture features at different scales. To obtain global features, a large receptive field is needed to ensure that important information is not missed. On the one hand, increasing the convolutional kernel size can intuitively increase the receptive field; on the other hand, stacking more convolutional layers to make the network structure deeper will theoretically linearly increase the model's receptive field.

[0006] To address the receptive field problem, methods such as dilated convolution and deformable convolution have been developed. Dilated convolution expands the receptive field by increasing the sampling interval, while keeping the number of parameters constant. However, when stacked convolutional layers use the same sampling interval, a grid effect occurs, leading to information loss. Deformable convolution learns an offset for each sampling point through the convolutional layers, using the offset sampling points to achieve adaptive receptive field sampling. Compared to the fixed-interval sampling method of dilated convolution, the offset of deformable convolution is learnable. However, HSI typically uses block-based classification, limiting the spatial offset range of deformable convolution. Increasing the window size introduces interfering pixels, causing blurred boundaries and loss of detail. The application of deformable convolution in HSI classification requires further research. Summary of the Invention

[0007] The purpose of this invention is to provide a hyperspectral image classification method based on deformable convolution, which enhances the feature extraction capability and visual receptive field of the model. The technical problem to be solved by this method is as follows:

[0008] 1. How to fully extract the spatial-spectral joint features of HSI using CNN;

[0009] 2. How to enable the model to learn an adaptive receptive field with only a small number of parameters added;

[0010] 3. How to overcome the problems of excessively large window size and limited offset range in spatially deformable convolution.

[0011] To address the aforementioned technical problems, this invention employs the following technical solutions: First, a novel spatial-spectral feature extraction model is designed, achieving high classification accuracy with only a small number of convolutional layers. Second, deformable convolution is applied to spectral feature extraction, and a spectral deformable convolution module is proposed. This module achieves adaptive receptive field through offset sampling, concentrating spectral features in the effective band. Finally, the proposed model is combined with the spectral deformable convolution module to design a spectral deformable convolutional neural network (SDCNN).

[0012] To achieve the above functions, this invention designs a hyperspectral image classification method based on deformable convolution. For a target region, hyperspectral images of the target region containing various preset categories of land features are acquired. Step A is performed to preprocess the hyperspectral images. A spectral deformable convolutional network described in steps B-I is constructed and trained. This network is then applied to classify each pixel in the hyperspectral image, thereby achieving land feature classification in the target region.

[0013] Step A. For the acquired hyperspectral image of the target region, extract the hyperspectral data in tensor form, divide the hyperspectral data into blocks using a 9×9 window, and obtain the feature map of each block. For each feature map after segmentation, perform the following steps B-Step I:

[0014] Step B. For the feature map obtained in Step A, a 1×1×9 three-dimensional convolution module is used to sample each pixel of the feature map, and spectral features are extracted from the sampled points. At the same time, the spectral dimension of the feature map is reduced. The three-dimensional convolution module is in Valid form.

[0015] Step C. For the feature map obtained in step B, a 1×1×9 three-dimensional convolutional module is used to sample each pixel of the feature map, and spectral features are extracted from the sampled points. The three-dimensional convolutional module is in the same form and does not change the size of the feature map.

[0016] Step D. Construct a spectral dimension dilation module to increase the spectral dimension of the feature map. The spectral dimension dilation module takes the feature map obtained in step C as input and the feature map with increased spectral dimension as output.

[0017] In this step, the spectral dimension of the feature map obtained in step C is defined as M. Based on a 1×1×M, Valid three-dimensional convolution module, the feature map is convolved and tensor shape is transformed. The channel dimension of the three-dimensional convolution module is greater than the spectral dimension M of the feature map. The channel dimension and the spectral dimension are swapped to obtain the feature map with the increased spectral dimension.

[0018] Step E. Construct a spectral dimension compression module to compress the spectral dimension of the feature map. The spectral dimension compression module takes the feature map obtained in step D as input and the spectral dimension compressed feature map as output.

[0019] In this step, the spectral dimension of the feature map obtained in step D is defined as N. Based on a 1×1×N, Valid three-dimensional convolution module, the feature map is convolved and tensor shape is transformed. The channel dimension of the three-dimensional convolution module is smaller than the spectral dimension N of the feature map. The channel dimension and the spectral dimension are swapped to obtain the feature map with spectral dimension compression.

[0020] Step F. Based on the feature map obtained in step E, perform spatial-spectral joint feature extraction on the feature map using a 5×5×12 three-dimensional convolution module to obtain the spatial-spectral joint features of the feature map;

[0021] Step G. Based on the spatial-spectral joint features of the feature map obtained in step F, the spatial-spectral joint features are integrated into global spatial features using the spatial global average pooling method, wherein the pooling window size of the spatial global average pooling method is consistent with the spatial dimension size of the feature map.

[0022] Step H. Based on the global spatial features obtained in step G, the global spatial features are flattened into a one-dimensional tensor by introducing a Flatten layer;

[0023] Step I. For the one-dimensional tensor obtained in step H, a fully connected layer is introduced. The one-dimensional tensor is used as the input of the fully connected layer, and the probability of each pixel belonging to each preset category is used as the output. Based on the probability of each preset category output by the fully connected layer, the classification of each pixel in the hyperspectral image is completed, thereby realizing the classification of land cover in the target area.

[0024] As a preferred embodiment of the present invention: the three-dimensional convolution module mentioned in step B is a spectrally deformable convolution module, and its convolution process includes the following steps:

[0025] Step B1. For the input feature map, a regular cube R is used for sampling. The feature map dimensions are defined as (B, H, W, S, C). A fully connected layer is introduced to simulate the offset learning function. The offset learned is (B, H, W, S, C), which corresponds to the offset of each sampling point in one spectral dimension. Here, B represents the batch size of the model training, H is the feature map height, W is the feature map width, S is the spectral dimension size, and C is the number of channels of the three-dimensional convolution.

[0026] Step B2. Transform the shape of the feature map and offset into the form of (B×C,H,W,C);

[0027] Step B3. Generate resampling coordinates. Based on the shape of the feature map, generate three-dimensional coordinate points. The three-dimensional coordinate points are in the form of (B×C,H×W×C,3), where the last dimension 3 represents the coordinates of the three-dimensional coordinate points in three directions.

[0028] Step B4. Convert the offset to the form (B×C,H×W×C,1), which corresponds to the coordinate offset of each sampling point in the spectral dimension;

[0029] Step B5. Add the offset to the initial coordinates of the sampling point to generate the offset coordinate value of the sampling point, and impose a preset limit on the offset coordinate value;

[0030] Step B6. Round the offset coordinates up or down to the nearest integer, resample the feature map, and then use linear interpolation to obtain the feature values ​​of the offset sampling points.

[0031] Step B7. Convert the shape of the feature map obtained in step B6 into the form of (B, H, W, S, C) to obtain the offset feature map;

[0032] Step B8. Perform a 1×1 convolution on the feature map obtained in step B7 to complete the convolution process of the spectral deformable convolution module.

[0033] As a preferred embodiment of the present invention: the feature value y(p0) of each sampling point in the output feature map of the spectral deformable convolution module is as follows:

[0034]

[0035] In the formula, w(p) n The weights corresponding to each sampling point, p0+p n Let Δp be the coordinates of the sampling points other than the center point. n Let p be the offset learned through the fully connected layer, in the form of (0, 0, offset), where offset is the offset of the sampling point in the spectral dimension. n p1 represents the coordinates of the sampling point, and p0 represents the coordinates of the convolution center.

[0036] As a preferred technical solution of the present invention: the preset limitation in step B5 is to set coordinate values ​​lower than 0 to 0, and set coordinate values ​​exceeding the preset range to the maximum coordinate value in that dimension.

[0037] As a preferred technical solution of the present invention: the Flatten layer in step H adopts the tf.layers.flatten method in the TensorFlow framework.

[0038] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0039] A novel spatial-spectral feature extraction model was designed, which achieved high classification accuracy using only a small number of convolutional layers;

[0040] Secondly, deformable convolution was applied to spectral feature extraction. A spectral deformable convolution module was proposed, which achieves adaptive receptive field through offset sampling, so that spectral features are concentrated in the effective band, and more global spectral features are obtained for classification, giving full play to the advantage of high spectral resolution of HSI.

[0041] Finally, the proposed model is combined with a spectral deformable convolution module, which addresses the limitations of spatially deformable convolution in the field of hyperspectral image classification by extending deformable convolution to the spectral dimension. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of a spectrally deformable convolutional network provided according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of a spectral deformable convolution module provided according to an embodiment of the present invention;

[0044] Figure 3 This is a parameter sensitivity analysis diagram of SDCNN based on window size provided according to an embodiment of the present invention;

[0045] Figure 4 This is a sensitivity analysis diagram of SDCNN parameters based on batch size provided in an embodiment of the present invention;

[0046] Figure 5 This is a sample generalization analysis diagram of the methods provided in the embodiments of the present invention on three datasets;

[0047] Figure 6 These are classification graphs of different methods provided in embodiments of the present invention on the Indian Pines dataset;

[0048] Figure 7 These are classification graphs on the University of Pavia dataset using different methods provided in embodiments of the present invention;

[0049] Figure 8 This is a classification graph on the University of Houston dataset, showing different methods provided in embodiments of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0051] Compared to existing methods for expanding the receptive field, deformable convolution learns an offset for each sampling point through convolutional layers, achieving adaptive receptive field sampling using these offset sampling points. Unlike dilated convolution with its fixed sampling interval, the offset in deformable convolution is learnable, learning different offsets based on different input feature maps. Spatial deformable convolution compresses the adjacent structural information of each pixel into a fixed grid using deformable sampling points. Further applying regular convolutions to the deformable feature map can effectively reflect complex structures, thereby improving feature representation capabilities.

[0052] To address the problems existing in the application of spatially deformable convolution in HSI classification, this invention proposes a hyperspectral image classification method based on deformable convolution, extending deformable convolution to the spectral dimension and proposing spectral deformable convolution. Different land cover categories exhibit varying classification effects in different bands. By offsetting along the spectral dimension, appropriate classification bands can be selected for each land cover, concentrating spectral feature extraction on effective bands, increasing the weight of feature values ​​in corresponding bands, and thus extracting more discriminative features. Furthermore, it only requires learning the offset along the spectral dimension, resulting in a computational complexity that is only half that of spatially deformable convolution.

[0053] Reference Figure 1 This invention provides a hyperspectral image classification method based on deformable convolution, characterized by: acquiring hyperspectral images of ground features in a target area containing various preset categories; performing preprocessing of the hyperspectral images in step A; constructing and training a spectral deformable convolutional network as described in steps B-I; and applying the spectral deformable convolutional network to classify each pixel in the hyperspectral image, thereby achieving ground feature classification in the target area.

[0054] Step A. For the acquired hyperspectral image of the target region, extract the hyperspectral data in tensor form, divide the hyperspectral data into blocks using a 9×9 window, and obtain the feature map of each block. For each feature map after segmentation, perform the following steps B-Step I:

[0055] Step B. For the feature map obtained in Step A, a 1×1×9 three-dimensional convolution module is used to sample each pixel of the feature map, and spectral features are extracted from the sampled points. At the same time, the spectral dimension of the feature map is reduced. The three-dimensional convolution module is in Valid form.

[0056] The three-dimensional convolution module mentioned in step B is a spectrally deformable convolution module, as shown in the reference. Figure 2 Its convolution process includes the following steps:

[0057] Step B1. For the input feature map, a regular cube R is used for sampling. The feature map dimensions are defined as (B, H, W, S, C). A fully connected layer is introduced to simulate the offset learning function. The offset learned is (B, H, W, S, C), which corresponds to the offset of each sampling point in one spectral dimension. The shape of the offset learned is consistent with the shape of the input feature map. The offset corresponds to the offset coordinate of each sampling point in the input feature map in the spectral dimension. Where B represents the batch size of the model training, H is the feature map height, W is the feature map width, S is the spectral dimension size, and C is the number of channels of the three-dimensional convolution.

[0058] Step B2. Transform the shape of the feature map and offset into the form of (B×C,H,W,C);

[0059] Step B3. Generate resampling coordinates. Based on the shape of the feature map, generate three-dimensional coordinate points. The three-dimensional coordinate points are in the form of (B×C,H×W×C,3), where the last dimension 3 represents the coordinates of the three-dimensional coordinate points in three directions.

[0060] Step B4. Convert the offset to the form (B×C,H×W×C,1), which corresponds to the coordinate offset of each sampling point in the spectral dimension;

[0061] Step B5. Add the offset to the initial coordinates of the sampling point to generate the offset coordinate value of the sampling point, and impose a preset limit on the offset coordinate value;

[0062] In one embodiment, the preset constraint is implemented using tf.clip by value in TensorFlow. The preset constraint sets coordinate values ​​below 0 to 0 and sets coordinate values ​​exceeding the preset range to the maximum coordinate value in that dimension, so as to ensure that the offset only occurs in the feature cube.

[0063] Step B6. Round the offset coordinates up or down to the nearest integer, resample the feature map, and then use linear interpolation to obtain the feature values ​​of the offset sampling points.

[0064] Step B7. Convert the shape of the feature map obtained in step B6 into the form of (B, H, W, S, C) to obtain the offset feature map;

[0065] Step B8. Perform a 1×1 convolution on the feature map obtained in step B7 to complete the convolution process of the spectral deformable convolution module.

[0066] The 3D convolution module mainly includes two steps:

[0067] 1. Sample the input feature map X using a regular cube R.

[0068] 2. Perform a weighted summation of the sampled values. The regular cube R defines the receptive field and sampling interval of the 3D convolution. For example, R = {(-1, -1, -1), (-1, -1, 0), ..., (1, 1, 1)} defines a 3×3×3 convolution with a sampling interval of 0. Then, the feature value y(p0) of each sampled point in the output feature map y is:

[0069]

[0070] Building upon 3D convolution, the spectral deformable convolution module adds an offset in the spectral dimension. The feature value y(p0) of each sampling point in the output feature map of the spectral deformable convolution module is as follows:

[0071]

[0072] In the formula, w(p) n The weights corresponding to each sampling point, p0+p n Let Δp be the coordinates of the sampling points other than the center point. n Let p be the offset learned through the fully connected layer, in the form of (0, 0, offset), where offset is the offset of the sampling point in the spectral dimension. n p1 represents the coordinates of the sampling point, and p0 represents the coordinates of the convolution center.

[0073] Step C. For the feature map obtained in step B, a 1×1×9 three-dimensional convolutional module is used to sample each pixel of the feature map, and spectral features are extracted from the sampled points. The three-dimensional convolutional module is in the same form and does not change the size of the feature map.

[0074] Step D. Construct a spectral dimension dilation module to increase the spectral dimension of the feature map. The spectral dimension dilation module takes the feature map obtained in step C as input and the feature map with increased spectral dimension as output.

[0075] In this step, the spectral dimension of the feature map obtained in step C is defined as M. Based on a 1×1×M, Valid three-dimensional convolution module, the feature map is convolved and tensor shape is transformed. The channel dimension of the three-dimensional convolution module is greater than the spectral dimension M of the feature map. The channel dimension and the spectral dimension are swapped to obtain the feature map with the increased spectral dimension.

[0076] Step E. Construct a spectral dimension compression module to compress the spectral dimension of the feature map. The spectral dimension compression module takes the feature map obtained in step D as input and the spectral dimension compressed feature map as output.

[0077] In this step, the spectral dimension of the feature map obtained in step D is defined as N. Based on a 1×1×N, Valid three-dimensional convolution module, the feature map is convolved and tensor shape is transformed. The channel dimension of the three-dimensional convolution module is smaller than the spectral dimension N of the feature map. The channel dimension and the spectral dimension are swapped to obtain the feature map with spectral dimension compression.

[0078] Step F. Based on the feature map obtained in step E, perform spatial-spectral joint feature extraction on the feature map using a 5×5×12 three-dimensional convolution module to obtain the spatial-spectral joint features of the feature map;

[0079] Step G. Based on the spatial-spectral joint features of the feature map obtained in step F, the spatial-spectral joint features are integrated into global spatial features using the spatial global average pooling method, wherein the pooling window size of the spatial global average pooling method is consistent with the spatial dimension size of the feature map.

[0080] Step H. Based on the global spatial features obtained in step G, the global spatial features are flattened into a one-dimensional tensor by introducing a Flatten layer;

[0081] The Flatten layer uses the tf.layers.flatten method in the TensorFlow framework.

[0082] Step I. For the one-dimensional tensor obtained in step H, a fully connected layer is introduced. The one-dimensional tensor is used as the input of the fully connected layer, and the probability of each pixel belonging to each preset category is used as the output. Based on the probability of each preset category output by the fully connected layer, the classification of each pixel in the hyperspectral image is completed, thereby realizing the classification of land cover in the target area.

[0083] The hyperspectral image classification method based on deformable convolution designed in this invention was tested in practice using three internationally recognized hyperspectral remote sensing datasets: Indian Pines, University of Pavia, and University of Houston. The effectiveness of the proposed SDCNN classification model and its extensions was verified.

[0084] The validity verification process is as follows:

[0085] 1. Experimental setup

[0086] (1) Training sample setup

[0087] On the IP dataset, 10% of the samples from each class are randomly selected as the training set, and the remaining samples are used as the test set; on the PU dataset, 5% of the samples from each class are randomly selected as the training set, and the remaining samples are used as the test set; on the HU dataset, 50 samples from each class are selected as the training set, and the remaining samples are used as the test set.

[0088] In the sample generalization experiment, [1%, 2%, ..., 10%] of each class were randomly selected as training samples on the IP dataset; [0.5%, 1.0%, ..., 5.0%] of each class were randomly selected as training samples on the PU dataset; and [5, 10, ..., 50] of each class were selected as training samples on the HU dataset, with the remaining samples used as test samples.

[0089] (2) Selection of comparison method

[0090] To verify the superiority of the proposed algorithm, 2D CNN and 3D CNN methods were compared. Two classification methods based on spatially deformable convolutions, DCNNs (Deformable convolutional neural networks) and DHCNet (Deformable HSI classification networks), were selected for comparison, along with the multi-scale dilated residual CNN (MDR-CNN). Finally, three advanced deep learning classification methods, DR-CNN (Diverse Region-based CNN), SSRN (Spectral-Spatial Residual Network), and SSUN (Spectral-Spatial Unified Networks), were compared.

[0091] (3) Evaluation indicators

[0092] The classification results were quantitatively evaluated by statistically analyzing and comparing overall accuracy (OA), class accuracy (CA), mean accuracy (AA), and the Kappa coefficient (κ). For all classification algorithms used, the evaluation metric was the average of the results from 10 independent runs with randomly initialized training samples.

[0093] 2. Experimental Results

[0094] (1) Effect of regularization parameters

[0095] To analyze the impact of window size on the classification accuracy of the proposed method, experiments were conducted to test the classification performance under window sizes of [3×3, 5×5, 7×7, 9×9, 11×11, 13×13, 15×15]. Figure 3 As shown, the classification accuracy of the model on the three datasets initially increases and then decreases with the increase of the window size. When the window size increases, the number of features in the feature map also increases, so the classification accuracy initially shows an upward trend. However, when the window size increases to a certain extent, the excessively large feature map size increases the computational complexity of the model, affecting the learning of offsets and the offset effect; furthermore, it introduces interfering pixels, blurring the boundaries between classes and leading to a decrease in classification accuracy. Considering the model's classification performance on the three datasets, a window size of 9×9 is recommended.

[0096] (2) Batch size influence

[0097] To analyze the impact of batch size on the classification accuracy of the proposed method, experiments were conducted to test the classification performance under batch sizes of [16, 32, 48, 64, 80, 96]. Figure 4As shown, on the IP and PU datasets, batch size has a relatively small impact on model classification accuracy, and the accuracy change curve is relatively flat. On the HU dataset, however, classification accuracy shows a significant upward trend as the batch size increases. It can be seen that the model achieves the best classification results on all three datasets when the batch size is 64. Therefore, the batch size for the model is set to 64.

[0098] (3) Sample generalization analysis of different methods

[0099] To verify the generalization ability of the model under different sample conditions, experiments were conducted on three datasets with different numbers of training samples. Figure 5 The classification results of each method under different sample conditions in three datasets are plotted, with the x-axis representing the number of training samples selected for each class and the y-axis representing the overall classification accuracy. For the IP dataset, except for DCNN and DHCNet, the classification accuracy of all methods steadily increases with the increase of the number of classification samples, with SDCNN consistently maintaining the highest classification accuracy. For the PU dataset, the accuracy increase trend is relatively gradual, with SDCNN achieving a classification accuracy of 99% when each class has 1.0% of the samples. For the HU dataset, only the MDR-CNN method shows significant fluctuations. The method presented in this paper achieves the highest classification accuracy in most cases, especially with a smaller sample size, where the accuracy improvement is more significant.

[0100] (4) Classification performance analysis of different algorithms on IP dataset

[0101] Table 1 lists the classification accuracy of each method on the IP dataset. The proposed method (SDCNN) achieved the highest classification accuracy, with an OA of 98.86%, which is 0.22%-21.15% higher than other methods. Furthermore, the standard deviation of the 10-fold results is significantly smaller than that of other comparative methods, indicating that SDCNN is less affected by sample selection. SDCNN also achieved the highest Kappa score. For individual classes, SDCNN achieved the highest classification accuracy in 6 out of 16 classes.

[0102] Figure 6 The classification maps obtained by each method on the IP data show that SDCNN significantly reduces speckle noise, has higher intra-class smoothness, clearer class boundaries, and more accurate classification results. DCNN and DHCNet methods use large windows, leading to blurred class boundaries and reduced detail extraction capabilities. Compared with ground validation data, SDCNN classifies Hay-windrowed and Woods categories very accurately, corresponding to the high classification accuracy of these categories in the table.

[0103] Table 1

[0104]

[0105] (5) Classification performance analysis of different algorithms on the PU dataset

[0106] Table 2 lists the classification accuracy of each method on the PU dataset. SDCNN achieves an OA of 99.81 ± 0.06%, which is 0.28%–9.64% higher than other methods, and also achieves the highest Kappa and AA. For individual classes, SDCNN achieves the highest classification accuracy in 6 out of 9 classes on this dataset.

[0107] Figure 7 For the classification maps of the PU data and the classification results obtained by each model, SDCNN achieved smoother and more accurate results. Compared with the ground validation data, the SDCNN classification results map is very accurate in the corresponding positions of categories such as Meadows and Bitumen, which corresponds to the high classification accuracy of specific categories in Table 2.

[0108] Table 2

[0109]

[0110] (6) Classification performance analysis of different algorithms on the HU dataset

[0111] Table 3 lists the classification accuracy of each method on the HU dataset. SDCNN achieved the highest OA, Kappa, and AA, with OA at 97.41±0.59%, representing an improvement of 0.68%-12.05% compared to other models. For specific categories, SDCNN achieved the highest classification accuracy in 10 out of 15 categories.

[0112] Figure 8 The table shows the classification results of the HU data and the classification results obtained by each method. From the classification results, the classification results of SDCNN are more continuous and smooth, with significantly fewer noise points than other methods, which corresponds to the classification accuracy of the models in Table 2.

[0113] (7) Complexity analysis of different algorithms

[0114] To examine the complexity of different methods, the complexity of each method was analyzed using runtime and parameter size, as shown in Table 3. Among all the compared methods, SDCNN uses the fewest parameters because it uses only a small number of convolutional layers for feature extraction and has a relatively small number of channels. DR-CNN trains six models, resulting in a large number of parameters, second only to 3D-CNN. In terms of runtime, SDCNN has longer training and testing times because deformable convolution involves feature interpolation operations, thus increasing the overall runtime. DR-CNN, on the other hand, requires training six models, resulting in a significantly longer overall computation time compared to other methods.

[0115] Table 3

[0116]

[0117] As can be seen from the above embodiments, the hyperspectral image classification method based on deformable convolution designed in this invention first uses 1×1 convolution to extract and reduce the dimensionality of spectral features in the basic model, and then inputs them into the spectral dimension dilation and compression module to achieve spectral feature aggregation. A three-dimensional convolutional layer is used to simultaneously extract joint spatial and spectral features, and finally, global average pooling is used to integrate global information. In the spectral deformable convolution module, firstly, an additional fully connected layer simulates the offset learning function to learn the offset in the spectral dimension for each sampling point. This offset is adaptively adjusted according to different feature maps, thereby achieving the purpose of adaptive receptive field. Secondly, since the learned offset is usually not an integer, the offset coordinates are rounded up and down. Using the feature values ​​of the rounded coordinates, linear interpolation is used to obtain the offset feature values, generating the offset feature map. Finally, a 1×1 convolution is performed on the offset feature map to achieve spectral deformable convolution. The method designed in this invention has outstanding performance in terms of model parameter quantity, classification accuracy, and sample generalization, proving the effectiveness of the proposed method.

[0118] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A hyperspectral image classification method based on deformable convolution, characterized in that, For the target area, hyperspectral images of land features of various preset categories are acquired. Step A is performed to preprocess the hyperspectral images. The spectrally deformable convolutional network described in steps B-I is constructed and trained. The spectrally deformable convolutional network is then applied to classify each pixel in the hyperspectral image, thereby achieving land feature classification in the target area. Step A. Extract hyperspectral data in tensor form from the acquired hyperspectral image of the target region, in order to... The hyperspectral data is segmented into blocks using a window to obtain feature maps for each block. For each segmented feature map, the following steps B-I are performed: Step B. For the feature map obtained in Step A, use... The 3D convolution module samples each pixel of the feature map, extracts spectral features from the sampled points, and reduces the spectral dimension of the feature map. The 3D convolution module is in Valid form. The three-dimensional convolution module mentioned in step B is a spectrally deformable convolution module, and its convolution process includes the following steps: Step B1. For the input feature map, a regular cube R is used for sampling. The feature map dimensions are defined as (B, H, W, S, C). A fully connected layer is introduced to simulate the offset learning function. The offset learned is (B, H, W, S, C), which corresponds to the offset of each sampling point in one spectral dimension. Here, B represents the batch size of the model training, H is the feature map height, W is the feature map width, S is the spectral dimension size, and C is the number of channels of the three-dimensional convolution. Step B2. Convert the shape of the feature map and offset to... form; Step B3. Generate resampling coordinates. Based on the shape of the feature map, generate three-dimensional coordinate points. The three-dimensional coordinate points are in the following form: The last dimension, 3, represents the coordinates of a three-dimensional point in three directions; Step B4. Convert the offset to... The form corresponds to the coordinate offset of each sampling point in the spectral dimension; Step B5. Add the offset to the initial coordinates of the sampling point to generate the offset coordinate value of the sampling point, and impose a preset limit on the offset coordinate value; Step B6. Round the offset coordinates up or down to the nearest integer, resample the feature map, and then use linear interpolation to obtain the feature values ​​of the offset sampling points. Step B7. Convert the shape of the feature map obtained in step B6 into the form of (B, H, W, S, C) to obtain the offset feature map; Step B8. Perform the following steps on the feature map obtained in step B7: Convolution, completing the convolution process of the spectral deformable convolution module; Step C. For the feature map obtained in step B, use... The 3D convolution module samples each pixel of the feature map and extracts spectral features from the sampled points. The 3D convolution module is in the same form and does not change the size of the feature map. Step D. Construct a spectral dimension dilation module to increase the spectral dimension of the feature map. The spectral dimension dilation module takes the feature map obtained in step C as input and the feature map with increased spectral dimension as output. Wherein, the spectral dimension of the feature map obtained in step C is defined as M ,based on A valid 3D convolutional module performs convolution on the feature map and tensor shape transformation, and the channel dimension of this 3D convolutional module is greater than the spectral dimension of the feature map. M By exchanging the channel dimension and the spectral dimension, a feature map with the increased spectral dimension is obtained. Step E. Construct a spectral dimension compression module to compress the spectral dimension of the feature map. The spectral dimension compression module takes the feature map obtained in step D as input and the spectral dimension compressed feature map as output. Wherein, the spectral dimension of the feature map obtained in step D is defined as N ,based on A valid 3D convolutional module performs convolution on the feature map and tensor shape transformation, and the channel dimension of this 3D convolutional module is smaller than the spectral dimension of the feature map. N By exchanging the channel dimension and the spectral dimension, a feature map after spectral dimension compression is obtained; Step F. Based on the feature map obtained in step E, The 3D convolution module performs spatial-spectral joint feature extraction on the feature map to obtain the spatial-spectral joint features of the feature map; Step G. Based on the spatial-spectral joint features of the feature map obtained in step F, the spatial-spectral joint features are integrated into global spatial features using the spatial global average pooling method, wherein the pooling window size of the spatial global average pooling method is consistent with the spatial dimension size of the feature map. Step H. Based on the global spatial features obtained in step G, the global spatial features are flattened into a one-dimensional tensor by introducing a Flatten layer; Step I. For the one-dimensional tensor obtained in step H, a fully connected layer is introduced. The one-dimensional tensor is used as the input of the fully connected layer, and the probability of each pixel belonging to each preset category is used as the output. Based on the probability of each preset category output by the fully connected layer, the classification of each pixel in the hyperspectral image is completed, thereby realizing the classification of land cover in the target area.

2. The hyperspectral image classification method based on deformable convolution according to claim 1, characterized in that, The spectral deformable convolution module outputs feature values ​​of each sampling point in the feature map. As shown in the following formula: ； In the formula, The weight corresponding to each sampling point, These are the coordinates of the sampling points other than the center point. Let be the offset learned through the fully connected layer, in the form of (0, 0, offset), where offset is the offset of the sampling point in the spectral dimension. The coordinates of the sampling point, The coordinates are the center position of the convolution.

3. The hyperspectral image classification method based on deformable convolution according to claim 1, characterized in that, The preset limit mentioned in step B5 is to set coordinate values ​​below 0 to 0, and set coordinate values ​​exceeding the preset range to the maximum coordinate value in that dimension.

4. The hyperspectral image classification method based on deformable convolution according to claim 1, characterized in that, The Flatten layer in step H uses the tf.layers.flatten method in the TensorFlow framework.

Images