A hyperspectral image mining area tree species classification method combining cascaded 3DCNN and SDTA encoding

Abstract

The application discloses a hyperspectral image mining area tree species classification method combining cascaded 3DCNN and SDTA coding, and comprises the following steps: constructing a cascaded 3D convolutional neural network to extract local spectral features and spatial features; performing dynamic updating on channel dimension feature weights through a segmentation depth transposed attention encoder; extracting and separating features between tree species with similar spectra in mining area tree species classification through combination of the 3D convolution and the segmentation depth transposed attention encoder; constructing a module fusing high-level features and low-level features based on residual connection, and finally flattening the features to input into a fully connected layer, and outputting final classification probability by using a softmax activation function. A mining area tree species classification deep network suitable for airborne hyperspectral data is constructed, and the high-dimensional feature extraction capability is improved through the cascaded 3D convolutional network architecture.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of mine area tree species classification, and particularly relates to a hyperspectral image mine area tree species classification method combining a cascade 3DCNN and SDTA coding. BACKGROUND

[0002] Mine exploitation causes problems such as surface subsidence, underground water level drop and vegetation degradation, which seriously threaten ecological safety. China is accelerating the promotion of mine ecological restoration and the development of green mines. Vegetation plays an important role in water conservation, soil conservation, carbon balance regulation and regional microclimate mitigation, so vegetation restoration is regarded as a key link of mine ecological restoration, and afforestation is regarded as an important means of vegetation restoration. Fine classification of tree species is an important basis for revealing the spatiotemporal evolution law of mine area vegetation restoration and reasonably planning and designing the mine area ecological restoration mode, helps to deeply understand the complexity and diversity of forest ecosystems, helps to effectively prevent and control forest diseases, guides mine area ecological management and restoration, optimizes the utilization and protection of forest resources, and further coordinates the symbiotic relationship between man and nature, which has important scientific guiding significance and practical significance for mine area ecological restoration.

[0003] Different tree species have unique spectral characteristics, and the hundreds of band reflection information of hyperspectral images have great potential in tree species classification, especially with the rapid development of unmanned aerial vehicles, airborne hyperspectral data can make up for the time-consuming and laborious defects of conventional field investigation. However, with the increase of spectral channel dimension, problems such as increased data volume, reduced correlation between bands, and difficult feature extraction occur, which leads to reduced final classification accuracy and efficiency. Therefore, the key to using hyperspectral data for tree species classification is to explore an efficient and accurate feature extraction scheme.

[0004] In recent years, tree species supervised classification technology based on hyperspectral remote sensing technology has been developed. For example, machine learning algorithms such as support vector machine, random forest and stacked ensemble have been applied in tree species classification, proving that it is feasible to use spectral information differences of tree species to distinguish between classes. However, machine learning requires manual feature construction and selection, and it is difficult to capture complex patterns between data in tree species classification tasks with complex relationships between data classes. In addition, the spatial variation of tree species spectral characteristics, i.e. spectral uncertainty, makes it difficult for the extracted features to adapt to all scenarios, although the spectral information fusion technology can make up for the single knowledge provided by hyperspectral data, but these methods are heavily dependent on prior knowledge and expert domain knowledge. In addition, deep learning methods that realize end-to-end automatic feature extraction have been widely applied in remote sensing image classification tasks. In particular, convolutional neural networks have attracted widespread attention due to their excellent feature extraction capabilities, and many scholars have applied CNN and its variants to tree species classification tasks.

[0005] Although the existing tree species classification methods have achieved good results, there are still the following problems.

[0006] (1) High-dimensional feature extraction requires complex network structure and superposition of multiple layers of network, which causes network structure parameter quantity to increase rapidly, making the network more difficult to train.

[0007] (2) The traditional convolutional neural network has a limited convolution kernel receptive field, and it is difficult to capture global feature information. Especially in the mine repair area, the crown of the coniferous tree species with a short repair period is small, and the features are not obvious, and the exposed soil under the forest is prone to misclassification and confusion.

[0008] (3) The repair period of the ecological repair area in the mining area is different, the tree age is different, the same object has different spectrum, and further automatic extraction of high-level features is required for the tree species classification task. SUMMARY

[0009] The purpose of the present application is to provide a hyperspectral image mining area tree species classification method combined with cascaded 3DCNN and SDTA coding, to construct a mining area tree species classification deep network suitable for airborne hyperspectral data, and to improve the high-dimensional feature extraction capability through cascaded 3D convolution network architecture.

[0010] The technical scheme of the present application is as follows:

[0011] A hyperspectral image mining area tree species classification method combined with cascaded 3DCNN and SDTA coding comprises the following steps:

[0012] Construct a cascaded 3D convolutional neural network to extract local spectral features and spatial features;

[0013] Update the channel dimension feature weight dynamically through a segmentation depth transposed attention encoder;

[0014] After combining 3D convolution with a segmentation depth transposed attention encoder, the features between tree species with similar spectra are extracted and separated in the mining area tree species classification;

[0015] Based on residual connection, a module is constructed to fuse high-level features and low-level features, and finally the features are flattened and input into a fully connected layer, and a softmax activation function is used to output the final classification probability.

[0016] Further, the cascaded 3D convolutional neural network for extracting local spectral features and spatial features specifically comprises the following steps:

[0017] First, perform principal component dimensionality reduction on the original hyperspectral image;

[0018] The feature bands after principal component dimensionality reduction are sequentially input into two consecutive 3D convolution layers to obtain output feature one;

[0019] The output feature one is further input into three 3D convolution layers to obtain an output feature two;

[0020] The output feature one and the output feature two are fused and connected, and then input into three 3D convolution layers to obtain an output feature three;

[0021] The output feature one, the output feature two and the output feature three are connected and fused to be input into a next network component.

[0022] Further, the algorithm of the 3D convolution is as follows:

[0023] ,

[0024] wherein, is an activation value of a feature map in the i-th layer at a position x, is a bias of the feature map in the i-th layer, is an index of all feature maps in the i-th layer, is a value of a convolution kernel connecting the feature map in the i-th layer and the feature map in the j-th layer at a position x, , , , , , , , , , , , , , are respectively a height, a width and a depth of the convolution kernel.

[0025] Further, the split depth transpose attention encoder comprises a feature encoding module and a self-attention module, and in the feature encoding module:

[0026] The input feature is divided into n subsets by direct channel splitting; The number of subsets is adaptive based on a level sequence;

[0027] The calculation manner of each subset is that the output feature of a previous subset is fused and then encoded by a depth-with convolution in the i-th layer to obtain a feature in the i-th layer.

[0028] The calculation manner of each subset is that the output feature of a previous subset is fused and then encoded by a depth-with convolution in the i-th layer to obtain a feature in the i-th layer.

[0029] Finally, the features of the subsets are spliced to obtain an output feature with a multi-scale receptive field. ​​​​​​

[0030] Furthermore, the aforementioned The expression is:

[0031] .

[0032] Furthermore, in the self-attention module:

[0033] Input features Modified via reshape The query is obtained after passing through three layers. ,key ,value Before calculating the cross-covariance attention, by... , application Norm stability training;

[0034] Only for Transpose. get The calculation results are combined with softmax to obtain the attention score matrix, which is then applied. The final attention map is obtained.

[0035] Furthermore, the self-attention module includes:

[0036] ,

[0037] ,

[0038] in, For input, To output a tensor.

[0039] Furthermore, the module for constructing the fusion of high-level and low-level features based on residual connections specifically includes the following steps:

[0040] The features output by the depth transpose attention are passed through a 3D convolutional layer for preliminary feature extraction and batch normalization is applied. Then, they are passed through a 3D convolutional layer with a larger kernel to capture deeper structural information and are batch normalized again. After passing through another 3D convolutional layer, batch normalization is performed again to obtain deep high-level features.

[0041] The parallel convolutional branches start from the previous feature extraction, go through 3D convolution and batch normalization in sequence, and finally add to the output of the deep feature extraction branch to form a residual connection;

[0042] The final output is converted into a one-dimensional vector through a flattening layer and then input into a fully connected layer for classification.

[0043] The output layer using a softmax activation function produces a classification decision.

[0044] Further, the sizes of the 3-dimensional convolution layers are different.

[0045] Further, the principal component dimension reduction on the original hyperspectral image is specifically retaining the first 30 principal components through spectral feature dimension reduction.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] 1. A hyperspectral image mining area tree species classification method combining cascaded 3DCNN and SDTA coding, a mining area tree species classification deep network suitable for airborne hyperspectral data is constructed, the high-dimensional feature extraction capability is improved through the cascaded 3D convolution network architecture; the self-attention mechanism (Self-Attention) is introduced, the feature extraction capability of the model is improved, and the global information of the hyperspectral image is better captured; the residual connection (Residual Connection) is used to connect the entire network model, so that the classification network is more stable, and the final tree species classification precision is improved.

[0048] 2. A hyperspectral image mining area tree species classification method combining cascaded 3DCNN and SDTA coding, the classification network not only provides technical support for tree species classification in the mining area ecological restoration area, but also has wide applicability to other tree species classification scenes, and provides decision support for forest resource inventory and forestry intelligent management. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 It is a cascaded 3DCNN network structure of the present application.

[0050] Figure 2 It is an SDTA encoder structure diagram of the present application.

[0051] Figure 3 It is a residual connection module diagram of the present application.

[0052] Figure 4 It is a comparison diagram of tree species classification results of different methods of the present application (a): SDTA-3DCNN, (b): 2DCNN, (c): 3D-1DCNN, (d): 3DCNN, (e): RF, (f): SVM, (g): Stacking.

[0053] Figure 5 It is a comparison diagram of classification accuracy of different categories of the present application.

[0054] Figure 6The overall flowchart of the present application. DETAILED DESCRIPTION

[0055] It should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0056] The features and performance of the present application will be further described in detail below in conjunction with embodiments.

[0057] Referring to Figures 1-6 , a hyperspectral image mining area tree species classification method combining cascaded 3D CNN and SDTA encoding, as shown in Figure 6 , includes the following steps:

[0058] A cascaded 3D convolutional neural network is constructed to extract local spectral features and spatial features; the structure is as shown in Figure 1 .

[0059] Through a split depth transposed attention (SDTA) encoder, the channel dimension feature weight is dynamically updated; SDTA achieves efficient combination while not increasing the additional parameter amount and multiplication-addition operation amount. SDTA mainly includes dynamic multi-scale feature fusion and global information encoding, and the structure of the SDTA encoder is as shown in Figure 2 .

[0060] After combining 3D convolution with a split depth transposed attention encoder, the features between tree species with similar spectra are extracted and separated in the mining area tree species classification;

[0061] Based on residual connection, a module that fuses high-level features and low-level features is constructed, and finally the features are flattened and input to the fully connected layer, and a softmax activation function is used to output the final classification probability.

[0062] The cascaded 3D convolutional neural network extracts local spectral features and spatial features, which specifically includes the following steps:

[0063] First, the original hyperspectral image is subjected to principal component dimension reduction, and the first 30 principal components are retained through spectral feature dimension reduction;

[0064] The feature bands that have undergone principal component dimensionality reduction are sequentially passed through two consecutive 3D convolutional layers to obtain output feature one.

[0065] The output feature one is then passed through three 3D convolutional layers to obtain the output feature two.

[0066] Output feature 1 and output feature 2 are fused and concatenated, and then passed through 3 layers of 3D convolutional layers to obtain output feature 3;

[0067] Output feature 1, output feature 2, and output feature 3 are concatenated and fused before being input into the next network component. The dimensions of the aforementioned 3D convolutional layers are all different.

[0068] The algorithm for 3D convolution is as follows:

[0069] ,

[0070] in, It is the first Layer Each feature is mapped at position activation value, It is the first Layer The deviation of a feature mapping Index No. All feature maps of the layer, To connect the first The first layer The feature map and the first Layer and first The convolution kernels for each feature map are located at... The value, , , These are the height, width, and depth of the convolution kernel, respectively.

[0071] The segmentation depth transpose attention encoder consists of a feature encoding module and a self-attention module. In the feature encoding module:

[0072] Input features It is divided into by direct channel segmentation. Subset

[0073] subset The quantity is based on the series Adaptive;

[0074] Except for the first subset, the calculation method for each subset is to fuse the output features of the previous subset and then... depth-with convolution Perform feature encoding ;

[0075] Finally, the features of each subset are spliced to obtain output features with multi-scale receptive fields.

[0076] The expression is:

[0077] .

[0078] In the self-attention module:

[0079] The input features are modified by reshape to , and three linear layers are used to obtain query , key , and value . Before calculating cross-covariance attention, apply , L2 norm to stabilize training;

[0080] Only transpose , to obtain . The attention score matrix is obtained by combining the calculation results with softmax, and is applied to to obtain the final attention map.

[0081] The self-attention module includes:

[0082] ,

[0083] ,

[0084] wherein, is the input, is the output tensor.

[0085] After combining 3D convolution with channel attention mechanism, in the classification of tree species in mining areas, the features between tree species with similar spectra, such as similar coniferous tree species Yunnan pine and Huashan pine, can be effectively extracted and separated, which can improve the model's understanding of complex spatial relationships, effectively reduce irrelevant or redundant feature information, and thus improve the accuracy and efficiency of classification or recognition. Of course, this combination also faces some difficulties, mainly reflected in the increase of model design and computational complexity. Integrating 3D convolution and channel attention mechanism requires fine-tuning of superparameters and complex model architecture, while increasing the computational cost and resource demand in the training and inference process. SDTA, as a lightweight attention mechanism module, effectively realizes the lightweight of the model through adaptive channel cutting, speeds up the inference speed, and the fastest training speed reaches 8s / epoch.

[0086] ​The module for fusing high-level features and low-level features based on residual connection comprises the following steps:

[0087] The features output by the split depth transpose attention are input into a 3D convolution layer for preliminary feature extraction, batch normalization is applied, and then a 3D convolution layer with a larger convolution kernel is used to capture deeper structural information, followed by batch normalization, and then a 3D convolution layer, and batch normalization again to obtain depth high-level features.

[0088] The parallel convolution branch starts from the pre-sequence feature extraction and sequentially passes through a 3D convolution and batch normalization, and finally adds the output of the depth feature extraction branch to form a residual connection.

[0089] The final output is converted into a one-dimensional vector through a flattening layer and input into a fully connected layer for classification. A Dropout layer (dropout rate of 0.5) is provided after the fully connected layer to prevent overfitting.

[0090] The output layer using a softmax activation function produces a classification decision.

[0091] Help the model capture constant and low-level features and directly pass them to the deep part of the network, solve the problem of gradient disappearance that may occur in the deep network, and help the network train better.

[0092] Accuracy evaluation:

[0093] In order to quantitatively evaluate the performance of the classification results, F1-score, OA (Overall Accuracy) and Kappa are introduced as evaluation indicators. F1-score is used to evaluate the classification effectiveness of a single class, and OA and Kappa are used to evaluate the overall classification effectiveness.

[0094] ,

[0095] ,

[0096] ,

[0097] .

[0098] All experiments were implemented on a desktop computer equipped with an Intel(R) Core(TM) i7-9700K CPU, 256.0 GB of memory, and an NVIDIA GeForce RTX3080. All algorithms were executed on Jupyter Notebook using Python language (v3.8.0). The deep learning framework was TensorFlow (V2.5.0). During the experiment, the optimal parameter selection for training the network was determined by manual tuning. The network hyperparameters are shown in Table 1.

[0099] Table 1 Hyperparameter settings of the network

[0100]

[0101] Comparison of different classification methods:

[0102] In order to verify the classification effect of the proposed method on hyperspectral data, seven classification methods including RF, SVM, Stacking, 3DCNN, 2DCNN, 3D-1DCNN and the proposed SDTA-3DCNN are compared and analyzed. All these classification methods use the same data set and the same data division method.

[0103] In order to verify the classification effectiveness of the proposed method, six methods of machine learning and deep learning are selected to classify the tree species in the repair area of the tailings pond in the mining area, Figure 4 The hyperspectral image (RGB band synthesis) and the classification results of the seven classification methods are shown. The quantitative evaluation results are shown in Table 2. Through visual interpretation, it is found that the three machine learning methods show high consistency in the classification results, and the similarity between the classification results of the four deep learning methods is high. However, there is a big difference in the spatial distribution of the classification results between the machine learning classification method and the deep learning classification network, especially between the Pinus densata and Pinus yunnanensis. The reason for this phenomenon may be that the spectral characteristics of the two species are similar, and it is difficult for traditional machine learning algorithms to extract deep features. From the quantitative evaluation results, among the seven methods compared, the proposed SDTA3DCNN network has the highest overall classification accuracy (OA=99.03%, Kappa=0.99), and the worst classification accuracy is SVM (OA=89.51%, Kappa=0.85). The highest accuracy among the machine learning classification methods is Stacking (OA=94.65%, Kappa=0.93). Although the overall classification accuracy of each deep learning classification network is not much different, from the quantitative evaluation of each class classification result, the SDTA3DCNN classification network performs better in each class ( Figure 5 ), and even in the case of sample imbalance, the classification accuracy is still high, for example, the Tibetan cypress has only 2501 training samples, but the classification accuracy is the highest among all methods (Precision=97.49%, recall=0.93, F1-score=0.95).

[0104] Table 2 Comparison of classification accuracy of different methods

[0105]

[0106] The above embodiments only express the specific implementation of the present application, which is described in more detail and specifically, but cannot be understood as a limitation to the protection scope of the present application. It should be noted that for those skilled in the art, without departing from the technical concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A method for hyperspectral image mining area tree species classification combined with cascaded 3D CNN and SDTA encoding, characterized in that, The method comprises the following steps: A cascaded 3D convolutional neural network is constructed to extract local spectral features and spatial features; A segmented depth transposed attention encoder is used to dynamically update the channel dimension feature weight; The 3D convolution is combined with the segmented depth transposed attention encoder to extract and separate the features between tree species with similar spectra in a mining area; A module is constructed based on residual connection to fuse high-level features and low-level features, and finally the features are flattened and input into a fully connected layer, and a softmax activation function is used to output the final classification probability; The cascaded 3D convolutional neural network for extracting local spectral features and spatial features comprises the following steps: First, the original hyperspectral image is subjected to principal component dimension reduction; The feature bands subjected to principal component dimension reduction are sequentially subjected to two consecutive 3D convolution layers to obtain output feature one; Output feature two is obtained by subjecting output feature one to three 3D convolution layers; Output feature three is obtained by fusing and connecting output feature one and output feature two and then subjecting them to three 3D convolution layers; Output feature one, output feature two and output feature three are connected and fused and input into the next network component; The algorithm of the 3D convolution is as follows: ， wherein, is the activation value of the layer at the th feature map at position , is the bias of the layer at the th feature map, indexes all feature maps of the th layer, is the value of the th feature map of the th layer and the th feature map of the th layer at position , , , are the height, width and depth of the convolution kernel, respectively. 2.The method of claim 1, wherein the method is characterized in that, The segmented depth transposed attention encoder comprises a feature encoding module and a self-attention module, in the feature encoding module: input features are divided into subsets subset number of based on series adaptive; Each subset is calculated by fusing the output features of the previous subset and passing them through a depth-with convolution ;​​ Finally, the various subset features are spliced to obtain output features with a multi-scale receptive field. 3.The method of claim 2, wherein, The Expression is: 。 4. The hyperspectral image mining area tree species classification method of combined cascade 3DCNN and SDTA encoding according to claim 2 or 3, characterized in that, In the self-attention module: The input features are reshaped By reshaping to , three line layers are obtained to query , key , value , before calculating the cross-covariance attention, the , application norm stable training; only to transpose, get , the calculation results combined with softmax get attention score matrix, applied to get the final attention map.

5. The hyperspectral image mining area tree species classification method of combined cascade 3DCNN and SDTA encoding according to claim 4, characterized in that, The self-attention module comprises: ， ， wherein, is an input, is an output tensor. 6.The method of claim 1, wherein the method is characterized in that, The module for fusing high-level features and low-level features based on residual connection comprises the following steps: The features output by the segmented depth transposed attention are subjected to a 3D convolution layer for preliminary feature extraction, batch normalization is applied, and then a 3D convolution layer with a larger convolution kernel is used to capture deeper structural information, followed by batch normalization processing, then a 3D convolution layer, and batch normalization again to obtain deep high-level features; A parallel convolution branch sequentially passes through a 3D convolution and batch normalization starting from the previous feature extraction, and finally adds the output of the deep feature extraction branch to form a residual connection; The final output is converted into a one-dimensional vector by a flattening layer and input into a fully connected layer for classification; The output layer uses a softmax activation function to make a classification decision.

7. The hyperspectral image mining area tree species classification method of combined cascade 3DCNN and SDTA encoding according to claim 1, characterized in that, The sizes of the 3D convolution layers are different. 8.The method of claim 1, wherein the method is characterized in that, The principal component dimension reduction of the original hyperspectral image is specifically performed by retaining the first 30 principal components through spectral feature dimension reduction.

Images