Tree species classification method and system based on spectral depth extraction convolutional neural network SDA-CNN

Abstract

The invention discloses a tree species classification method and system based on a spectral depth extraction convolutional neural network SDA-CNN, and can overcome the technical defects that the utilization efficiency of spectral information in a hyperspectral image is relatively low, and effective features of each spectral channel cannot be effectively extracted, so that the tree species classification performance is limited. And finally, tree species classification performance and classification accuracy are improved. The tree species classification method comprises the steps of using a deep learning sample set constructed by hyperspectral forest image data to perform model training and tree species classification on a spectral depth extraction convolutional neural network SDA-CNN model, adopting a hybrid neural network module comprising two-dimensional convolution and depth-by-depth convolution in the SDA-CNN model, and newly arranging a field pixel sensing module, the two-dimensional convolution is used for extracting global spatial information of the image to generate global features; the depth-by-depth convolution is used for independently executing convolution operation on each spectrum channel without sharing weight among the spectrum channels so as to extract correlation information among the spectrum channels, and the domain pixel sensing module is used for extracting environment information around a target pixel contained in adjacent pixels.

Description

Technical Field

[0001] The embodiments of the present application relate to the fields of computer vision and deep learning hyperspectral imaging technology, and in particular to a tree species classification method and system based on a spectral deep extraction convolutional neural network (SDA-CNN). Background Art

[0002] As a hub for material circulation and energy exchange, forests have the key functions of regulating climate, conserving water resources and maintaining biodiversity. Carrying out forest resource surveys and monitoring is of great significance for the protection and sustainable development of forest resources. In order to improve the automation and intelligence level of forest surveys, this patent aims to perform refined image classification of trees based on hyperspectral drone image data to achieve automatic identification of trees, reduce waste of human resources, and achieve a win-win situation of ecological protection and cost savings.

[0003] Traditional tree species identification methods mainly rely on manual field surveys to determine tree species by observing the external morphology of trees. Although this method has excellent accuracy, it has great limitations in practical applications, especially in areas with inconvenient transportation and complex terrain, where the difficulty and risk of investigation are significantly increased.

[0004] In the early applications of forest remote sensing vegetation classification and mapping, the spectrum, vegetation index, and phase differences were used as features by analyzing single-phase or multi-phase multispectral images, and forest vegetation classification was achieved through unsupervised clustering (such as K-means), syntactic classification (such as decision trees), or statistical classifiers (such as maximum likelihood classification). However, due to the small spectral differences between forest tree species, traditional methods have difficulty distinguishing the subtle spectral differences between different tree species, and they rely heavily on the tuning of manual algorithm parameters. The classification results also have the problems of low accuracy and fragmented images.

[0005] Methods based on deep learning have gained wide attention in the study of remote sensing data classification. Compared with traditional machine learning methods, deep learning models are more suitable for processing high-dimensional, nonlinear and complex data. When processing hyperspectral data, traditional two-dimensional convolution focuses on the extraction of spatial features, but has low efficiency in utilizing spectral information in hyperspectral images. Three-dimensional convolution can make up for the shortcomings of two-dimensional convolution when processing hyperspectral data, and improve classification performance by capturing spatial and spectral features at the same time, but its design is not specifically optimized for information redundancy between bands, which makes the model unable to effectively extract effective features of each spectral channel, thereby limiting the performance of tree species classification tasks. Summary of the invention

[0006] In order to overcome the above-mentioned technical defects of low efficiency in utilizing spectral information in hyperspectral images and inability to effectively extract effective features of each spectral channel, which leads to limited tree species classification performance, a tree species classification method based on spectral deep extraction convolutional neural network SDA-CNN is provided in the embodiment of the present application, and its specific technical solution is as follows:

[0007] S1. Construction of deep learning sample set:

[0008] Firstly, the PCA method is used to reduce the dimension of the hyperspectral forest image data. Secondly, the reduced-dimensional image data is annotated based on the image-tree information mapping results. Finally, the annotated image data is divided into training set, validation set and test set in a ratio of 1:1:8 to complete the construction of the deep learning sample set.

[0009] S2. SDA-CNN model training:

[0010] The deep learning sample set constructed using hyperspectral forest image data was used to train the spectral depth extraction convolutional neural network SDA-CNN model. The SDA-CNN model uses a hybrid neural network module including two-dimensional convolution and depth-by-depth convolution, and a new domain pixel perception module is added.

[0011] The two-dimensional convolution is used to extract the global spatial information of the image to generate global features; the depth-by-depth convolution is used to perform convolution operations independently on each spectral channel without sharing weights between the spectral channels to extract the correlation information between the spectral channels; the domain pixel perception module is used to extract the environmental information around the target pixel contained in the neighboring pixels, and then capture the detail information of the local pixels to improve the model's ability to extract local detail features.

[0012] S3. Tree species classification based on SDA-CNN model:

[0013] The collected forest image data is input into the trained SDA-CNN model to classify the tree species in the forest image data and obtain the tree species classification results.

[0014] Optionally, in a possible implementation of the above tree species classification method,

[0015] The hybrid neural network module includes a two-dimensional convolutional layer and a depth-wise convolutional layer, and the two convolutional layers are alternately connected to each other.

[0016] Optionally, in a possible implementation of the above tree species classification method,

[0017] The domain pixel perception module includes a point-by-point convolution layer. In the point-by-point convolution layer, point-by-point convolution is used to perform convolution operations on all input channels through a 1×1 convolution kernel to fuse the features of different channels.

[0018] Optionally, in a possible implementation of the above tree species classification method,

[0019] The domain pixel perception module also includes a neighboring pixel perception activation layer. In the neighboring pixel perception activation layer of the feature input after the point-by-point convolution operation is fused, the input features are processed using the neighborhood pixel perception activation function to enhance the extraction of local spatial information in the hyperspectral forest image data.

[0020] Optionally, in a possible implementation of the above tree species classification method,

[0021] The domain pixel perception module also includes an SE layer for performing Squeeze and Excitation operations. The Squeeze operation generates channel descriptors by aggregating feature maps across spatial dimensions. The Excitation operation weights the channel descriptors obtained by Squeeze to emphasize important channels.

[0022] Optionally, in a possible implementation of the above tree species classification method,

[0023] The SDA-CNN model also includes a fully connected module consisting of two linear layers, which is used to map the high-dimensional feature space to the probability space of classification labels for classification and output the classification results.

[0024] Optionally, in a possible implementation of the above tree species classification method,

[0025] The hyperspectral forest image data is reduced to 30 bands, with one band corresponding to one data dimension.

[0026] Optionally, in a possible implementation of the above tree species classification method,

[0027] The specific steps of data dimensionality reduction include:

[0028] S1.1, Data standardization: The original image data is mean-centered and standard-deviation normalized to make different bands have the same scale;

[0029] S1.2, Covariance calculation: Calculate the covariance matrix of the standardized image data to measure the correlation between different bands;

[0030] S1.3, Eigenvalue decomposition: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvectors and eigenvalues;

[0031] S1.4, principal component selection: select the first k eigenvectors with the largest eigenvalues ​​as principal components, where k is the desired dimension after dimensionality reduction;

[0032] S1.5. Feature space transformation: Multiply the original data with the selected principal components to obtain a representation in the new feature space.

[0033] The present application also provides a tree species classification system in an embodiment, which is applicable to a spectral depth extraction convolutional neural network SDA-CNN, including:

[0034] A processor and a memory, with a communication connection established between the two;

[0035] The memory is used to store computer instructions, and also stores hyperspectral forest image data and its corresponding deep learning sample set;

[0036] When the computer instruction is called, the processor executes the technical solution described in any one of the tree species classification methods and implementation methods thereof.

[0037] The embodiment of the present application adopts the above-mentioned technical solution to achieve the following technical effects: the spectral depth extraction convolutional neural network SDA-CNN model in the present application is a hybrid convolutional neural network architecture that can capture the correlation between channels and local detail features. A hybrid neural network module that can capture the correlation between channels and local detail features is constructed by using two-dimensional convolution and depth-by-depth convolution. In view of the rich spectral and spatial information contained in airborne hyperspectral images, a strategy of alternating two-dimensional convolution and depth-by-depth convolution is adopted: two-dimensional convolution is used to capture global features, while depth-by-depth convolution performs independent convolution operations on each channel to capture the correlation information between channels. In addition, a neighboring pixel perception module is also introduced in the above-mentioned SDA-CNN model. By considering the information of neighboring pixels, the understanding and perception of the spatial information and local detail features of the input data are enhanced. Ultimately, the low utilization efficiency of the spectral information in the hyperspectral image and the inability to effectively extract the effective features of each spectral channel, which leads to the defect of limited tree species classification performance, improve the tree species classification performance and classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0039] Figure 1 A flowchart of a tree species classification method based on spectral depth extraction convolutional neural network SDA-CNN in this application;

[0040] Figure 2 A schematic diagram of the network structure of the SDA-CNN model in this application;

[0041] Figure 3 A structural diagram of a hybrid neural network module in the SDA-CNN model in this application;

[0042] Figure 4 A schematic diagram of the structure of the domain pixel perception module in the SDA-CNN model in this application;

[0043] Figure 5 This is the Gaofeng Forest Farm-A forest image data;

[0044] Figure 6 Hyperspectral forest image data of the southern Sierra Nevada Mountains in California, USA;

[0045] Figure 7 This is the classification result of Gaofeng Forest Farm-A hyperspectral forest dataset;

[0046] Figure 8 This is the classification result of the hyperspectral forest dataset in the southern Sierra Nevada Mountains in California, USA;

[0047] Fig. 9 A structural diagram of the tree species classification system in this application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0050] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0051] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0052] First, the following terms are explained:

[0053] Principal Component Analysis (PCA): Principal Component Analysis (PCA): is a commonly used dimensionality reduction technique, widely used in data preprocessing and pattern recognition, and is particularly suitable for the analysis of high-dimensional data, such as remote sensing images, genomic data, etc. In the field of remote sensing and image processing, the main purpose of PCA is to retain as much effective information as possible in the original data by reducing the dimension of the data. The core idea of ​​PCA is to project the original high-dimensional data onto a set of new, mutually orthogonal coordinate axes through linear transformation. These new coordinate axes are called principal components. The principal components are the eigenvectors of the data covariance matrix, sorted by the size of the variance they explain. The first principal component captures the maximum variance in the data, the second principal component captures the largest of the remaining variance, and so on.

[0054] 2D Convolution: It is one of the core operations in Convolutional Neural Network (CNN) and is widely used in image processing and pattern recognition. 2D convolution extracts local features such as edges and textures by sliding the convolution kernel (filter) on a 2D image. Define a small matrix, such as a 3x3 or 5x5 weight matrix, which represents the features we want to extract in the image. The size of the convolution kernel is usually much smaller than the image. The convolution kernel starts from the upper left corner of the image and slides step by step according to the defined stride. Each time, the dot product of the area covered by the convolution kernel is calculated and the result is stored in a new matrix, which is called a feature map or convolution output. Let the input image be I and the convolution kernel be K. The operation of 2D convolution can be expressed as:

[0055] Among them, I(i,j) represents the pixel value of the input image at position (i,j), and K(m,n) represents the weight value of the convolution kernel.

[0056] Depth-wise convolution: It is a type of convolution in deep learning and is a special convolution operation. The difference between depth-wise convolution and standard convolution is that the convolution kernel of two-dimensional convolution is a single-channel mode, and each channel of the input needs to be convolved, so that an output feature map with the same number of channels as the input feature map will be obtained. Depth-wise convolution performs convolution operations independently on each spectral channel without sharing weights between channels. Specifically, depth-wise convolution uses multiple independent convolution kernels, each of which acts only on one spectral channel. This method ensures the independence of information between channels, thereby better retaining and extracting the unique information of each channel. This feature enables depth-wise convolution to extract high-level features in each spectral channel. Since there is no mixing of information between channels, depth-wise convolution can refine the feature processing of each channel, helping the model to better capture the deep-level correlations between spectral channels.

[0057] Neighborhood Pixel Aware Activation Module (NPPM): The neighborhood pixel-aware module first uses a point-by-point convolution module in a serial connection to efficiently adjust the number of channels of the input feature map while keeping the spatial dimension unchanged, and then uses the neighborhood pixel-aware activation function to enhance the extraction of local spatial information of the hyperspectral image. Finally, SELayer adaptively emphasizes important features and suppresses unimportant features, so that the model pays more attention to the most useful features for tree species classification and improves the accuracy of tree species classification tasks.

[0058] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below. The technical solutions in the present application are described in detail in conjunction with the accompanying drawings and their specific embodiments, as follows:

[0059] Embodiment 1

[0060] like Figure 1 As shown, the tree species classification method based on spectral depth extraction convolutional neural network SDA-CNN in this application includes the following steps:

[0061] S1. Construction of deep learning sample set:

[0062] Firstly, the PCA method is used to reduce the dimension of hyperspectral forest image data. Secondly, the reduced image data are annotated based on the image-tree information mapping results. Finally, the annotated image data are divided into training set, validation set and test set in a ratio of 1:1:8 to complete the construction of deep learning sample set.

[0063] Optionally, in one implementation of the present application, data dimensionality reduction is to reduce the hyperspectral forest image data to 30 bands, where one band corresponds to one data dimension. PCA is a statistical technique that retains the variance of the original data to the greatest extent by projecting high-dimensional data into a low-dimensional space. Specifically, the first 30 principal components are retained in this solution, and these principal components together contain 98% of the spectral information in the original data.

[0064] Further optionally, the specific steps of data dimension reduction for the hyperspectral forest image data include: data standardization, covariance calculation, eigenvalue decomposition, principal component selection and feature space transformation. The specific technical solution is described in the second embodiment below.

[0065] S2. SDA-CNN model training:

[0066] The deep learning sample set constructed using hyperspectral forest image data was used to train the spectral depth extraction convolutional neural network SDA-CNN model. The SDA-CNN model adopted a hybrid neural network module including two-dimensional convolution and depth-by-depth convolution, and a new domain pixel perception module was added.

[0067] Specifically, two-dimensional convolution is used to extract the global spatial information of the image to generate global features; depth-wise convolution is used to perform convolution operations independently on each spectral channel without sharing weights between spectral channels to extract correlation information between spectral channels; the domain pixel perception module is used to extract the environmental information around the target pixel contained in the neighboring pixels, and then capture the detail information of local pixels to improve the model's ability to extract local detail features.

[0068] Further optionally, the domain pixel perception module includes a point-by-point convolution layer, a neighboring pixel perception activation layer and a SE layer.

[0069] In the point-by-point convolution layer, point-by-point convolution is used to perform convolution operations on all input channels through a 1×1 convolution kernel to fuse the features of different channels.

[0070] In the neighboring pixel-aware activation layer of the feature input after the point-by-point convolution operation is fused, the input features are processed using the neighborhood pixel-aware activation function to enhance the extraction of local spatial information in the hyperspectral forest image data, so as to realize the function of extracting the environmental information around the target pixel contained in the neighboring pixels.

[0071] The SE layer is used to perform the Squeeze and Excitation operations. The Squeeze operation generates channel descriptors by aggregating feature maps across spatial dimensions (H×W). The Excitation operation weights the channel descriptors obtained by Squeeze to emphasize important channels. The SE layer can adaptively emphasize important features and suppress unimportant features, so that the model pays more attention to the most useful features for tree species classification and improves the accuracy of tree species classification tasks.

[0072] Further optionally, the SDA-CNN model also includes a fully connected module consisting of two linear layers, which is used to map the high-dimensional feature space to the probability space of the classification label for classification and output the classification result.

[0073] S3. Tree species classification based on SDA-CNN model:

[0074] The collected forest image data is input into the trained SDA-CNN model to classify the tree species in the forest image data and obtain the tree species classification results.

[0075] In the technical solution of the tree species classification method based on the spectral depth extraction convolutional neural network SDA-CNN of the present application, a hybrid neural network module that can capture the correlation between channels and local detail features is constructed by two-dimensional convolution and depth-by-depth convolution. In view of the rich spectral and spatial information contained in airborne hyperspectral images, a strategy of alternating two-dimensional convolution and depth-by-depth convolution is adopted: two-dimensional convolution is used to capture global features, while depth-by-depth convolution performs independent convolution operations on each channel to capture the correlation information between channels. In addition, a neighboring pixel perception module is also introduced in the above-mentioned SDA-CNN model. By considering the information of neighboring pixels, the understanding and perception of the spatial information and local detail features of the input data are enhanced.

[0076] Next, the important technical solutions involved in the first embodiment of the present application are further described as follows:

[0077] Embodiment 2

[0078] Hyperspectral image data has great potential in many applications due to its rich spectral information, but it is also accompanied by high dimensionality and data redundancy problems. This not only increases the computational complexity, but also places high demands on storage. Undoubtedly, the hyperspectral forest image data in this application also belongs to a type of hyperspectral image data. In order to solve the above-mentioned problems of high dimensionality and data redundancy, and at the same time ensure that the main spectral information is extracted, the PCA method is used in this application to reduce the dimension of the hyperspectral forest image to 30 bands. One band corresponds to one data dimension, that is, the first 30 principal components in the hyperspectral forest image data are retained. These principal components together contain a large amount of spectral information in the original data, which can account for up to 98%.

[0079] Specifically, the specific steps for data dimensionality reduction of hyperspectral forest image data include:

[0080] S1.1. Data standardization: The original image data is mean-centered and standard-deviation normalized so that different bands have the same scale.

[0081] Specifically, data standardization uses the following calculation formula: In the formula, X i is the image of the i-th band, μ i is the mean of the ith band, σ i is the standard deviation of the i-th band, and the value of i is [1, 30].

[0082] S1.2. Covariance calculation: Calculate the covariance matrix of the standardized image data to measure the correlation between different bands.

[0083] Specifically, the calculation formula of covariance is: In the formula, C is the covariance matrix, N is the number of bands, and X is i is the normalized image of the ith band, and μ is the mean vector.

[0084] S1.3. Eigenvalue decomposition: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvectors and eigenvalues.

[0085] The eigenvector represents the main direction of the original data, and the eigenvalue represents the importance of the data in these directions. The decomposition expression is: C = UΛU T , where U is the eigenvector matrix and Λ is the diagonal eigenvalue matrix.

[0086] S1.4, principal component selection: select the first k eigenvectors with the largest eigenvalues ​​as principal components, where k is the desired dimension after dimensionality reduction.

[0087] Specifically, the principal components are arranged in descending order according to the eigenvalues, where W is a matrix composed of eigenvectors corresponding to the first k largest eigenvalues, and its expression is: W = U (:, 1: k).

[0088] S1.5. Feature space transformation: Multiply the original data with the selected principal components to obtain a representation in the new feature space.

[0089] Specifically, the expression in the new feature space is: F=XW, where F is the new feature space image and X is the original standardized image data.

[0090] By using the above scheme to reduce the dimension of hyperspectral forest image data, the problems of high dimensionality and data redundancy in hyperspectral image data can be effectively solved, while ensuring that the first 30 principal components in the hyperspectral forest image data are extracted to retain a large amount of spectral information in the original data, even making it account for up to 98%.

[0091] Embodiment 3

[0092] Deep learning technology can effectively process the complex spectral and spatial information in hyperspectral images, automatically extract features, and achieve high-precision performance in tree species classification. This method greatly improves the accuracy and efficiency of tree species classification based on traditional methods.

[0093] Deep learning outperforms traditional classification methods in hyperspectral tree species classification, mainly due to its powerful feature learning ability. Typical deep learning network architectures include convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), etc. The appropriate framework is selected according to the task requirements. Research on hyperspectral image tree species classification based on deep learning network models is constantly underway, and the scope of model application is constantly expanding.

[0094] This application improves the deep learning network architecture to overcome the shortcomings of traditional convolutional neural networks that are mainly designed for natural images, focus on the extraction of spatial features, and have low efficiency in utilizing spectral information when processing hyperspectral images.

[0095] Next, combine with Figure 2 The network structure of the improved SDA-CNN model in this application is described as follows:

[0096] like Figure 2As shown, the SDA-CNN model in the embodiment of the present application includes: a hybrid neural network module, a domain pixel perception module and a fully connected module. The hybrid neural network module described in the above embodiment one includes a two-dimensional convolution layer and a depth-by-depth convolution layer, and the two convolution layers are alternately connected to each other; the domain pixel perception module includes a point-by-point convolution layer, a neighboring pixel perception activation layer and an SE layer; the fully connected module consists of two linear layers.

[0097] The network structure of the hybrid neural network module is as follows Figure 3 As shown:

[0098] Two-dimensional convolution is an operation commonly used in image processing in deep learning, and can be used to extract spatial features in hyperspectral images. Hyperspectral images contain not only spectral dimensions (spectral information of each pixel), but also spatial dimensions (spatial relationships between pixels). Two-dimensional convolution is often used to process spatial information. Here is how two-dimensional convolution extracts features from hyperspectral data:

[0099] A hyperspectral image can be viewed as a three-dimensional data cube, where two dimensions are spatial dimensions (horizontal and vertical coordinates) and the third dimension is the spectral dimension (i.e., the spectral information of each pixel). For a two-dimensional convolution operation, the convolution kernel (filter) K∈R K×K×D Applied to the spatial dimension of the entire image. The convolution kernel slides over the image to extract local spatial features.

[0100] The operation of two-dimensional convolution can be described as:

[0101]

[0102] Among them, X is the input image, K is the convolution kernel, and Y is the output feature map

[0103] Although this convolution operation helps capture the global spatial information of the image, it has limitations in extracting independent information between spectral channels. To address this problem, depth-wise convolution is introduced.

[0104] Unlike 2D convolution, depthwise convolution performs convolution operations independently on each spectral channel without sharing weights across channels.

[0105] Specifically, depth-wise convolution uses multiple independent convolution kernels, each of which acts on only one spectral channel. K×K×D , depth-wise convolution applies an independent convolution kernel to each band to generate the spatial feature map of each band. The convolution formula is:

[0106]

[0107] Where, X dis the input data of the dth band, K d is the convolution kernel of the dth band, Y d is the output feature map of the dth band

[0108] Depth-wise convolution specializes in processing the spectral dimension of hyperspectral images. The key idea of ​​depth-wise convolution is to use a convolution kernel to perform independent convolution operations on each band without mixing information across bands. This approach ensures the independence of information between channels, thereby better preserving and extracting the unique information of each channel. This feature enables depth-wise convolution to extract high-level features in each spectral channel. Since there is no mixing of information between channels, depth-wise convolution can refine the feature processing of each channel, helping the model better capture the deep correlation between spectral channels.

[0109] like Figure 4 As shown in the figure, the domain pixel perception module includes a point-by-point convolution layer, a neighboring pixel perception activation layer and a SE layer.

[0110] Neighboring pixels contain the environmental information around the target pixel. This contextual information helps the network extract local detail features. First, the point convolution module is used to efficiently adjust the number of channels of the input feature map while keeping the spatial dimension unchanged. Point-by-point convolution performs convolution operations on all input channels through a 1x1 convolution kernel, thereby fusing the features of different channels. For hyperspectral images, each channel usually represents different spectral band information, and point-by-point convolution can integrate the feature information of these bands together.

[0111] The specific operation of point-by-point convolution can be expressed as:

[0112]

[0113] X d (i, j) represents the value of the dth channel of the input feature map at position (i, j). K d It is a 1×1×D convolution kernel, which is used to sum the values ​​of all input channels in a weighted manner to generate the value Y(i, j) of the output channel.

[0114] The neighborhood pixel-aware activation function used in this embodiment uses a square window centered on x' to define the neighborhood range of x', and then marks all pixels n in the window as x1, x2, xn in sequence, where n is the number of pixels. The NPAF of input x' is denoted as A(x'), and its calculation process can be summarized as follows:

[0115] y i =f(x i +b c );In the above two formulas, w irepresents the weight of each activated pixel, f is the basic ReLU activation function, b c As channel-shared bias.

[0116] The neighborhood pixel-aware activation function is based on the interaction with neighboring pixels. If the neighboring pixels have several positive values ​​and larger weights, the value of the pixel can be "recovered" from 0, which produces a cluster correction effect. By introducing a neighborhood pixel-aware activation layer after the convolution operation, the module can better capture the detail information of local pixels, thereby improving the model's extraction of local detail features.

[0117] Finally, the SE layer (i.e., SELayer) adaptively emphasizes important features and suppresses unimportant features, so that the model pays more attention to the features that are most useful for tree species classification and improves the accuracy of tree species classification tasks.

[0118] The two main operations of the SE module are Squeeze and Excitation. The corresponding formulas are as follows: s = F ex (z, W) = σ(g(z, W)) = σ(W2δ(W1z)), where z represents the input feature vector or feature map, W1 and W2 represent the weight matrices of the fully connected layer, δ represents the ReLU activation function, and σ represents the sigmoid activation function.

[0119] The first is the Squeeze operation, which produces channel descriptors by aggregating feature maps across spatial dimensions (H×W).

[0120] Assume that there is an input feature map with a dimension of H×W×C. For each channel, a global average pooling operation is performed. Specifically, for the i-th channel, the average value of all spatial positions on the channel is calculated. This can be achieved by averaging all elements of each channel, where each element represents the average value on the corresponding channel. When the above operation is performed on all channels in the input feature map, C average values ​​will be obtained, forming a vector of length C. Thus, the Squeeze operation compresses the input H×W×C feature map into a C-dimensional vector, where each element represents the average value on the corresponding channel. This vector can be regarded as a "descriptor" of the entire feature map in the channel dimension.

[0121] Next is the Excitation operation, which is the second step in the SE module. Its main purpose is to weight the channel descriptors obtained by the previous Squeeze step to emphasize important channels.

[0122] The C-dimensional vector obtained by Squeeze is further input into a fully connected layer (FC layer), the purpose of which is to learn channel weights. This fully connected layer includes a nonlinear activation function, such as ReLU, to introduce nonlinear transformations. Through learning, the channel weights obtained by the fully connected layer pass through a Sigmoid activation function to limit their range between 0 and 1. The learned weights can be regarded as the degree of activation of each channel. Finally, the original C-dimensional vector is multiplied by the learned channel weights to obtain a weighted vector. This weighted vector reflects the importance and contribution of each channel in the task. This process enables the network to dynamically adjust the attention of the channel, thereby more effectively utilizing information to complete the task.

[0123] The domain pixel perception module first uses the point convolution module in a serial connection to efficiently adjust the number of channels of the input feature map while keeping the spatial dimension unchanged, and then uses the neighborhood pixel perception activation function to enhance the extraction of local spatial information of the hyperspectral image. Finally, SELayer adaptively emphasizes important features and suppresses unimportant features, so that the model pays more attention to the most useful features for tree species classification and improves the accuracy of tree species classification tasks.

[0124] Finally, for ease of understanding, the technical solution in this application is described below with reference to specific examples, as follows:

[0125] For example, in some specific implementations of deep learning sample construction, the Gaofeng Forest Farm-A hyperspectral forest dataset used has 9 types of forest vegetation, and 3 types of non-forest vegetation, such as building land, roads, and logging sites. The size of this hyperspectral open set is 906*572 pixels, with 125 spectral bands. In order to ensure the data independence between the test set and the training set, so as to accurately evaluate the model performance, this study randomly divides the samples into training set, validation set, and test set in a ratio of 1:1:8. The number of samples in each set is shown in Table 1.

[0126] Table 1

[0127]

[0128]

[0129] Furthermore, in a specific SDA-CNN model, 4 convolutional layers, 3 depth-wise convolutional layers, 1 domain pixel perception module, 1 flattening layer, and 2 linear connection layers are set. It is easy to know that the flattening layer is the above-mentioned SE layer, and the 2 linear connection layers constitute the fully connected module mentioned above.

[0130] The number of convolution kernels is determined by referring to famous CNN models such as ImageNet, and the number of kernels in the next layer is twice that of the previous layer. The algorithm flow and parameters are shown in Table 2.

[0131] Table 2

[0132]

[0133]

[0134] The model enhances the ability to extract inter-channel correlation features by combining depth-wise convolution and two-dimensional convolution. When processing hyperspectral data, traditional two-dimensional convolution usually mixes the information of all channels, which may lead to the loss or confusion of spectral features. To solve this problem, this study introduced the depth-wise convolution module in MobileNet. This module generates feature maps by using independent convolution kernels on each input channel, thereby retaining the independent characteristics of the channels. The model better captures the deep correlation between channels at different levels by alternating between depth-wise convolution and general convolution.

[0135] Subsequently, a two-dimensional convolution with a kernel number of 256 is used to combine the extracted spectral and spatial features for input into the neighborhood pixel perception module to enhance the extraction of spatial information of the hyperspectral image. In further processing, a flattening layer is used to flatten the downsampled feature map into a one-dimensional vector.

[0136] The ReLU activation function is applied to all convolutional and dense layers to introduce nonlinear transformations and help extract complex and rich features, where:

[0137] f(x)=max(0,x);

[0138] From the above formula, we can see that if the input is greater than 0, the output is the original value; if the input is 0 or a negative number, the output is 0. For values ​​greater than 0, the function is linear, so when back propagation is used to train a neural network, it has the characteristics of a linear activation function. For values ​​less than or equal to 0, the function is nonlinear, also known as a piecewise linear function. From the perspective of the function model, the ReLU function is simpler to calculate and has a lower computational cost. It can also accelerate learning and simplify the model, avoiding problems such as gradient disappearance.

[0139] Finally, multi-class classification is performed through two linear layers. The linear layer (LinearLayer), also called the fully connected layer (FullyConnectedLayer) or fully connected module, plays a key role in classification tasks, especially in the final stage of the neural network. It works by mapping the high-dimensional feature space to the probability space of the classification label. In the classification task, the features extracted by the neighborhood pixel perception module are flattened into a feature vector of 30976. This feature vector contains the important features of the input data (such as an image or spectrum). Pass it to the linear layer. The operation performed by the linear layer is a simple linear transformation. The output of the given input vector linear layer is calculated by the following formula:

[0140] y=Wx+b; where x is the input feature vector, W is the weight matrix, size m×n, where m is the number of output categories and n is the number of dimensions of the input features. b is the bias term, which is used to enhance the model's fitting ability. y is the output of the linear layer, also known as logits.

[0141] The output y of the linear layer does not directly give the classification result, but a raw score (logits). In order to convert these scores into probability distributions, an activation function is usually used.

[0142] For multi-classification problems, the commonly used activation function is the Softmax function:

[0143]

[0144] The Softmax function converts the output of the linear layer into a probability distribution, which represents the probability value of each category. These probability values ​​add up to 1.

[0145] Once the probability of each category is obtained through the Softmax function, the model can be classified according to these probabilities. Usually, the category corresponding to the maximum probability is the prediction result, that is:

[0146] To train the model, we usually use the Cross-Entropy Loss function. It measures the difference between the probability distribution of the model output and the probability distribution of the true label.

[0147] The formula for cross entropy loss is: In the formula, y i is the one-hot encoding of the true category is the probability value predicted by the model

[0148] Through backpropagation, the error is propagated from the loss function back to the linear layer, and the weight matrix W and bias b in the linear layer are adjusted through gradient descent methods (such as SGD, Adam, etc.), thereby improving the classification accuracy.

[0149] The features are finally classified into 12 categories through a linear layer.

[0150] After determining the network structure, the model training parameters are configured and optimized. According to the best parameters obtained from training, the batch size is set to 64, the epoch is set to 300, the window size is 11, the learning rate is 0.0001, and SGD is used as the optimizer.

[0151] The tree species classification experiment of hyperspectral data using deep learning networks was programmed using Python (3.8), running on Tesla T4 hardware with Cuda version 11.8. All deep learning models were based on the open source Pytorch (2.0) deep learning framework.

[0152] In order to comprehensively evaluate the classification accuracy, the overall accuracy (OA), average accuracy (AA) and Kappa coefficient were used as model evaluation indicators.

[0153] Overall Accuracy (OA): OA represents the ratio of correctly classified pixels to the total number of pixels in all classes.

[0154] Average Precision (AA): AA means the average precision of each class.

[0155] Kappa coefficient, as a quantitative evaluation indicator of the model:

[0156] The Kappa coefficient measures the agreement between the predicted classification and the ground truth classification, taking into account the agreement that could occur by chance.

[0157] In addition, precision, recall and F1 score were used as evaluation indicators to evaluate each tree species:

[0158]

[0159] The higher the precision, recall, OA, and F1 score, the closer the predicted value is to the true value.

[0160] The macro-average evaluation model is used to evaluate the average performance of each category, and the weighted average reflects the overall performance of the model. The classification accuracy of each tree species using this method is shown in Table 3

[0161] Table 3

[0162] species Accuracy Recall F1 score quantity Chinese fir 0.99 0.99 0.99 64594 Masson pine 1.00 0.99 1.00 4272 slash pine 0.99 0.99 0.99 3340 Eucalyptus macrophylla 1.00 1.00 1.00 11945 Eucalyptus urophylla 1.00 1.00 1.00 32220 Red thorn 0.97 0.98 0.98 20369 Rice and Pork 1.00 1.00 1.00 2400 Camellia oleifera 1.00 1.00 1.00 6659 Other broadleaf trees 1.00 0.98 0.99 8593 the way 0.99 1.00 1.00 8378 Logging site 1.00 1.00 1.00 8136 Building land 0.96 0.98 0.97 54 Accuracy 0.99 170960 Macro average 0.99 0.99 0.99 170960 Weighted average 0.99 0.99 0.99 170960

[0163] Based on the above, this patent realizes automatic classification of tree species in drone hyperspectral images based on the improved SDA-CNN network model architecture, achieving high-precision and highly automated vegetation classification.

[0164] In order to display the classification results more clearly, the following actual research case illustrates the classification effect of the technical solution of this application. Figure 5 and Figure 6 is the hyperspectral image of the study area, where Figure 5 This is the Gaofeng Forest Farm-A hyperspectral forest image data; Figure 6 is the hyperspectral forest image data of the southern Sierra Nevada Mountains in California, USA. Figure 5 and Figure 6 The classification result diagram obtained by classifying tree species based on the forest data in . Figure 7 The picture shows the classification result of Gaofeng Forest Farm-A hyperspectral forest dataset. Figure 8 Shown is the classification result of the hyperspectral forest dataset in the southern Sierra Nevada Mountains in California, USA.

[0165] like Figure 7 As shown in the figure, on the Gaofenglinchang-A hyperspectral forest dataset, it can be seen that the classification boundary is extracted clearly and the fine patches are reduced, which proves the effectiveness of the proposed method in the open set classification of hyperspectral images.

[0166] SDA-CNN model adaptability:

[0167] In order to further test the stability of the SDA-CNN network structure and determine the applicability of the structure, the following Figure 6 The hyperspectral forest dataset of the southern Sierra Nevada Mountains in California, USA is shown. The data is substituted into the SDA-CNN model for tree species classification.

[0168] In this dataset, the hyperspectral data contains a wavelength range of 280 to 2510 nm, 426 bands, a sampling interval of 5 nm, and a spatial resolution of 1 m. The hyperparameters are set as follows: batch size is 64, learning rate is 0.0001, number of iterations is 300, and window size is 13. With a training set to validation set ratio of 6:4, the overall accuracy is 99.71% and the average accuracy is 98.19%. The prediction results of the model are shown in Figure 2. Figure 8 As shown, this study demonstrates the universality and potential of the SDA-CNN model, indicating that it can be applied to multiple fields and tasks rather than being limited to specific applications.

[0169] Embodiment 4

[0170] Fig. 9The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the tree species classification method according to an embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig. 9 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:

[0171] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the tree species classification method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0172] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0173] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0174] It should be pointed out that Fig. 9 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0175] In this embodiment, the tree species classification method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0176] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0177] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A tree species classification method based on spectral depth extraction convolutional neural network SDA-CNN, characterized in that: include: S1. Construction of deep learning sample set: Firstly, the PCA method is used to reduce the dimension of the hyperspectral forest image data. Secondly, the reduced-dimensional image data is annotated based on the image-tree information mapping results. Finally, the annotated image data is divided into training set, validation set and test set in a ratio of 1:1:8 to complete the construction of the deep learning sample set. S2. SDA-CNN model training: The deep learning sample set constructed using hyperspectral forest image data was used to train the spectral depth extraction convolutional neural network SDA-CNN model. The SDA-CNN model uses a hybrid neural network module including two-dimensional convolution and depth-by-depth convolution, and a new domain pixel perception module is added. The two-dimensional convolution is used to extract the global spatial information of the image to generate global features; the depth-by-depth convolution is used to perform convolution operations independently on each spectral channel without sharing weights between the spectral channels to extract the correlation information between the spectral channels; the domain pixel perception module is used to extract the environmental information around the target pixel contained in the neighboring pixels, and then capture the detail information of the local pixels to improve the model's ability to extract local detail features; S3. Tree species classification based on SDA-CNN model: The collected forest image data is input into the trained SDA-CNN model to classify the tree species in the forest image data and obtain the tree species classification results.

2. The tree species classification method according to claim 1, characterized in that: The hybrid neural network module includes a two-dimensional convolutional layer and a depth-wise convolutional layer, and the two convolutional layers are alternately connected to each other.

3. The tree species classification method according to claim 1 or 2, characterized in that: The domain pixel perception module includes a point-by-point convolution layer. In the point-by-point convolution layer, point-by-point convolution is used to perform convolution operations on all input channels through a 1×1 convolution kernel to fuse the features of different channels.

4. The tree species classification method according to claim 3, characterized in that: The domain pixel perception module also includes a neighboring pixel perception activation layer. In the neighboring pixel perception activation layer of the feature input after the point-by-point convolution operation is fused, the input features are processed using the neighborhood pixel perception activation function to enhance the extraction of local spatial information in the hyperspectral forest image data.

5. The tree species classification method according to claim 4, characterized in that: The domain pixel perception module also includes an SE layer for performing Squeeze and Excitation operations. The Squeeze operation generates channel descriptors by aggregating feature maps across spatial dimensions. The Excitation operation weights the channel descriptors obtained by Squeeze to emphasize important channels.

6. The tree species classification method according to claim 1, characterized in that: The SDA-CNN model also includes a fully connected module consisting of two linear layers, which is used to map the high-dimensional feature space to the probability space of classification labels for classification and output the classification results.

7. The tree species classification method according to claim 1, characterized in that: The hyperspectral forest image data is reduced to 30 bands, with one band corresponding to one data dimension.

8. The tree species classification method according to claim 7, characterized in that: The specific steps of data dimensionality reduction include: S1.1, Data standardization: The original image data is mean-centered and standard-deviation normalized to make different bands have the same scale; S1.2, Covariance calculation: Calculate the covariance matrix of the standardized image data to measure the correlation between different bands; S1.3, Eigenvalue decomposition: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvectors and eigenvalues; S1.4, principal component selection: select the first k eigenvectors with the largest eigenvalues ​​as principal components, where k is the desired dimension after dimensionality reduction; S1.

5. Feature space transformation: Multiply the original data with the selected principal components to obtain a representation in the new feature space.

9. A tree species classification system, characterized in that: The system is suitable for use in a spectral depth extraction convolutional neural network SDA-CNN, and the system includes: A processor and a memory, with a communication connection established between the two; The memory is used to store computer instructions, and also stores hyperspectral forest image data and its corresponding deep learning sample set; When the computer instructions are called, the processor is caused to execute the tree species classification method described in any one of claims 1-8.

Images