Tree species recognition, classification and counting method based on PLS feature fusion convolutional neural network

Abstract

The application discloses a tree species identification, classification and counting method based on a PLS feature fusion convolutional neural network, and relates to the technical field of tree species identification, classification and counting, and can be used for tree crown identification of a hyperspectral image of a tree, and can divide the image into a plurality of single-tree crown images; a partial least squares (PLS) feature fusion CaffeNet convolutional neural network is established, feature extraction is performed on the output of each pooling layer and the final output of the CaffeNet convolutional neural network, deep layer features and shallow layer features are fused by using a PLS method, and the feature fusion results of spatial information and the feature fusion of spectral information are fused and classified; the segmented single-tree crown images are taken as input, a trained PLS feature fusion CaffeNet convolutional neural network is used for tree species identification, and the identified trees are counted according to species. The application combines shallow layer features and deep layer features, and performs parallel feature fusion by using a partial least squares method, so that accurate classification and counting of tree species are realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of target recognition and computer vision, and relates to a tree species identification and counting method based on a partial least squares (PLS) feature fusion convolutional neural network. BACKGROUND

[0002] Trees play an important role in urban road greening and garden greening. Trees, as a necessary supporting facility for road functions, have important roles in improving road service quality, improving regional ecological environment, eliminating noise, purifying air, regulating climate, and conserving water and preventing wind and sand. Traditional tree species identification and counting still largely relies on manpower, resulting in low efficiency, high cost, and unguaranteed correctness and completeness of statistics.

[0003] In recent years, hyperspectral sensors have been integrated into unmanned aerial vehicles (UAVs), which have great advantages in terms of operating cost and flexibility. The pictures of urban trees taken by UAVs provide a rich data source for tree species identification. Hyperspectral images have high spectral and spatial resolution, and can realize tree species level classification. The classification and mapping of tree species by hyperspectral images can be converted into a hyperspectral image classification task, which classifies each pixel vector. Many methods have been applied to the hyperspectral image classification task, such as support vector machines and random forests. However, these methods only use the spectral features of each pixel of the hyperspectral image for classification, ignoring the neighborhood of the pixel.

[0004] Convolutional neural networks (CNNs) are a class of feedforward neural networks that contain convolutional computations and have a deep structure, and are one of the representative algorithms of deep learning. There are related works that use convolutional neural networks to identify and classify tree species. In a convolutional neural network (CNN), the information at higher layers is more abstract and more task-specific, so people usually focus on the fully connected (FC) layer features and believe that the lower layer features have weaker discriminative ability, without considering the amount of information provided by the lower layers for image representation. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the deficiencies of the prior art and provide a tree identification and counting method based on a partial least squares (PLS) feature fusion convolutional neural network, which performs image segmentation on hyperspectral images provided by a UAV to obtain a single-tree crown image set, and simultaneously builds a PLS feature fusion convolutional neural network model to identify and count tree species.

[0006] To solve the above technical problems, the present application adopts the following technical solutions:

[0007] A tree species identification and classification counting method based on a PLS feature fusion convolutional neural network, comprising:

[0008] S100 performs tree crown recognition on the hyperspectral image of the tree and divides the image into a plurality of single-tree crown images;

[0009] S200 establishes a partial least squares (PLS) feature fusion CaffeNet convolutional neural network, which has the following structure:

[0010] On the basis of the CaffeNet convolutional neural network structure, a branch CNN composed of a 1×1 convolutional layer, a nonlinear activation layer and a global average pooling layer is inserted after each pooling layer to extract features.

[0011] The branch CNNs after the first and second pooling layers are connected to a first PLS fusion module; the branch CNN after the third pooling layer and the second fully connected layer are connected to a second PLS fusion module; and the first and second feature fusion modules are connected to a third PLS fusion module.

[0012] The spatial information and the spectral information of the hyperspectral image are respectively taken as inputs to obtain feature fusion results output by the third PLS fusion module; and the feature fusion results of the spatial information and the feature fusion results of the spectral information are input into a fourth PLS fusion module.

[0013] The fourth feature fusion module is sequentially connected to three fully connected layers and a classifier.

[0014] S300 collects hyperspectral images of trees and manually labels tree species to construct a tree image training set and a verification set, which are used to train the PLS feature fusion CaffeNet convolutional neural network established in S200.

[0015] S400 takes the segmented single-tree crown images as inputs, and uses the trained PLS feature fusion CaffeNet convolutional neural network to identify tree species and count the identified trees according to species.

[0016] As a preferred embodiment, the hyperspectral images of the trees are obtained from the air by using a drone carrying an imaging spectrometer.

[0017] As a preferred embodiment, the Faster R-CNN network is used to recognize the tree crowns.

[0018] As a preferred embodiment, after the Faster R-CNN network is used to recognize the tree crowns, each tree crown is labeled with a target bounding box, and the image is segmented by taking the midpoint between adjacent bounding boxes as a boundary.

[0019] As a preferred embodiment, the nonlinear activation layer uses a ReLU activation function.

[0020] As a preferred embodiment, the feature extraction process of the branch CNN is as follows:

[0021] The output of the pooling layer is input into a 1x1 convolutional layer, and the output is as follows:

[0022]

[0023] where b is the branch number, assuming there are B branches, L is the number of output channels, is the weight; is the feature of the kth input channel, and there are K (b) input channels;

[0024] The output of the 1x1 convolutional layer is input into a GAP layer after being suppressed by a nonlinear activation layer;

[0025] Let denote the (i,j)th element of Y l (b) The GAP layer output is as follows:

[0026]

[0027] Let:

[0028]

[0029] This L-dimensional vector is the output of the bth branch.

[0030] As a preferred embodiment, the PLS fusion process performed by the first PLS fusion module, the second PLS fusion module, the third PLS fusion module, and the fourth PLS fusion module includes:

[0031] The data matrix is given to the shallow features and the deep features respectively;

[0032] After decomposing the data matrix, the best feature pair is found;

[0033] A matrix composed of all feature projection vectors is established, the original feature vectors are projected into two subspaces respectively, the highest degree of covariant feature pair is added to the fusion vector, and a new feature, i.e., the PLS fusion result, is obtained.

[0034] As a preferred embodiment, the PLS fusion process is as follows:

[0035] Take and parallel fusion by PLS algorithm, where R p indicates the dimension of the shallow features, and R q indicates the dimension of the deep features;

[0036] The data matrix is given to g (i) and g (i+1) respectivelyp×n with where the central variable ψ T and is assumed to be zero mean, and is decomposed into S uv = ψψ T with respectively, where u∈g , v∈g (i) , and is decomposed into S (i+1) according to the following equations (4)-(7):

[0037]

[0038] ψ = TP T + E (5)

[0039]

[0040]

[0041] where p i and q i are the loading vectors, t i and u i are the projection weights, T and U are n x l matrices, which are the projections of ψ and respectively, P and Q are m x l and p x l orthogonal loading matrices respectively, and E and F are error terms, which are independent and identically distributed normal distributed random variables;

[0042] After the optimal feature pairs are found, the matrix composed of all the feature projection vectors is established, the original feature vectors are projected into two subspaces respectively, the highest degree of covariant feature pairs are added to the fusion vector, and the new features, i.e. the PLS fusion results, are obtained.

[0043] As a preferred embodiment, the deep features and the shallow features are fused in parallel.

[0044] As a preferred embodiment, the classifier is selected as a softmax classifier.

[0045] ​The model constructed by the method of this invention can simultaneously process the spectral and spatial information of hyperspectral images. By correlating the spectral and spatial features of hyperspectral images for classification, higher classification accuracy is achieved. Continuous tree canopy images are segmented into individual tree canopy image sets to facilitate tree species identification and counting. Then, shallow and deep features are combined using partial least squares for parallel feature fusion, fully utilizing the rich multi-layered information of shallow and deep features to improve the accuracy of tree species identification, ultimately achieving accurate tree species classification and counting. Compared to the traditional CaffeNet model's recognition accuracy (89.53%), the model constructed by the method of this invention achieves a recognition accuracy of 94.65%, an improvement of 5.12%. Attached Figure Description

[0046] Figure 1 This is a tree canopy image segmentation diagram for the present invention.

[0047] Figure 2 The flowchart in the figure shows the PLSFFCNN model, which is the PLS feature fusion convolutional neural network constructed in this invention.

[0048] Figure 3 This is a diagram of the PLS feature fusion convolutional neural network structure, where c1, c2, c3-5, and 1×1C refer to convolutional layers, p1, p2, and p3 refer to pooling layers, and FC, FC1, and FC2 refer to fully connected layers. Detailed Implementation

[0049] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0050] This invention provides a tree identification and counting method based on a partial least squares feature fusion convolutional neural network. First, hyperspectral images of the trees are acquired. Then, a Faster R-CNN network is used to segment and identify the tree crowns. The segmented individual tree crown images are then input into a constructed deep learning network to identify and count the tree species.

[0051] like Figure 2 The flowchart shown is a tree recognition and counting method based on a partial least squares feature fusion convolutional neural network provided by an embodiment of the present invention. The method includes the following steps:

[0052] Step 1: Use a drone equipped with an imaging spectrometer to fly in the air and photograph the trees to obtain hyperspectral images of the trees;

[0053] Step 2: The high-spectral image described in step 1 is subjected to Faster R-CNN network to identify the tree crown. Image segmentation is performed at the center of the distance between two bounding boxes;

[0054] As shown in Figure 1 , the Faster R-CNN network is used to identify and segment the tree crown, specifically as follows:

[0055] Step a, the single tree crown image after the Faster R-CNN network identification is labeled with a target bounding box. The position information of the target bounding box i is represented by the coordinates of the upper left corner and the lower right corner of the bounding box;

[0056] Step b, for the target bounding box i and the target bounding box i+1, image segmentation is performed at ;

[0057] Step c, generate the segmented single tree crown image set.

[0058] Step 3: Collect high-spectral images of trees, label the tree images through artificial investigation, and construct a tree image training set and a verification set;

[0059] Step 4: Build a CaffeNet convolutional neural network based on partial least squares feature fusion and train it;

[0060] As shown in Figure 3 , a CaffeNet convolutional neural network based on partial least squares feature fusion is built and trained. The built CaffeNet convolutional neural network integrates 1-D-CNN and 2-D-CNN, which can process spectral and spatial information at the same time, and each dimension network performs parallel fusion of shallow and deep features based on partial least squares. Specifically as follows:

[0061] Step a, the input image is represented as I∈R h×w×d , where h, w, d represent the height, width and number of light channels respectively, and the spectral information at pixel (m, n) is represented as The spatial information of each pixel can be represented as , where p mn is the center position of pixel A mn , e is the number of light channels after dimension reduction, and the method used in the dimension reduction process is principal component analysis;

[0062] Step b-1, for the input spatial information, we take the existing CaffeNet model as the basis, which includes convolution layer 1, pooling layer 1, convolution layer 2, pooling layer 2, convolution layer 3, convolution layer 4, convolution layer 5, pooling layer 3, full connection layer 1, full connection layer 2. After each pooling layer, a branch CNN is inserted, each branch CNN consists of three layers: 1x1 convolution, nonlinear activation ReLU and global average pooling GAP;

[0063] Step b-2, assuming that a certain pre-trained CNN (main CNN, namely CaffeNet model) emits B branch CNNs, and the input of the bth branch (b = 1, 2, … B) has K (b) channels (i.e. feature maps) Then the output of the 1x1 convolution of the bth branch is:

[0064]

[0065] where L is the number of output channels, are the weights of 1x1 convolution;

[0066] Step b-3, after each 1x1 convolution, there is a nonlinear activation layer such as ReLU, which adds more nonlinearity and expression ability to the data. Then there is a GAP layer, denoted by the (i,j)th element of Y l (b) , which gives:

[0067]

[0068] Let:

[0069]

[0070] This L-dimensional vector is the final output of the bth branch;

[0071] Step b-4, we take and to perform PLS algorithm parallel fusion, R p is the dimension of shallow features, R q is the dimension of deep features, for g (i) and g (i+1) , respectively, given data matrix ψ ∈ R p×n and We further assume that the central variable ψ T and are zero mean. Let S uv = ψψ T and denote the covariance matrix of ψ and , respectively, where u ∈ g(i) , v e g (i+1) . Then, according to equations (4)-(7), ψ and decomposition:

[0072]

[0073] ψ = TP T + E (5)

[0074]

[0075]

[0076] where p i and q i are loading vectors, t i and u i are projection weights, T and U are n x l matrices, projections of ψ and respectively, P and Q are m x l and p x l orthogonal loading matrices, respectively, and matrices E and F are error terms, which are independent and identically distributed normal distributed random variables. Finally, the best pair of features is found by equation (8):

[0077]

[0078] Step b-5, let the matrix composed of all characteristic projection vectors be W ψ = (a1, a2,..., a d ), Project the original sample (feature vector) into two subspaces, respectively, that is, z (i) = W ψ ψ, Add the highest degree of covariant feature pair to the fusion vector in parallel to obtain new features:

[0079]

[0080] Step b-6, repeat the above PLS parallel fusion algorithm until the spatial feature f1 is obtained;

[0081] Step c, for the input spectral information, the same operation as in step b is used to obtain the spectral feature f2;

[0082] Step d, f1 and f2 are connected and processed through three fully connected layers, and finally a softmax classifier is used for classification.

[0083] Step e, use the tree image training set and validation set obtained in step 3 to train and verify the convolutional neural network built in steps a-d, and obtain the trained convolutional neural network;

[0084] Step 5: input the segmented image obtained in step 2 into the convolutional neural network trained in step 4 to realize tree species recognition;

[0085] Step 6: count the tree species identified in step 5 by 1, and process successively until the identification and counting of all segmented crown images are completed.

[0086] The model constructed by the embodiment and the traditional CaffeNet model are used to process the same hyperspectral data, the recognition accuracy of the model constructed by the method is 94.65%, which is increased by 5.12% compared with the recognition accuracy (89.53%) of the traditional CaffeNet model.

[0087] The above describes the present application and its embodiments in a schematic manner, which is not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired thereby, without departing from the purpose of the present application, similar structural modes and embodiments can be designed without creativity, which shall belong to the protection scope of the present application.

Claims

1. A tree species identification, classification, and counting method based on PLS feature fusion convolutional neural network, characterized in that, include: The S100 performs tree canopy recognition on hyperspectral images of trees and segments the images into several individual tree canopy images; S200 establishes a partial least squares (PLS) feature fusion CaffeNet convolutional neural network, the structure of which is as follows: Based on the CaffeNet convolutional neural network structure, a branch CNN consisting of a 1×1 convolutional layer, a non-linear activation layer, and a global average pooling layer is inserted after each pooling layer for feature extraction. The first pooling layer and the branch of the second pooling layer are connected to the first PLS fusion module; the branch of the third pooling layer and the second fully connected layer are connected to the second PLS fusion module; the first feature fusion module and the second feature fusion module are connected to the third PLS fusion module. The feature fusion results output by the third PLS fusion module are obtained by taking the spatial information and spectral information of the hyperspectral image as inputs respectively. The feature fusion results of the spatial information and the feature fusion results of the spectral information are then input into the fourth PLS fusion module. The fourth feature fusion module is followed by three fully connected layers and a classifier in sequence; The branch CNN feature extraction process is as follows: The output of the pooling layer is input into a 1×1 convolutional layer, and the output is as follows: （1） Where b is the branch number, assuming there are B branches in total, and L is the number of output channels. It is weight; It is the feature of the k-th input channel, totaling One input channel; The output of the 1×1 convolutional layer is subjected to one-sided suppression by a nonlinear activation layer before being input into the GAP layer. use express The The following is the output of the GAP layer: （2） make: （3） This L-dimensional vector is the output of the b-th branch; The PLS fusion process executed by the first PLS fusion module, the second PLS fusion module, the third PLS fusion module, and the fourth PLS fusion module includes: Pick and Parallel fusion of PLS ​​algorithms is performed, where The dimension of shallow features. The dimension of deep features; right and Given data matrices respectively and Where the central variable is assumed and With zero mean, using and They represent and The covariance matrix, where , According to the following formulas (4)-(7) and Decompose: (4) (5) (6) (7) in, and These are the loading vectors, and These are the projection weights, and T and U are n×l matrices, respectively. and The projections of P and Q are the orthogonal loading matrices of m×l and p×l, respectively, and the matrices E and F are error terms that follow independent and identically distributed normal random variables. After finding the best feature pair, a matrix composed of all feature projection vectors is constructed. The original feature vectors are projected onto the two subspaces respectively. The feature pair with the highest covariance is added to the fusion vector to obtain new features, i.e., the PLS fusion result. The S300 collects hyperspectral images of trees and manually labels tree species to build a training set and a validation set of tree images, which are used to train the PLS feature fusion CaffeNet convolutional neural network built by the S200. The S400 takes the segmented individual tree crown image as input, uses the trained PLS features to fuse with the CaffeNet convolutional neural network to identify tree species, and counts the identified trees by species.

2. The method according to claim 1, characterized in that, Hyperspectral images of trees are acquired from the air using drones equipped with imaging spectrometers.

3. The method according to claim 1, characterized in that, Use the Faster R-CNN network to identify tree canopies.

4. The method according to claim 3, characterized in that, After identifying tree canopies using the Faster R-CNN network, each tree canopy is labeled with a target bounding box, and image segmentation is performed using the midpoint of the distance between adjacent bounding boxes as the boundary.

5. The method according to claim 1, characterized in that, The nonlinear activation layer uses the ReLU activation function.

6. The method according to claim 1, characterized in that, The classifier used is the softmax classifier.

Images