Convolutional neural network-based multi-spectral image coniferous forest tree species classification method

Abstract

The invention discloses a multi-spectral image coniferous forest tree species classification method based on a convolutional neural network, and relates to the technical field of tree species identification, and the method comprises the steps: collecting tree species multi-spectral images, constructing a tree species multi-spectral image data set, and carrying out the weighted principal component analysis and data enhancement processing of the tree species multi-spectral image data set; a U-Net model is used as a framework, a mode of combining deep convolution and cavity convolution of a continuous expansion rate is utilized, a Leaky ReLU activation function is applied, a multi-head cross attention mechanism is added on a jump path, and a tree species classification model is constructed; using the processed data set to optimize tree species classification model parameters through a mixed loss function and an Adam optimizer; and carrying out coniferous forest tree species classification based on the tree species classification model. According to the method, the learning and classification capabilities of complex tree species features are improved in a mode of combining deep convolution and cavity convolution with continuous expansion rate.

Description

Technical Field

[0001] The invention relates to the technical field of tree species identification, and in particular to a multispectral image coniferous forest tree species classification method based on convolutional neural network. Background Art

[0002] Coniferous forests are an important part of forest ecosystems, and accurate classification of their tree species has far-reaching significance for the rational use of forest resources, ecosystem monitoring, and biodiversity conservation. However, traditional methods for classifying coniferous forest species mainly rely on manual field surveys and remote sensing classification techniques based on single features, which have many limitations. Manual field surveys are time-consuming and labor-intensive, inefficient, and difficult to conduct comprehensive monitoring of large areas of coniferous forests. Remote sensing classification methods based on single features, such as using only spectral features or texture features, often fail to fully utilize the rich information in multispectral images, and have low accuracy in classifying coniferous forest species in complex environments, and the classification effect is difficult to meet actual needs.

[0003] Traditional machine learning-based classification methods are usually unable to fully explore the complex relationships and deep feature information between different bands in multispectral images. For vegetation such as coniferous forests with special spectral and structural characteristics, their classification performance is limited. In addition, the lack of an effective feature fusion mechanism leads to limited accuracy and generalization ability of tree species classification. Summary of the invention

[0004] The purpose of the present invention is to provide a coniferous forest tree species classification method based on multispectral images of coniferous forests based on convolutional neural networks to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a multispectral image coniferous forest tree species classification method based on convolutional neural network, the method comprising the steps of:

[0006] S1. Collect multispectral images of tree species, construct a tree species multispectral image dataset, and perform weighted principal component analysis and data enhancement processing on the tree species multispectral image dataset;

[0007] S2. Using the U-Net model as the architecture, we combined deep convolution with dilated convolution with continuous dilation rate, applied LeakyReLU activation function, and added multi-head cross attention mechanism on the skip path to build a tree species classification model.

[0008] S3. Using the processed data set, the difference between the predicted and true labels is calculated through a hybrid loss function that combines cross entropy loss and Dice loss, and then the Adam optimizer is used to optimize the tree species classification model parameters;

[0009] S4. Classify coniferous forest tree species based on the tree species classification model.

[0010] According to the above scheme, the weighted principal component analysis includes:

[0011] The original pixel value matrix X of the tree species multispectral image is (x ij ), where x ij It is represented as the pixel value of the i-th pixel on the j-th band; i = 1, 2, ..., n; i represents the number of samples, that is, the pixel point, n represents the total number of samples, j = 1, 2, ..., p; j represents the number of bands, and p represents the total number of bands;

[0012] The multispectral images of the tree species are standardized to eliminate the differences between different bands due to dimensions. The formula is as follows:

[0013]

[0014] Among them, z ij Represented as a matrix of normalized pixel values; Represented as the mean value of the j-th band data; s j Expressed as the standard deviation of the j-th band data;

[0015] Set the weight vector w=(w1,w2,…,w j ), and give a higher weight to the near-infrared band to effectively highlight the tree species information;

[0016] When a higher weight is given to the near-infrared band, vegetation information can be effectively highlighted because the near-infrared band is particularly sensitive to vegetation reflectance. In addition, coniferous tree species remain green all year round and can maintain a high near-infrared reflectance even in winter. Weighted principal component analysis that gives the near-infrared band a higher weight can reduce the impact of seasonal changes on classification results because the near-infrared band has little change in reflectance for tree species in different seasons and can effectively distinguish coniferous forest species even in the non-growing season.

[0017] Based on the standardization process and the weight vector, a covariance matrix C is obtained; the covariance matrix reflects the correlation between different bands and the discrete degree of data of each band;

[0018] Based on the covariance matrix C, the eigenvalues ​​λ1,λ2,…,λ p and the corresponding eigenvectors v1,v2,…,v p ; Among them, the eigenvalue represents the original data information contained in the corresponding principal component;

[0019] Sort the eigenvalues ​​in descending order and select the eigenvectors v1, v2, ..., v corresponding to the first m larger eigenvalues.m ; Wherein, m is a positive integer and m≤p;

[0020] Based on the pixel value matrix z after the normalization process ij and the selected eigenvector matrix V = (v1, v2, ..., v m ), obtain the weighted principal component matrix;

[0021] In the weighted principal component matrix, each column of data is the coniferous forest tree species characteristics extracted after weighted principal component analysis.

[0022] According to the above solution, the data enhancement processing includes:

[0023] Performing data enhancement processing on each image in the tree species multispectral image dataset one by one; the data enhancement processing includes rotation, horizontal flipping and scaling operations;

[0024] The rotation randomly selects a rotation angle, rotates each image in the tree species multispectral image data set one by one, and generates a new image each time the rotation is performed;

[0025] The horizontal flipping randomly selects part of the images in the tree species multispectral image dataset, mirror-flips the part of the images in the horizontal direction, and generates a new image.

[0026] The scaling includes setting a scaling ratio range, randomly selecting a scaling ratio value within the scaling ratio range, scaling each image in the tree species multispectral image dataset, and generating a new image each time the scaling is performed;

[0027] The new image is added to the data set as derivative data of the original image.

[0028] According to the above scheme, constructing a tree species classification model includes:

[0029] The tree species classification model is based on a U-Net model, which includes an encoder, a decoder, a skip connection and a final convolutional layer;

[0030] The encoder consists of a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a bottleneck convolution block, and is used to downsample the input image and extract features;

[0031] The decoder is composed of a first upsampling block, a second upsampling block, a third upsampling block and a fourth upsampling block, and is used to upsample the feature map output by the encoder to a larger spatial size;

[0032] The jump connection connects the corresponding blocks of the encoder and the decoder, splices the feature maps of the same size, effectively combines the feature information of different scales, and improves the accuracy of segmentation; and adds a multi-head cross attention mechanism on the jump connection path;

[0033] The multi-head cross attention mechanism captures different feature subspaces by computing multiple attention heads in parallel;

[0034] The final convolution layer uses a 1×1 convolution kernel to reduce the number of feature map channels to the required number of output channels.

[0035] According to the above scheme, the first convolution block receives an input image and transmits the output feature map to the next convolution block; the bottleneck convolution block includes three layers of convolution operations and outputs a fifth feature map of the same size as the feature map output by the fourth convolution block;

[0036] Four maximum pooling layers are inserted between the first convolution block, the second convolution block, the third convolution block, the fourth convolution block and the bottleneck convolution block to reduce the spatial dimension of the feature map;

[0037] The maximum pooling layer uses a pooling window to slide on the feature map output by the convolution block, and each time selects the pixel with the largest pixel value in the window as the value of the corresponding position of the output feature map, thereby reducing the spatial dimension of the feature map;

[0038] The first convolution block, the second convolution block, the third convolution block and the fourth convolution block respectively include two layers of convolution operations, and after each layer of convolution operation, a nonlinear transformation is performed using a LeakyReLU activation function;

[0039] The first upsampling block, the second upsampling block, the third upsampling block and the fourth upsampling block respectively include two layers of convolution operations;

[0040] The first upsampling block performs feature concatenation on the feature map output by the bottleneck convolution block and the feature map output by the fourth convolution block obtained through the jump connection to form the input of the first upsampling block, and performs upsampling;

[0041] After the first upsampling block performs upsampling, the output image is transmitted to the next upsampling block, and the feature map of the corresponding block is obtained through the jump connection to perform feature splicing;

[0042] The fourth upsampling block transmits the output image to the final convolutional layer;

[0043] Based on the final convolutional layer, a classification result graph is output.

[0044] According to the above scheme, the method of combining depth convolution with dilated convolution with continuous expansion rate includes:

[0045] The two-layer convolution operation in the encoder and the decoder is a combination of depthwise convolution and dilated convolution with continuous dilation rate;

[0046] Based on the deep convolution, feature extraction is performed;

[0047] The feature extraction is performed by sliding a 3×3 convolution kernel point by point at each position of the image. For each sliding position, the pixels in the surrounding 3×3 area are calculated to capture the local features of the image. The formula is as follows:

[0048]

[0049] Among them, y a Represented as a pixel in the output feature map of the deep convolution, with pixel position a; x c Represented as the pixel value at the position corresponding to the convolution kernel in the input image; w c represents the weight value of the convolution kernel; b represents the bias term; k represents the size of the convolution kernel; a and c represent the pixel positions of the output image and the input image respectively;

[0050] Based on the dilated convolution, the receptive field of the convolution kernel is increased by the dilation rate to capture a wider range of features;

[0051] The expansion rate is used to increase the receptive field of the convolution kernel. During the sliding process of the 3×3 convolution kernel, the adjacent calculation positions skip pixels according to the interval of the expansion rate, covering a larger range of image areas, so as to achieve the effect of capturing a larger range of features. The formula is as follows:

[0052]

[0053] Among them, z a Represented as a pixel in the output feature map of the dilated convolution, with pixel position a; x rate(c) It is represented as the pixel value sampled according to the dilation rate rate of the dilated convolution in the input image; rate(c) is represented as the sampling interval of the dilation rate;

[0054] The expansion rates of the dilated convolutions in the first convolution block, the second convolution block, the third convolution block, and the fourth convolution block are rate1, rate2, rate3, and rate4, respectively.

[0055] Different dilation rates enable each convolution block to obtain features of different scales and emphases; by setting dilated convolutions with different dilation rates in each convolution block, the model can learn richer and more representative features, from simple local features to complex environment-related features; the accumulation and combination of these features enhances the tree species classification model's ability to extract and analyze tree species features, thereby improving the accuracy and generalization ability of the tree species classification model in tree species classification tasks, and helping to accurately distinguish different tree species.

[0056] According to the above scheme, adding a multi-head cross attention mechanism on the skip connection path includes:

[0057] When the feature map output by the fourth convolution block is transmitted to the upsampling block, feature splicing is performed with the feature map output by the bottleneck convolution block to obtain a feature splicing map;

[0058] Based on the multi-head cross attention mechanism, the feature map output by the bottleneck convolution block is used as a query, and the feature map output by the fourth convolution block is used as a key and a value to obtain a multi-head cross attention output;

[0059] The multi-head cross attention output is combined with the feature splicing map to obtain a decoder feature map to help enhance feature information;

[0060] Based on the first upsampling block, performing the upsampling on the decoder feature map;

[0061] The use of multi-head cross-attention mechanism on skip connections helps to integrate low-level local detail information from the encoder and high-level semantic information from the decoder, improves the fusion effect of feature information, enables the model to better understand and utilize information at different scales, and better capture and utilize the subtle feature differences of different tree species in multispectral images.

[0062] According to the above scheme, the nonlinear transformation of the LeakyReLU activation function includes:

[0063] The LeakyReLU activation function sets the α value so that the gradient in the negative semi-axis area still exists. The formula is as follows:

[0064]

[0065] Where f(d) is the LeakyReLU activation function, d is the input value, and α is a small constant used to control the slope of the activation function when d<0, that is, the negative slope.

[0066] The tree species classification model is optimized based on the Adam optimizer, and the optimal α value is selected as the negative slope.

[0067] According to the above scheme, the processed data set is used to calculate the difference between the prediction and the true label through a hybrid loss function combining cross entropy loss and Dice loss, and then the Adam optimizer is used to optimize the tree species classification model parameters, including:

[0068] Randomly divide the processed data set into a training set, a validation set and a test set;

[0069] In each training cycle, a tree species classification model is trained based on the training set;

[0070] In the process of training the tree species classification model, a hybrid loss function that combines the cross entropy loss function and the Dice loss function is used to compare the prediction results of the tree species classification model with the true labels of the training set, calculate the hybrid loss value, and measure the difference between the model prediction and the actual situation;

[0071] The cross entropy loss function calculates the difference between the predicted category and the true category, and guides the Adam optimizer to adjust the model so that the classification accuracy of each pixel is higher, solving the problem of sample category imbalance. The Dice loss function evaluates the overlap between the predicted and true label areas. The Dice loss function pays more attention to the prediction accuracy of the tree species classification model in the region, which can help the network improve its sensitivity to object boundaries and avoid over-smoothing or inaccurate segmentation. The combination of the two can not only optimize the classification accuracy of each pixel, but also ensure the quality of the segmented area, thereby improving the accuracy of the tree species classification model.

[0072] Through the back propagation algorithm, based on the mixed loss value, starting from the mixed loss function, back propagating along the network layer, the gradient of each parameter is calculated; the gradient represents the rate of change of the loss function to each parameter, indicating the direction of parameter adjustment;

[0073] The Adam optimizer automatically adjusts the learning rate for each parameter based on the gradient using the first-order moment and the second-order moment of the gradient; the learning rate determines the step size of the parameter update in the gradient direction;

[0074] Automatically adjusting the learning rate based on each parameter, and continuously updating each parameter of the tree species classification model in each training cycle, so as to accelerate the training process and improve the stability of convergence;

[0075] Each of the parameters includes weight, bias and negative slope α value of LeakyReLU activation function;

[0076] Set the batch size and learning rate to ensure that the training is stable and effective;

[0077] After the training cycle is over, the performance of the tree species classification model is evaluated based on the validation set to help select hyperparameters; when the performance of the validation set no longer improves after several cycles, the early stopping mechanism is triggered to stop training to avoid overfitting;

[0078] Based on the test set, the final performance of the tree species classification model is evaluated to simulate the performance of the tree species classification model in actual data.

[0079] According to the above scheme, the tree species classification model outputs a classification result map of the same size as the input tree species multispectral image; the classification result map indicates the location of each tree species by color coding;

[0080] Based on the classification result map, the proportions of different tree species in the classification result map are counted and labeled for display.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] 1. The present invention combines the characteristics of coniferous forests and uses weighted principal component analysis to stably extract tree species characteristics and improve classification accuracy;

[0083] 2. The present invention improves the learning and classification capabilities of complex tree species features by combining deep convolution with dilated convolution with continuous expansion rates;

[0084] 3. The present invention uses the Adam optimizer to automatically adjust the learning rate for each parameter, avoiding the problem of manually adjusting the global learning rate and improving the efficiency of the tree species classification model;

[0085] 4. The present invention utilizes a hybrid loss function that combines the cross entropy loss function and the Dice loss function to enable the tree species classification model to achieve better results in both pixel classification and region segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 A flowchart of the steps of a method for classifying coniferous forest species using multispectral images based on a convolutional neural network according to the present invention;

[0087] Figure 2 The present invention is a structural schematic diagram of a tree species classification model of a multispectral image coniferous forest tree species classification method based on a convolutional neural network. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, a multispectral image coniferous forest tree species classification method based on convolutional neural network, the method comprising the steps of:

[0090] S1. Collect multispectral images of tree species, construct a tree species multispectral image dataset, and perform weighted principal component analysis and data enhancement processing on the tree species multispectral image dataset;

[0091] Specifically, drones, satellite remote sensing or aerial smoke sensing equipment are used to obtain multispectral images of coniferous forest areas. The images cover coniferous forests of different seasons, different geographical regions, and different growth states, covering all possible situations. The resolution of each image is 1024x1024 pixels, with a total of 150 images. The collected images are annotated according to tree species to construct a tree species multispectral image dataset.

[0092] The weighted principal component analysis is performed on the tree species multispectral image dataset. Specifically, first, the original pixel value matrix X of the tree species multispectral image is obtained. ij ), where x ij It is represented as the pixel value of the i-th pixel on the j-th band; i = 1, 2, ..., n; i represents the number of samples, that is, the pixel point, n represents the total number of samples, j = 1, 2, ..., p; j represents the number of bands, and p represents the total number of bands; for example, the acquired multispectral image data set contains 1000 samples, each sample has 9 bands, and the original pixel value matrix X = (x ij ), where i = 1, 2, ..., 1000; j = 1, 2, ..., 9;

[0093] The multispectral images of the tree species are standardized to eliminate the differences between different bands due to dimensions. The formula is as follows:

[0094]

[0095] Among them, z ij Represented as a matrix of normalized pixel values; Represented as the mean value of the j-th band data; s j Expressed as the standard deviation of the j-th band data;

[0096] Set the weight vector w=(w1,w2,…,w j ), and give a higher weight to the near-infrared band to effectively highlight the tree species information; for example, the near-infrared band w NIR =0.2, other bands are 0.1;

[0097] When a higher weight is given to the near-infrared band, vegetation information can be effectively highlighted because the near-infrared band is particularly sensitive to vegetation reflectance. In addition, coniferous tree species remain green all year round and can maintain a high near-infrared reflectance even in winter. Weighted principal component analysis that gives the near-infrared band a higher weight can reduce the impact of seasonal changes on classification results because the near-infrared band has little change in reflectance for tree species in different seasons and can effectively distinguish coniferous forest species even in the non-growing season.

[0098] Based on the standardization process and the weight vector, a covariance matrix C is obtained; the covariance matrix reflects the correlation between different bands and the discrete degree of data of each band;

[0099] Based on the covariance matrix C, the eigenvalues ​​λ1,λ2,…,λ p and the corresponding eigenvectors v1,v2,…,v p ; Among them, the eigenvalue represents the original data information contained in the corresponding principal component;

[0100] Sort the eigenvalues ​​in descending order and select the eigenvectors v1, v2, ..., v corresponding to the first m larger eigenvalues. m ; Wherein, m is a positive integer and m≤p;

[0101] Based on the pixel value matrix z after the normalization process ij and the selected eigenvector matrix V = (v1, v2, ..., v m ), obtain the weighted principal component matrix;

[0102] In the weighted principal component matrix, each column of data is the coniferous forest tree species characteristics extracted after weighted principal component analysis.

[0103] Performing data enhancement processing on a tree species multispectral image dataset; specifically, performing data enhancement processing on each image in the tree species multispectral image dataset one by one; the data enhancement processing includes rotation, horizontal flipping and scaling operations;

[0104] Specifically, the rotation angle is randomly selected, and each image in the tree species multispectral image dataset is rotated one by one, and a new image is generated each time the rotation is performed; for example, the random rotation angle θ is in the range of [-30°, 30°]. For the first image, the rotation angle θ=15° is randomly selected, and the image is rotated 15° clockwise to generate a new image; the horizontal flipping randomly selects part of the images in the tree species multispectral image dataset, and mirror-flips the part of the images in the horizontal direction to generate a new image; for example, 50% of the images in the dataset are randomly selected and flipped. It is mirror-flipped in the horizontal direction to generate a new image; for the scaling, a scaling ratio range is set, and within the scaling ratio range, a scaling ratio value is randomly selected to scale each image in the tree species multispectral image dataset, and a new image is generated each time it is scaled; for example, the scaling ratio range is set to [0.8, 1.2], and for the first image, a scaling ratio of 0.9 is randomly selected, and it is scaled at a ratio of 0.9 to generate a new image; the new image is added to the dataset as the derivative data of the original image, so that the original dataset is expanded from 150 to 4160.

[0105] S2. Using the U-Net model as the architecture, we combined deep convolution with dilated convolution with continuous dilation rate, applied LeakyReLU activation function, and added multi-head cross attention mechanism on the skip path to build a tree species classification model.

[0106] Specifically, the tree species classification model is based on the U-Net model, which includes an encoder, a decoder, a jump connection and a final convolution layer; the encoder consists of a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a bottleneck convolution block, which are used to downsample the input image and extract features; the decoder consists of a first upsampling block, a second upsampling block, a third upsampling block and a fourth upsampling block, which are used to upsample the feature map output by the encoder to a larger spatial size; the jump connection connects the corresponding blocks of the encoder and the decoder, splices the feature maps of the same size, effectively combines feature information of different scales, and improves the accuracy of segmentation; and adds a multi-head cross attention mechanism on the jump connection path; the multi-head cross attention mechanism captures different feature subspaces by calculating multiple attention heads in parallel; the final convolution layer uses a 1×1 convolution kernel to reduce the number of feature map channels to the required number of output channels.

[0107] Specifically, first, the first convolution block receives the input image and transmits the output feature map to the next convolution block; for example: the resolution of the input image is 1024x1024 pixels; the image undergoes two 3×3 convolutions in the first convolution block, and after each convolution operation, the LeakyReLU activation function is used for nonlinear transformation. After completing the convolution operation in the first convolution block, it enters the maximum pooling layer, performs pooling operation, and outputs a feature map with a resolution of 512x512 pixels; the feature map with a resolution of 512x512 pixels is transmitted to the second convolution block, and a feature map with a resolution of 256x256 pixels is output; the feature map with a resolution of 256x256 pixels is transmitted to the third convolution block, and a feature map with a resolution of 128x128 pixels is output; the feature map with a resolution of 128x128 pixels is transmitted to the fourth convolution block, and a feature map with a resolution of 64x64 pixels is output;

[0108] The bottleneck convolution block includes three layers of convolution operations and outputs a feature map with a resolution of 64x64 pixels;

[0109] When the feature map with a resolution of 64x64 pixels output by the fourth convolution block is passed to the first upsampling block, feature splicing is performed with the feature map with a resolution of 64x64 pixels output by the bottleneck convolution block to obtain a feature splicing map; and based on the multi-head cross attention mechanism, the feature map output by the bottleneck convolution block is used as a query, and the feature map output by the fourth convolution block is used as a key and value to obtain a multi-head cross attention output;

[0110] Specifically, the multi-head cross-attention output is combined with the feature splicing map to obtain the decoder feature map, which helps to enhance the feature information; based on the first upsampling block, the decoder feature map is upsampled; the use of the multi-head cross-attention mechanism on the jump connection helps to integrate the low-level local detail information from the encoder and the high-level semantic information from the decoder, thereby improving the fusion effect of the feature information, allowing the model to better understand and utilize information at different scales, and better capture and utilize the subtle feature differences of different tree species in multispectral images.

[0111] The first upsampling block, the second upsampling block, the third upsampling block and the fourth upsampling block respectively include two layers of convolution operations; the two layers of convolution operations include the depth convolution and the hole convolution;

[0112] Specifically, first, the first upsampling block performs feature concatenation on the feature map output by the bottleneck convolution block and the feature map output by the fourth convolution block obtained through the jump connection to form the input of the first upsampling block, and performs upsampling; after the first upsampling block performs upsampling, the output image is transmitted to the second upsampling block; the second upsampling block performs feature concatenation on the image output by the first upsampling block and the feature map output by the third convolution block obtained through the jump connection to form the input of the second upsampling block, and performs upsampling; after the second upsampling block performs upsampling, the output image is transmitted to the third upsampling block; The three upsampling blocks perform feature concatenation on the image output by the second upsampling block and the feature map output by the second convolution block obtained through the jump connection to form the input of the third upsampling block, and perform upsampling; after the third upsampling block performs upsampling, the output image is transmitted to the fourth upsampling block; the fourth upsampling block performs feature concatenation on the image output by the third upsampling block and the feature map output by the first convolution block obtained through the jump connection to form the input of the fourth upsampling block, and performs upsampling; the fourth upsampling block transmits the output image to the final convolution layer; based on the final convolution layer, the classification result map is output.

[0113] Specifically, the two-layer convolution operation includes depthwise convolution and dilated convolution. Based on depthwise convolution, feature extraction is performed. For feature extraction, a 3×3 convolution kernel is used to slide point by point at each position of the image. For each sliding position, the pixels in the surrounding 3×3 area are calculated to capture the local features of the image. The formula is as follows:

[0114]

[0115] Among them, y a Represented as a pixel in the output feature map of the deep convolution, with pixel position a; x c Represented as the pixel value at the position corresponding to the convolution kernel in the input image; w c represents the weight value of the convolution kernel; b represents the bias term; k represents the size of the convolution kernel; a and c represent the pixel positions of the output image and the input image respectively;

[0116] The dilated convolution increases the receptive field of the convolution kernel by the dilation rate to capture features in a wider range. The dilation rate is used to increase the receptive field of the convolution kernel. During the sliding process of the 3×3 convolution kernel, adjacent calculation positions skip pixels according to the interval of the dilation rate, covering a wider range of image areas, thereby capturing features in a wider range. The formula is as follows:

[0117]

[0118] Among them, z aRepresented as a pixel in the output feature map of the dilated convolution, with pixel position a; x rate(c) It is represented as the pixel value sampled according to the dilation rate rate of the dilated convolution in the input image; rate(c) is represented as the sampling interval of the dilation rate; for example, when rate = 2, during the sliding process, the adjacent calculation positions skip pixels according to the interval 2, and calculate a pixel z in the output feature map a ; Specifically, the dilation rates of the dilated convolutions in different convolution blocks are: dilation rate rate1 of the first convolution block, dilation rate rate2 of the second convolution block, dilation rate rate3 of the third convolution block, and dilation rate rate4 of the fourth convolution block; for example: dilation rate rate1 of the first convolution block is 1, dilation rate rate2 of the second convolution block is 2, dilation rate rate3 of the third convolution block is 3, and dilation rate rate4 of the fourth convolution block is 4; different dilation rates enable each convolution block to obtain features of different scales and different focuses; by setting dilated convolutions with different dilation rates in each convolution block, the model can learn richer and more representative features, from simple local features to complex environment-related features; the accumulation and combination of these features enhance the ability of the tree species classification model to extract and analyze tree species features, thereby improving the accuracy and generalization ability of the tree species classification model in tree species classification tasks, and helping to accurately distinguish different tree species.

[0119] Specifically, the LeakyReLU activation function sets the α value so that the gradient in the negative semi-axis area still exists. The formula is as follows:

[0120]

[0121] Among them, f(d) represents the LeakyReLU activation function, d represents the input value; α is a small constant used to control the slope of the activation function when d<0, that is, the negative slope; based on the Adam optimizer to optimize the tree species classification model, the optimal α value is selected as the negative slope.

[0122] S3. Using the processed data set, the difference between the predicted and true labels is calculated through a hybrid loss function that combines cross entropy loss and Dice loss, and then the Adam optimizer is used to optimize the tree species classification model parameters;

[0123] Specifically, first, the processed data set is randomly divided into a training set, a validation set and a test set; for example, the processed 4160 image data sets are randomly divided into a training set (2912 images), a validation set (624 images) and a test set (624 images) in a ratio of 70:15:15; in each training cycle, the tree species classification model is trained based on the training set; in the process of training the tree species classification model, a mixed loss function that combines the cross entropy loss function and the Dice loss function is used to compare the prediction results of the tree species classification model with the true labels of the training set, and the mixed loss value is calculated to measure the difference between the model prediction and the actual situation; through the back propagation algorithm, based on the mixed loss value, starting from the mixed loss function, back propagating along the network layer, the gradient of each parameter is calculated; the gradient represents the rate of change of the loss function to each parameter, indicating the direction of parameter adjustment; the Adam optimizer, based on the gradient, uses the first-order moment and second-order moment of the gradient to automatically adjust the learning rate for each parameter; the learning rate determines the parameter in the gradient direction. a new step size; automatically adjusting the learning rate based on each parameter, and continuously updating each parameter of the tree species classification model in each training cycle to accelerate the training process and improve the stability of convergence; wherein each parameter includes the weight, bias, and negative slope α value of the LeakyReLU activation function; setting the batch size and learning rate to ensure that the training is stable and effective; after the training cycle, evaluating the performance of the tree species classification model based on the validation set to help select hyperparameters; when the performance of the validation set no longer improves after several cycles, triggering the early stopping mechanism to stop training to avoid overfitting; for example, when the performance of the validation set no longer improves after 10 consecutive cycles, triggering the early stopping mechanism to stop training to avoid overfitting; using the gradient-based optimization algorithm, using the α value as the optimized hyperparameter, and optimizing according to the validation set as the objective function; evaluating the final performance of the tree species classification model based on the test set, for example, using indicators such as accuracy, recall, and F-value to evaluate model performance, helping to select hyperparameters, and simulating the performance of the tree species classification model in actual data.

[0124] S4. Classify coniferous forest tree species based on the tree species classification model.

[0125] Specifically, the tree species classification model outputs a classification result map with the same size as the input tree species multispectral image; the classification result map indicates the location of each tree species by color coding, for example: the spruce area is marked as blue, the pine area is marked as green, and the fir area is marked as red, indicating the location of each tree species. This is only an example and is not limited. Based on the classification result map, the proportion of different tree species in the classification result map is counted and labeled for display, for example: the blue area (spruce) accounts for 30%, the green area (pine) accounts for 50%, and the red area (fir) accounts for 20%. These proportions are labeled and displayed to intuitively understand the distribution of different tree species in the test area.

[0126] The present invention provides a technical solution, a tree species classification model of a multispectral image coniferous forest tree species classification method based on a convolutional neural network;

[0127] Specifically, the tree species classification model is based on the U-Net model architecture, which includes an encoder, a decoder, a skip connection and a final convolution layer; the encoder consists of a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a bottleneck convolution block; the decoder consists of a first upsampling block, a second upsampling block, a third upsampling block and a fourth upsampling block; the skip connection connects the corresponding blocks of the encoder and the decoder; a multi-head cross attention mechanism is added on the skip connection path; the final convolution layer consists of a 1×1 convolution kernel.

[0128] Specifically, the steps of the tree species classification model include: first, input the processed multispectral coniferous forest image into the first convolution block; in the first convolution block, after two 3×3 convolutions, and after each convolution operation, use the LeakyReLU activation function for nonlinear transformation; after completing the convolution and activation function operations, input to the maximum pooling layer, perform pooling operations, and transfer the output feature map to the second convolution block; the operations of the second convolution block, the third convolution block and the fourth convolution block are the same as those of the first convolution block, and the second convolution block transfers the output feature map to the third convolution block; the third convolution block transfers the output feature map to the fourth convolution block; and the bottleneck convolution block includes three layers of convolution operations, and outputs a feature map of the same size as the feature map output by the fourth convolution block;

[0129] Next, when the fourth convolution block transmits the output feature map to the first upsampling block, it performs feature splicing with the feature map output by the bottleneck convolution block to obtain a feature splicing map; and based on the multi-head cross attention mechanism, the feature map output by the bottleneck convolution block is used as a query, and the feature map output by the fourth convolution block is used as a key and value to obtain a multi-head cross attention output;

[0130] The first upsampling block, the second upsampling block, the third upsampling block and the fourth upsampling block respectively include two layers of convolution operations; the two layers of convolution operations include the depth convolution and the hole convolution; when the first upsampling block receives the multi-head cross attention output of the feature splicing map of the fourth convolution block and the bottleneck convolution block, it performs upsampling; after the first upsampling block performs upsampling, the output image is transmitted to the second upsampling block; the second upsampling block performs feature splicing on the image output by the first upsampling block and the feature map output by the third convolution block obtained through the jump connection to form the input of the second upsampling block, and performs upsampling; after the second upsampling block performs upsampling, the input The output image is transmitted to the third upsampling block; the third upsampling block performs feature splicing on the image output by the second upsampling block and the feature map output by the second convolution block obtained through the jump connection to form the input of the third upsampling block, and performs upsampling; after the third upsampling block performs upsampling, the output image is transmitted to the fourth upsampling block; the fourth upsampling block performs feature splicing on the image output by the third upsampling block and the feature map output by the first convolution block obtained through the jump connection to form the input of the fourth upsampling block, and performs upsampling; the fourth upsampling block transmits the output image to the final convolution layer; based on the final convolution layer, the classification result map is output.

[0131] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A coniferous forest tree species classification method based on multispectral images using convolutional neural networks, characterized by: The method comprises the steps of: S1. Collect multispectral images of tree species, construct a tree species multispectral image dataset, and perform weighted principal component analysis and data enhancement processing on the tree species multispectral image dataset; S2. Using the U-Net model as the architecture, we combined deep convolution with dilated convolution with continuous dilation rate, applied LeakyReLU activation function, and added multi-head cross attention mechanism on the skip path to build a tree species classification model. S3. Using the processed data set, the difference between the predicted and true labels is calculated through a hybrid loss function that combines cross entropy loss and Dice loss, and then the Adam optimizer is used to optimize the tree species classification model parameters; S4. Classify coniferous forest tree species based on the tree species classification model.

2. The method for classifying coniferous forest species based on multispectral images of coniferous forests based on convolutional neural networks according to claim 1, characterized in that: The weighted principal component analysis includes: The original pixel value matrix X of the tree species multispectral image is (x ij ), where x ij It is represented as the pixel value of the i-th pixel on the j-th band; i = 1, 2, ..., n; i represents the number of samples, that is, the pixel point, n represents the total number of samples, j = 1, 2, ..., p; j represents the number of bands, and p represents the total number of bands; The multispectral images of the tree species are standardized to eliminate the differences between different bands due to dimensions. The formula is as follows: Among them, z ij Represented as a matrix of normalized pixel values; Represented as the mean value of the j-th band data; s j Expressed as the standard deviation of the j-th band data; Set the weight vector w=(w1,w2,…,w j ), and give a higher weight to the near-infrared band to effectively highlight the tree species information; Based on the standardization process and the weight vector, a covariance matrix C is obtained; the covariance matrix reflects the correlation between different bands and the discrete degree of data of each band; Based on the covariance matrix C, the eigenvalues ​​λ1,λ2,…,λ p and the corresponding eigenvectors v1,v2,…,v p ; Among them, the eigenvalue represents the original data information contained in the corresponding principal component; Sort the eigenvalues ​​in descending order and select the eigenvectors v1, v2, ..., v corresponding to the first m larger eigenvalues. m ; Wherein, m is a positive integer and m≤p; Based on the pixel value matrix z after the normalization process ij and the selected eigenvector matrix V = (v1, v2, ..., v m ), obtain the weighted principal component matrix; In the weighted principal component matrix, each column of data is the coniferous forest tree species characteristics extracted after weighted principal component analysis.

3. The method for classifying coniferous forest species based on multispectral images of coniferous forests based on convolutional neural networks according to claim 1, characterized in that: The data enhancement process includes: Performing data enhancement processing on each image in the tree species multispectral image dataset one by one; the data enhancement processing includes rotation, horizontal flipping and scaling operations; The rotation randomly selects a rotation angle, rotates each image in the tree species multispectral image data set one by one, and generates a new image each time the rotation is performed; The horizontal flipping randomly selects part of the images in the tree species multispectral image dataset, mirror-flips the part of the images in the horizontal direction, and generates a new image. The scaling includes setting a scaling ratio range, randomly selecting a scaling ratio value within the scaling ratio range, scaling each image in the tree species multispectral image dataset, and generating a new image each time the scaling is performed; The new image is added to the data set as derivative data of the original image.

4. The method for classifying coniferous forest species based on multispectral images of coniferous forests based on convolutional neural networks according to claim 1, characterized in that: The constructing of the tree species classification model comprises: The tree species classification model is based on a U-Net model, which includes an encoder, a decoder, a skip connection and a final convolutional layer; The encoder consists of a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a bottleneck convolution block, and is used to downsample the input image and extract features; The decoder is composed of a first upsampling block, a second upsampling block, a third upsampling block and a fourth upsampling block, and is used to upsample the feature map output by the encoder to a larger spatial size; The jump connection connects the corresponding blocks of the encoder and the decoder, splices the feature maps of the same size, effectively combines the feature information of different scales, and improves the accuracy of segmentation; and adds a multi-head cross attention mechanism on the jump connection path; The multi-head cross attention mechanism captures different feature subspaces by computing multiple attention heads in parallel; The final convolution layer uses a 1×1 convolution kernel to reduce the number of feature map channels to the required number of output channels.

5. The method for classifying coniferous forest species based on multispectral images of coniferous forests based on convolutional neural networks according to claim 4, characterized in that: The first convolution block receives an input image and transmits an output feature map to a next convolution block; the bottleneck convolution block includes three layers of convolution operations and outputs a fifth feature map of the same size as the feature map output by the fourth convolution block; Four maximum pooling layers are inserted between the first convolution block, the second convolution block, the third convolution block, the fourth convolution block and the bottleneck convolution block to reduce the spatial dimension of the feature map; The maximum pooling layer uses a pooling window to slide on the feature map output by the convolution block, and each time selects the pixel with the largest pixel value in the window as the value of the corresponding position of the output feature map, thereby reducing the spatial dimension of the feature map; The first convolution block, the second convolution block, the third convolution block and the fourth convolution block respectively include two layers of convolution operations, and after each layer of convolution operation, a nonlinear transformation is performed using a LeakyReLU activation function; The first upsampling block, the second upsampling block, the third upsampling block and the fourth upsampling block respectively include two layers of convolution operations; The first upsampling block performs feature concatenation on the feature map output by the bottleneck convolution block and the feature map output by the fourth convolution block obtained through the jump connection to form the input of the first upsampling block, and performs upsampling; After the first upsampling block performs upsampling, the output image is transmitted to the next upsampling block, and the feature map of the corresponding block is obtained through the jump connection to perform feature splicing; The fourth upsampling block transmits the output image to the final convolutional layer; Based on the final convolutional layer, a classification result graph is output.

6. The method for classifying coniferous forest species based on multispectral images using convolutional neural networks according to claim 5, characterized in that: The method of combining depth convolution with dilated convolution with continuous expansion rate includes: The two-layer convolution operation in the encoder and the decoder is a combination of depthwise convolution and dilated convolution with continuous dilation rate; Based on the deep convolution, feature extraction is performed; The feature extraction is performed by sliding a 3×3 convolution kernel point by point at each position of the image. For each sliding position, the pixels in the surrounding 3×3 area are calculated to capture the local features of the image. The formula is as follows: Among them, y a Represented as a pixel in the output feature map of the deep convolution, with pixel position a; x c Represented as the pixel value at the position corresponding to the convolution kernel in the input image; w c represents the weight value of the convolution kernel; b represents the bias term; k represents the size of the convolution kernel; a and c represent the pixel positions of the output image and the input image respectively; Based on the dilated convolution, the receptive field of the convolution kernel is increased by the dilation rate to capture a wider range of features; The expansion rate is used to increase the receptive field of the convolution kernel. During the sliding process of the 3×3 convolution kernel, the adjacent calculation positions skip pixels according to the interval of the expansion rate, covering a larger range of image areas, so as to achieve the effect of capturing a larger range of features. The formula is as follows: Among them, z a Represented as a pixel in the output feature map of the dilated convolution, with pixel position a; x rate(c) It is represented as the pixel value sampled according to the dilation rate rate of the dilated convolution in the input image; rate(c) is represented as the sampling interval of the dilation rate; The expansion rates of the hole convolutions in the first convolution block, the second convolution block, the third convolution block and the fourth convolution block are rate1, rate2, rate3 and rate4, respectively.

7. The method for classifying coniferous forest species based on multispectral images of coniferous forests based on convolutional neural networks according to claim 4, characterized in that: Adding a multi-head cross attention mechanism on the skip connection path includes: When the feature map output by the fourth convolution block is transmitted to the upsampling block, feature splicing is performed with the feature map output by the bottleneck convolution block to obtain a feature splicing map; Based on the multi-head cross attention mechanism, the feature map output by the bottleneck convolution block is used as a query, and the feature map output by the fourth convolution block is used as a key and a value to obtain a multi-head cross attention output; The multi-head cross attention output is combined with the feature splicing map to obtain a decoder feature map to help enhance feature information; Based on the first upsampling block, the upsampling is performed on the decoder feature map.

8. The method for classifying coniferous forest species based on multispectral images of coniferous forests based on convolutional neural networks according to claim 4, characterized in that: The nonlinear transformation performed by the LeakyReLU activation function includes: The LeakyReLU activation function sets the α value so that the gradient in the negative semi-axis area still exists. The formula is as follows: Where f(d) is the LeakyReLU activation function, d is the input value, and α is a small constant used to control the slope of the activation function when d<0, that is, the negative slope. The tree species classification model is optimized based on the Adam optimizer, and the optimal α value is selected as the negative slope.

9. The method for classifying coniferous forest species based on multispectral images of coniferous forests based on convolutional neural networks according to claim 1, characterized in that: The processed data set is used to calculate the difference between the prediction and the true label through a hybrid loss function combining cross entropy loss and Dice loss, and then the Adam optimizer is used to optimize the tree classification model parameters, including: Randomly divide the processed data set into a training set, a validation set and a test set; In each training cycle, a tree species classification model is trained based on the training set; In the process of training the tree species classification model, a hybrid loss function that combines the cross entropy loss function and the Dice loss function is used to compare the prediction results of the tree species classification model with the true labels of the training set, calculate the hybrid loss value, and measure the difference between the model prediction and the actual situation; Through the back propagation algorithm, based on the mixed loss value, starting from the mixed loss function, back propagating along the network layer, the gradient of each parameter is calculated; the gradient represents the rate of change of the loss function to each parameter, indicating the direction of parameter adjustment; The Adam optimizer automatically adjusts the learning rate for each parameter based on the gradient using the first-order moment and the second-order moment of the gradient; the learning rate determines the step size of the parameter update in the gradient direction; Automatically adjusting the learning rate based on each parameter, and continuously updating each parameter of the tree species classification model in each training cycle, so as to accelerate the training process and improve the stability of convergence; Each of the parameters includes weight, bias and negative slope α value of LeakyReLU activation function; Set the batch size and learning rate to ensure that the training is stable and effective; After the training cycle is over, the performance of the tree species classification model is evaluated based on the validation set to help select hyperparameters; when the performance of the validation set no longer improves after several cycles, the early stopping mechanism is triggered to stop training to avoid overfitting; Based on the test set, the final performance of the tree species classification model is evaluated to simulate the performance of the tree species classification model in actual data.

10. The method for classifying coniferous forest species based on multispectral images using convolutional neural networks according to claim 1, characterized in that: The tree species classification model outputs a classification result map of the same size as the input tree species multispectral image; the classification result map indicates the location of each tree species by color coding; Based on the classification result map, the proportions of different tree species in the classification result map are counted and labeled for display.

Images