Method for realizing tree species classification based on RSVMama and unmanned aerial vehicle RGB data

Abstract

The invention provides a tree species classification method based on RSVMama and unmanned aerial vehicle RGB data, and belongs to the technical field of image processing. Comprising the following steps: S1, introducing an unmanned aerial vehicle RGB tree species classification data set, and constructing a tree species classification data set; and S2, constructing a tree species classification model RSVMama, combining an improved VSS module and an AutoDownsampling module, and realizing effective classification based on the constructed data set on the model. According to the method, an unmanned aerial vehicle RGB tree species classification data set is introduced, the effectiveness of single-temporal unmanned aerial vehicle RGB data for realizing multi-tree species classification in a complex heterogeneity forest is verified, and the dependence of a tree species classification task on hyperspectral data is reduced. According to the method, an RSVMama model is provided, tree species classification based on single-temporal unmanned aerial vehicle RGB data in a complex forest environment is realized, and the classification precision of eight tree species, withered trees and other broad-leaved trees reaches 84.28%.

Description

Technical Field

[0001] The invention provides a tree species classification method based on RSVMamba and unmanned aerial vehicle RGB data, belonging to the technical field of image processing. Background Art

[0002] Effective tree species classification provides comprehensive data support for biodiversity conservation, sustainable forest management, environmental monitoring and ecological research. In recent years, UAV remote sensing has become an important data source for this task. In tree species classification research based on UAV remote sensing images, hyperspectral data is the mainstream data source. Although the fusion of hyperspectral data and multi-source data has brought significant classification effects, the collection cost of such data, especially hyperspectral data and LiDAR data, is high, and flights under specific conditions are required to obtain high-quality images. In addition, the volume and processing complexity of hyperspectral data lead to extremely high storage and processing costs, as well as a large amount of computing resources. Data redundancy and dimensionality disaster also increase the complexity of band selection and feature extraction, which limits its large-scale practical application.

[0003] Some existing studies have also explored the application of RGB data combined with different deep learning methods in tree species classification. Due to different tree species classification categories and different classification levels, the classification accuracy is distributed between 75% and 90%. At present, there are few studies on tree species classification based on RGB data, and most of them focus on comparative studies of different deep learning methods. Although these methods only use RGB data, they can usually only classify 3-5 fewer tree species categories, and most of them are implemented using multi-temporal RGB data, which is difficult to apply to complex forest environments containing a large number of tree species, especially those areas with high vegetation diversity. In addition, the need for multi-temporal data limits its real-time performance in specific applications.

[0004] Although deep learning methods have shown superior performance in tree species classification tasks based on remote sensing images, existing models rarely consider the particularity of tree species classification tasks, the complexity of forest environments, and the differences in tree species distribution. There is a lack of exploration of new deep neural network models with better performance and stronger feature expression capabilities specifically for tree species classification tasks. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a tree species classification method based on RSVMamba and UAV RGB data, collects UAV RGB images covering the forest and constructs a tree species classification dataset, and evaluates the potential of single-phase UAV RGB data in the classification of tree species in subtropical heterogeneous forests. Only single-phase UAV RGB data will be used to classify 8 tree species, dead trees and other broad-leaved trees in subtropical heterogeneous forests. A deep learning visual modeling method for tree species classification tasks is adopted to make full use of the rich spatial, semantic and visible spectral information of UAV RGB images. The overall accuracy of this method for 8 tree species, dead trees and other broad-leaved trees reaches 84.28%, which confirms the effectiveness of single-phase UAV RGB images in tree species classification in heterogeneous forest environments.

[0006] The complete technical solution provided by the present invention is:

[0007] S1. The UAV RGB tree species classification dataset was introduced to construct a tree species classification dataset;

[0008] S2. Construct a tree species classification model RSVMamba, combine it with the improved VSS module and AutoDownsampling module, and implement effective classification based on the constructed dataset on this model.

[0009] The dataset construction and processing in S1 includes the following sub-steps:

[0010] (1) Constructing a tree species classification dataset:

[0011] Collect detailed information including forest cover, forest type, tree species composition, stand age structure and forest land quality to define the scope of the study area;

[0012] Including RGB data collection and sample site data collection;

[0013] Dataset preprocessing: The original RGB images were stitched and cropped, and images containing non-forest areas were removed. Subsequently, tree species classification categories were determined based on the tree species composition and distribution in the study area and the ecological value of different tree species; images were manually annotated through visual interpretation based on sample site survey data and forest resource inventory data.

[0014] Dataset construction: 200 pairs of original images and annotated images with a size of 4096×2730 were randomly selected from the tree species classification dataset and cropped to 1024×1024, and the data was divided into training set and test set in a ratio of 3:1.

[0015] (2) Building a benchmark dataset:

[0016] The benchmark datasets are the public ISPRS Vaihingen and ISPRS Potsdam datasets. The Vaihingen dataset contains 33 pairs of original images and annotated images with an average size of 2064×2494, and the ISPRS Potsdam dataset contains 38 pairs of original images and annotated images with a size of 6000×6000. They are all cropped to 1024×1024 and randomly divided into training and test sets in a ratio of 2:1. The dataset contains a total of 6 categories: Impervious surface, Building, Low vegetation, Tree, Car, and Clutter.

[0017] The model structure and principle in S2 are as follows:

[0018] The input image with a size of 1024×1024 after preprocessing is first processed by the Stem module, which includes the traditional 2D convolution layer (Conv2d), layer normalization (LN), and GELU function, which maintains the two-dimensional structure of the input image and generates a feature map with a size of 256×256×3.

[0019] Four stages are then designed to further process the acquired feature maps. The first stage contains two stacked VSS modules. The second and third stages consist of a downsampling module and multiple stacked VSS modules to generate feature maps of size 128×128×256 and 64×64×512, respectively. The downsampling module uses a traditional 2D convolutional layer combined with layer normalization to halve the spatial size of the feature map. The fourth stage contains an AutoDownsampling module and two stacked VSS modules. The AutoDownsampling module weights the output of the third stage using an importance score matrix and halves the spatial size of the feature map again to generate a feature map of size 32×32×1024. Finally, the classification result is obtained through a Softmax operation.

[0020] Including improved visual state space module and adaptive downsampling module;

[0021] The VSS module is the visual state space module, which includes two branches. Branch 1 contains the Linear Layer and the SiLU activation function; Branch 2 contains the LayerNormalization, the SiLU activation function, the Depth-WiseConvolution, the 2D selective scanning module, and the LayerNormalization. The outputs of the two branches are multiplied element by element, and then added to the original input to obtain the final output of the VSS module.

[0022] In the VSS module, the core module SS2D consists of the CSM (cross scanning module) and the S6 module; the CSM divides the image into sequences in the row and column directions, and traverses the spatial domain along these sequences to integrate pixel information from different directions. These sequences are then linearly transformed and discretized through the S6 module, so that the model retains the global receptive field while maintaining linear computational complexity, and finally the processed sequences are merged to output the complete image.

[0023] The S6 module is based on a state space model in the continuous domain, expressed as a linear ordinary differential equation:

[0024] h′(t)=Ah(t)+Bx(t)

[0025] y(t)=Ch(t)+Dx(t)

[0026] Among them, A, B, C and D are the state transfer matrix, control input matrix, output matrix and direct transmission matrix respectively. h(t) represents the internal state of the system at time t, h′(t) is the derivative of the state change, x(t) and y(t) represent the input and output respectively.

[0027] The above continuous system state equation is converted into a discrete equation to adapt it to the calculation framework. The discrete form of the above state space model after discretization by the second-order Runge-Kutta method is as follows:

[0028]

[0029] In the above equation, represents the discrete system matrix, represents the discrete input matrix, $\overline{C}$ represents the output matrix, represents the direct transfer matrix.

[0030] The AutoDownsampling module is the adaptive downsampling module:

[0031] This module dynamically adjusts the sampling step size according to the importance of the Token in the feature map to retain important information. The importance score s of each Token i Calculated by the following formula:

[0032] s i =αp i +βl i

[0033] Among them, p i is the grid prior score, l iis the feature learning score calculated by the Sigmoid function, and α and β are adjustable hyperparameters. The sampling step size is dynamically allocated according to the average distance between each Token and its K nearest neighbors. Finally, the importance score of each Token forms a weight matrix S, which is multiplied element by element with the original feature map. After convolution and layer normalization, the downsampled output is generated.

[0034] This paper introduces a UAV RGB tree species classification dataset to verify the effectiveness of single-phase UAV RGB data in achieving multi-tree species classification in complex heterogeneous forests, reducing the dependence of tree species classification tasks on hyperspectral data.

[0035] The present invention proposes the RSVMamba model, which realizes the tree species classification based on single-phase UAV RGB data in complex forest environments, and the classification accuracy of 8 tree species, dead trees and other broad-leaved trees reaches 84.28%.

[0036] The present invention bridges the gap between the cost and accuracy of tree species classification, providing a more cost-effective and time-flexible solution to the problem.

[0037] The present invention can provide data support for fields such as sustainable forest management and monitoring, precise ecological protection and biodiversity research. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is the geographical location of the study area of ​​the embodiment;

[0039] Figure 2 is a complete drone RGB image of the study area of ​​the embodiment;

[0040] Figure 3 is a digital surface model of the study area of ​​the embodiment;

[0041] Figure 4 is a sample plot distribution map of an embodiment;

[0042] Figure 5 It is the RSVMamba network structure diagram of the present invention;

[0043] Figure 6 It is the structural diagram of the SS2D module of the present invention;

[0044] Figure 7 It is an implementation flow chart of the present invention;

[0045] Figure 8 is a visualization diagram of the tree species classification data set result of the embodiment;

[0046] Fig. 9 It is a visualization diagram of the data set results of the embodiment. DETAILED DESCRIPTION

[0047] The specific technical solution of the present invention is explained in conjunction with embodiments.

[0048] Tree species classification is realized based on RSVMamba and UAV RGB data, including: S1, constructing a tree species classification dataset; S2, constructing a tree species classification model RSVMamba, and realizing effective classification based on the constructed dataset on the model.

[0049] The dataset construction and processing in S1 includes the following sub-steps:

[0050] (1) Constructing a tree species classification dataset:

[0051] Determination of the study area: Based on the detailed information of the 2020 Qimen County forest resources inventory data, including forest coverage, forest types, tree species composition, forest age structure and forest land quality, the scope of the study area was determined, such as Figure 1 and Figure 2 The terrain in this area is mainly mountainous and hilly, with large undulations, such as Figure 3 As shown, different terrain conditions result in rich and diverse distribution of tree species, covering a variety of forest resources.

[0052] RGB data collection: The collection of drone RGB images was carried out under windless and clear weather conditions on October 18 and 19, 2022. It was completed by a DJI Matrice 300RTK drone equipped with a Zenmuse P1 high-resolution RGB camera, which has 45 million pixels, a flight altitude of approximately 490m, a flight speed of 13m / s, a ground resolution of 6.15cm / pixel, and a spatial resolution of 0.05m.

[0053] Sample plot data collection: The specific distribution of the sample plots for sampling survey is as follows: Figure 4 As shown, a total of 69 sample plots were set up in Qimen County, each of which was 25.82m×25.82m in size, and the survey factors included the longitude and latitude of the center and corner points of the sample plot, topography, soil, vegetation type, canopy density, dominant tree species, tree species type and average tree height, so as to ensure the accuracy of the results of the present invention.

[0054] Dataset preprocessing: ArcGIS 10.6 was used to stitch and crop the original RGB images, and images containing a large number of non-forest areas were removed. Subsequently, 11 classification categories were determined based on the tree species composition and distribution in the study area and the ecological value of different tree species: Pi.s (slash pine), Cu.l (fir), Qu.s (Quercus), Li.f (Liquidambar formosana), Eu.g (Eucalyptus globulus), Ph.e (bamboo), Pa.a (Paulownia), Ca.s (tea tree), and De.t (dead tree), Ot.b (other broad-leaved trees) and Ba.d (background). Images were manually annotated through visual interpretation based on sample site survey data and forest resource inventory data. To ensure the accuracy of the annotation, the boundary definition and category determination of each category were cross-checked by at least one other calibrator.

[0055] Dataset construction: 200 pairs of original images and annotated images with a size of 4096×2730 were randomly selected from the tree species classification dataset and cropped to 1024×1024, and the data was divided into training set and test set in a ratio of 3:1.

[0056] (2) Building a benchmark dataset:

[0057] The benchmark datasets are the public ISPRS Vaihingen and ISPRS Potsdam datasets. The Vaihingen dataset contains 33 pairs of original images and annotated images with an average size of 2064×2494, and the ISPRS Potsdam dataset contains 38 pairs of original images and annotated images with a size of 6000×6000. They are all cropped to 1024×1024 and randomly divided into training and test sets in a ratio of 2:1. The dataset contains a total of 6 categories: Impervious surface, Building, Low vegetation, Tree, Car, and Clutter.

[0058] The model structure and principle in S2 are as follows:

[0059] A novel RSVMamba neural network is proposed to process RGB data acquired by UAVs and realize tree species classification. The method introduces an improved VSS (Visual State Space) module and AutoDownsampling (Adaptive Downsampling) module in its design.

[0060] (1) VSS (Visual State Space) module:

[0061] The specific structure of the VSS module is as follows Figure 5As shown in (a), it includes two branches. Branch 1 contains Linear Layer and SiLU activation function; Branch 2 contains Layer Normalization, SiLU activation function, Depth-Wise Convolution, SS2D (2D Selective Scan) module and LayerNormalization. The outputs of the two branches are element-wise multiplied and then added to the original input to obtain the final output of the VSS module. In the VSS module, the core module SS2D consists of CSM (cross scan module) and S6 module, as shown in Figure 6 As shown. CSM divides the image into sequences in the row and column directions and traverses the spatial domain along these sequences to integrate pixel information from different directions. These sequences are then linearly transformed and discretized through the S6 module, so that the model maintains linear computational complexity while retaining the global receptive field, and finally merges the processed sequences to output the complete image. The S6 module is based on the state space model of the continuous domain, usually expressed as a linear ordinary differential equation:

[0062] h′(t)=Ah(t)+Bx(t)

[0063] y(t)=Ch(t)+Dx(t)

[0064] Wherein, A, B, C and D are the state transfer matrix, control input matrix, output matrix and direct transmission matrix respectively. h(t) represents the internal state of the system at time t, h′(t) is the derivative of the state change, and x(t) and y(t) represent the input and output respectively. Since computers process data based on discrete time steps, it is necessary to convert the above-mentioned continuous system state equation into a discrete equation to adapt it to the calculation framework. The present invention uses the second-order Runge Kutta method for discretization. The second-order Runge Kutta method calculates the intermediate state in each time step, so as to more accurately approximate the continuous dynamics of the system, reduce local truncation errors and improve numerical stability, better simulate nonlinear dynamic changes, and improve the accuracy of the model. The discrete form of the above-mentioned state space model after discretization by the second-order Runge Kutta method is as follows:

[0065]

[0066] In the above equation, represents the discrete system matrix, represents the discrete input matrix, $\overline{C}$ represents the output matrix, represents the direct transfer matrix.

[0067] (2) AutoDownsampling (Adaptive Downsampling) module:

[0068] In order to improve the effect of traditional grid downsampling, the present invention introduces the AutoDownsampling module. This module dynamically adjusts the sampling step size according to the importance of the Token in the feature map to retain important information. The importance score s of each Token i Calculated by the following formula:

[0069] s i =αp i +βl i

[0070] Among them, p i is the grid prior score, l i is the feature learning score calculated by the Sigmoid function, and α and β are adjustable hyperparameters. To ensure the robustness of the downsampling result, the present invention dynamically allocates the sampling step size according to the average distance between each Token and its K nearest neighbors. This method can reduce the impact of noise and outliers, thereby improving the stability of the model. Finally, the importance score of each Token forms a weight matrix S, which is multiplied element by element with the original feature map. After convolution and layer normalization, the downsampled output is generated. The specific structure is as follows: Figure 5 As shown in (b).

[0071] (3) RSVMamba overall structure:

[0072] The overall structure of RSVMamba is as follows Figure 5 After preprocessing, the input image with a size of 1024×1024 is first processed by the Stem module (the module structure is shown in Figure 5 As shown in (a), it contains the traditional 2D convolution layer (Conv2d), layer normalization (LN), GELU function, maintains the two-dimensional structure of the input image, and generates a feature map of size 256×256×3. Four stages are then designed to further process the acquired feature map. The first stage contains two stacked VSS modules. The second and third stages consist of a downsampling module and multiple stacked VSS modules, generating feature maps of size 128×128×256 and 64×64×512, respectively. The downsampling module is shown in Figure 5 As shown in (a), the traditional 2D convolutional layer is used, combined with layer normalization, to halve the spatial size of the feature map. The fourth stage contains an AutoDownsampling module mentioned above and two stacked VSS modules. The AutoDownsampling module uses the importance score matrix to weight the output of the third stage and halve the spatial size of the feature map again to generate a feature map of size 32×32×1024. Finally, the classification result is obtained through the Softmax operation.

[0073] In this example, the tree species classification dataset constructed by S1 and the RSVMamba model proposed by S2 are used to classify the tree species in the study area. The training is performed using NVIDIA A100 GPU based on the deep learning framework Pytorch, with 8000 training iterations and a batch size of 16. The specific process is as follows Figure 7 As shown in the figure, the training data of size 1024×1024 is input into the RSVMamba model for training. After the data is input into the RSVMamba model, the Steam module is used to obtain a 256×256×3 feature map, and then the feature extraction is successively performed through Stage1, Stage2, Stage3, and Stage4 to obtain a feature map of size 32×32×1024. Finally, the classification result is obtained through the Softmax operation. The model is predicted on the test set, and the three common evaluation indicators of semantic segmentation, mIoU, AF, and OA, are used to measure the classification performance of the model on the test set. The same data set and evaluation indicators are used to conduct experiments on a series of CNN, Transformer, and Mamba-based models for evaluation and comparison.

[0074] In addition, in order to verify the generalization and superiority of the model, the same experiments were performed on the benchmark datasets ISPRS Vaihingen and ISPRS Potsdam using the same parameters and settings.

[0075] Table 1: Classification results of tree species dataset

[0076]

[0077] The present invention realizes the classification of 8 tree species, dead trees, other broad leaves and background on the tree species classification dataset, and tests the generalization and effectiveness of the model on the benchmark datasets ISPRS Vaihingen and ISPRS Potsdam. The classification results on the tree species classification dataset are shown in Table 1. Based on the three common evaluation indicators mIoU, AF and OA, the classification accuracy of the model proposed in this patent reached 71.12%, 81.45% and 84.28% respectively, verifying the effectiveness of single-phase UAV RGB data in realizing tree species classification in complex heterogeneous forests.

[0078] Compared with VMamba, which has the best performance among the comparison models, the model of the present invention has improved by 0.94%, 1.38% and 1.05% in these three indicators respectively. In addition, the mIoU, AF and OA on the ISPRS Vaihingen and ISPRS Potsdam datasets reached 84.14%, 92.76%, 92.13% and 87.49%, 92.96%, 91.87% respectively, and the classification accuracy is higher than that of other methods, which verifies the superiority of the method of the present invention. The classification results of all models on the validation set are shown in Table 1, and the results on the tree species classification dataset and the benchmark dataset are visualized. Figure 8 and Fig. 9 shown.

Claims

1. Tree species classification based on RSVMamba and drone RGB data, characterized by: include: S1. The UAV RGB tree species classification dataset was introduced to construct a tree species classification dataset; S2. Construct a tree species classification model RSVMamba, combine it with the improved VSS module and AutoDownsampling module, and implement effective classification based on the constructed dataset on this model.

2. The method for realizing tree species classification based on RSVMamba and drone RGB data according to claim 1, characterized in that: The dataset construction and processing in S1 includes the following sub-steps: (1) Constructing a tree species classification dataset: Collect detailed information including forest cover, forest type, tree species composition, stand age structure and forest land quality to define the scope of the study area; Including RGB data collection and sample site data collection; Dataset preprocessing: splicing and cropping of original RGB images, removing images containing non-forest areas; then, determining tree species classification categories based on tree species composition and distribution in the study area and the ecological value of different tree species; manually annotating images through visual interpretation based on sample site survey data and forest resource inventory data; Dataset construction: 200 pairs of original images and annotated images with a size of 4096×2730 were randomly selected from the tree species classification dataset and cropped to 1024×1024, and the data was divided into training set and test set in a ratio of 3:1; (2) Building a benchmark dataset: The benchmark datasets are the publicly available ISPRS Vaihingen and ISPRS Potsdam datasets. The Vaihingen dataset contains 33 pairs of original images and annotated images with an average size of 2064×2494, and the ISPRS Potsdam dataset contains 38 pairs of original images and annotated images with a size of 6000×6000. They are all cropped to 1024×1024 and randomly divided into training and test sets in a ratio of 2:

1. The dataset contains a total of 6 categories: impervious surfaces, buildings, low vegetation, trees, cars, and debris.

3. The method for realizing tree species classification based on RSVMamba and drone RGB data according to claim 1, characterized in that: The specific method in S2 is: The input image with a size of 1024×1024 after preprocessing is first processed by the Stem module, which includes the traditional 2D convolution layer (Conv2d), layer normalization (LN), and GELU function, which maintains the two-dimensional structure of the input image and generates a feature map with a size of 256×256×3; Then four stages are designed to further process the acquired feature maps. The first stage contains two stacked VSS modules; The second and third stages consist of a downsampling module and multiple stacked VSS modules, generating feature maps of size 128×128×256 and 64×64×512 respectively; the downsampling module uses a traditional 2D convolutional layer combined with layer normalization to halve the spatial size of the feature map; The fourth stage contains an AutoDownsampling module and two stacked VSS modules; the AutoDownsampling module uses the importance score matrix to weight the output of the third stage and halve the spatial size of the feature map again to generate a feature map of size 32×32×1024; finally, the classification result is obtained through the Softmax operation.

4. The method for realizing tree species classification based on RSVMamba and drone RGB data according to claim 3, characterized in that: The VSS module, i.e., the visual state space module, comprises two branches, wherein branch 1 comprises a Linear Layer and a SiLU activation function; branch 2 comprises a LayerNormalization, i.e., a layer normalization, a SiLU activation function, a Depth-WiseConvolution, i.e., a 2D selective scanning module, and a Layer Normalization; the outputs of the two branches are element-wise multiplied, and then added to the original input to obtain the final output of the VSS module; In the VSS module, the core module SS2D consists of the CSM (cross scan module) and the S6 module; the CSM divides the image into sequences in the row and column directions and traverses the spatial domain along these sequences to integrate pixel information from different directions; These sequences are then linearly transformed and discretized through the S6 module, so that the model retains the global receptive field while maintaining linear computational complexity. Finally, the processed sequences are merged to output the complete image. The S6 module is based on a state space model in the continuous domain, expressed as a linear ordinary differential equation: h′(t)=Ah(t)+Bx(t) y(t)=Ch(t)+Dx(t) Where A, B, C and D are the state transfer matrix, control input matrix, output matrix and direct transmission matrix respectively; h(t) represents the internal state of the system at time t, h′(t) is the derivative of the state change, x(t) and y(t) represent the input and output respectively; The above continuous system state equation is converted into a discrete equation to adapt it to the calculation framework; the discrete form of the above state space model after discretization by the second-order Runge-Kutta method is as follows: In the above equation, represents the discrete system matrix, represents the discrete input matrix, $\overline{C}$ represents the output matrix, represents the direct transfer matrix.

5. The method for realizing tree species classification based on RSVMamba and drone RGB data according to claim 3, characterized in that: The AutoDownsampling module is an adaptive downsampling module: This module dynamically adjusts the sampling step size according to the importance of the Token in the feature map to retain important information; the importance score s of each Token i Calculated by the following formula: s i =αp i +βl i Among them, p i is the grid prior score, l i is the feature learning score calculated by the Sigmoid function, α and β are adjustable hyperparameters; the sampling step is dynamically allocated according to the average distance between each Token and its K nearest neighbors; finally, the importance score of each Token forms a weight matrix S, which is multiplied element by element with the original feature map, and after convolution and layer normalization, the downsampled output is generated.

Images