A deep learning method for single tree crown segmentation based on multimodal data

Abstract

A deep learning method for single-tree crown segmentation based on multimodal data belongs to the field of artificial intelligence technology in forestry research. Based on high-resolution GF-2 imagery, LiDAR point cloud features, and ultra-high-resolution RGB multimodal data, and integrated with a vectorized crown post-processing strategy, a deep learning model, the Multimodal Crown Segmentation Fusion Network (MTCDFNet), is proposed. By integrating forest structure, spectral, and texture features from multimodal remote sensing data, this method accurately segments tree crowns in multi-canopy, multi-species subtropical forests.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology in forestry science research and relates to a deep learning method for single tree crown segmentation based on multimodal data. Background Art

[0002] Accurate spatial information about tree crowns is crucial for forest management measures and forest health monitoring and assessment. The quality of individual tree crown segmentation is closely related to the estimation of basic forest parameters and tree species classification in individual-scale forest inventory. However, current crown segmentation methods have room for improvement in both applied data and model algorithms.

[0003] In terms of applied data: Traditional remote sensing-based methods rely primarily on single-modality data, namely CHM or RGB images, focusing on single features such as texture, spectrum, or forest structure, while ignoring the fusion of multimodal data features. At the same time, the complexity of forest characteristics, namely stand density, crown characteristics, and background environment, poses challenges to single-modality data methods, especially in subtropical forests dominated by multiple canopies and multiple tree species. For example, (i) closely adjacent or overlapping crowns, as well as variations in crown size, shape, and color, may affect segmentation accuracy, especially when trees have similar crown appearances; (ii) environmental background factors, such as the presence of shrubs, grasses, or other low-lying vegetation, may also affect the performance of the trained model, resulting in false positive predictions; and (iii) limitations in spatial or spectral resolution may hinder the effectiveness of deep learning models in identifying small crowns or distinguishing variability within crown types.

[0004] In terms of model algorithms: the main segmentation algorithms or models currently used, such as valley-following, region-growing and watershed segmentation, require manual setting of thresholds, and the parameters have specific scenario limitations, making it difficult to achieve accurate crown segmentation in complex subtropical forests; the currently widely used convolutional neural network (CNN) has the performance of extracting representative and distinctive features from the original input. It is a robust and effective segmentation method for describing crown UAV-RGB images, but the segmentation results of this method are prone to produce discontinuous and open crown boundaries, which to a certain extent affects the accuracy of crown segmentation. Summary of the Invention

[0005] In response to the problems in the prior art, the present invention provides a deep learning method for single tree crown segmentation based on multimodal data.

[0006] A deep learning method for single tree crown segmentation based on multimodal data includes the following steps:

[0007] Based on high-resolution GF-2 imagery, LiDAR point cloud features, and ultra-high-resolution RGB multimodal data, and integrating vectorized crown post-processing strategies, a deep learning model, the Multimodal Crown Segmentation Fusion Network (MTCDFNet), was proposed. By integrating the forest structure, spectral, and texture characteristics of multimodal remote sensing data, accurate crown segmentation in multi-canopy, multi-species subtropical forests can be achieved.

[0008] The multimodal crown segmentation fusion network adopts an encoder-decoder architecture. The encoder is used to extract abstract and distinguishable features, and the decoder uses the extracted features for accurate crown prediction. The multimodal crown segmentation fusion network forms a three-branch structure in the encoder: one is used for texture feature extraction of ultra-high resolution RGB (UHR, 0.0623m) images; one is used for spectral feature extraction of high-resolution (1m) GF-2 images; and one is used for forest structure feature extraction of LiDAR point cloud data. The features extracted by the three branches are complementary, and the multimodal crown segmentation fusion network constructs a three-branch feature The TBFF module is used to integrate the multi-level features extracted by the three branches into a joint feature space. The TBFF module generates channel and spatial attention maps for two consecutive sub-modules, namely the channel attention module and the spatial attention module. The context-aware module (CAM) includes a convolutional layer with a kernel size of 1 to extract specific features from the input data, asymmetric convolutions with expansions of 1, 3, and 5, which allow the convolutional layer to capture more contextual information over a larger area without excessively increasing the number of parameters, and a global average pooling layer to provide a holistic view of the features by capturing the global context of the feature map. A gated attention mechanism is used as the multi-level fusion (MLF) module. The MLF module uses high-level features to guide low-level features, highlighting important spatial details and filtering out irrelevant noise. It also considers the differences in the appearance and intrinsic properties of the crown at different developmental stages. A recurrent residual block (RR Conv) with a recursive residual structure is introduced to integrate residual connections and recursive connections within the CNN to improve the network's ability to capture temporal dependencies across different stages while alleviating the gradient vanishing caused by networks with too many layers.

[0009] The three branch structures include: RGB-based branch structure: CNN efficiently extracts spatial features from UHR images, uses ResNet-50 to encode UHR images and outputs multi-layer spatial features, excluding the last pooling layer and fully connected layer; LiDAR or spectral-based branch structure: CHM images and other feature maps or spectral images obtained by lidar are used for tree classification and segmentation, providing detailed forest structure or spectral information, and CNN with four different filters (i.e., 1×1, 3×3, 5×5, and 7×7) is used for data fusion and semantic segmentation.

[0010] The Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) extracts various types of features from RGB, GF-2 and LiDAR data, and combines the crown edge and shape, tree species type, crown height, density and vertical distribution complementary information to achieve more accurate crown segmentation.

[0011] The channel attention module and the spatial attention module are integrated into the TBFF fusion module to assign appropriate weights to multi-channel input features, focusing on the most relevant features and suppressing irrelevant noise. The channel attention module weights different channels of the input feature map and emphasizes the key channels for crown segmentation by multiplying them with the channel attention weights. The spatial attention module assigns weights to different spatial positions within the feature map and highlights the most relevant spatial regions for segmenting the crown by multiplying the channel-weighted feature map with the spatial attention weights. It learns "which features are important" through the channel attention mechanism and then uses the spatial attention mechanism to focus on "where the feature map is important" during training.

[0012] The advantages of the present invention are: the multimodal crown segmentation fusion network (MTCDFNet) realizes accurate segmentation of tree crowns in multi-canopy, multi-species subtropical forests by integrating forest structure, spectrum and texture characteristics of multimodal remote sensing data. Its accuracy fully meets the application requirements of large-area, single-tree scale forest resource inventory.

[0013] As shown in Figure 7(a) and Figure 7(b), to determine whether the multimodal crown segmentation fusion network (MTCDFNet) effectively obtains crown features from UHR, LiDAR and GF-2 data, 20 crowns were randomly selected from 6 sub-regions, and their completeness and correctness, as well as over-segmentation and under-segmentation errors, were visualized at the individual boundary level and object level.

[0014] The implementation results show that these 20 trees can effectively segment various sparse and dense scenes. At the boundary and object scales, the F1 score F edgeThe Completeness and Correctness indicators are mainly distributed above 0.750, while at the object scale, the total segmentation error TS, over-segmentation error OS and under-segmentation error US values ​​are mostly lower than 0.100.

[0015] In addition, the Z-score indicator was selected to describe the distance between each crown and the mean value, in units of standard deviation. The results showed that P2, P13 and P20 were small trees under the forest surrounded by large trees, and their Z-score was at the F1 score F of the border. edge , Completeness and Correctness index sizes are lower than -1.500, while the total segmentation error TS, over-segmentation error OS and under-segmentation error US index sizes are higher than 1.200. For the total segmentation error TS and under-segmentation error US, the Z-score value of P16 exceeds 1.500, and the F1 score F edge The Completeness and Correctness indicators range from 0.1 to 0.80. For the total segmentation error TS and under-segmentation error US, the Z-score size of P20 is distributed between -0.229 and 0.303, while the F1 score F edge , Completeness and Correctness indicators are lower than -1.100. The F1 score of other diagnostic trees at the boundary is edge , Completeness, Correctness, Total Segmentation Error TS, Over-segmentation Error OS and Under-segmentation Error US indicators have Z-scores ranging from -1.000 to 1.000. From the above statistical results, it is observed that larger trees in the upper forest layer can be effectively identified. However, smaller trees in dense conditions under the forest floor are not easily discovered or depicted, most likely because they are mistakenly classified as part of the surrounding or adjacent larger trees. Overall, the implementation results show that the Multimodal Crown Segmentation Fusion Network (MTCDFNet) model can effectively learn multi-channel features of various forest scenes, making it very effective in segmenting the crown. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. As shown in the figure:

[0017] Figure 1(A) Multimodal tree crown segmentation fusion network MTCDFNet of the present invention.

[0018] FIG1(B) shows the context-aware module CAM of the present invention.

[0019] FIG1(C) shows the multi-level fusion module MLF of the present invention.

[0020] FIG1(D) shows the cyclic residual convolution module RR_Conv of the present invention.

[0021] FIG2( a ) shows the basic structure of the LiDAR encoder of the present invention.

[0022] FIG2( b ) shows the multi-level downsampling module structure of the present invention.

[0023] Figure 3 This is the main structure of the three-branch feature fusion (TBFF) module of the present invention. Xspectral, Xlidar, and Xrgb represent the multi-modal features extracted by the spectral branch, the lidar branch, and the RGB branch, respectively.

[0024] FIG4( a ) is a diagram of a multi-channel data fusion strategy (mid-term fusion strategy) of the present invention.

[0025] FIG4( b ) is a diagram of the multi-channel data fusion strategy (early fusion strategy) of the present invention.

[0026] FIG4( c ) is a diagram of the multi-channel data fusion strategy (late fusion strategy) of the present invention.

[0027] Figure 5(a) Matthew correlation coefficient of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0028] Figure 5(b) F1 score of the crown of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0029] Figure 5(c) shows the F1 scores of the boundary of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0030] Figure 5(d) shows the total segmentation error of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0031] Figure 5(e) shows the accuracy of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0032] Figure 5(f) Recall rate of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0033] Figure 5(g) shows the completeness of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNET models of the present invention.

[0034] Figure 5(h) The accuracy of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0035] Figure 5(i) Over-segmentation error of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0036] Figure 5(j) Under-segmentation error of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0037] Figure 6 (S1-S4) represents sub-regions 1-4, used to illustrate the detailed errors of drawing tree crowns using MTCDFNet, ResUNet, RAUNet, AUNet, and UNet.

[0038] Figure 7(a) One of the 20 tree crown segmentation accuracy verifications under different sparse and dense scenarios in the implementation area.

[0039] Figure 7(b) is the second verification of the segmentation accuracy of 20 tree crowns selected under different sparse and dense scenarios in the implementation area. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] Example 1: As shown in Figure 1, Figure 2, Figure 3 , Figure 4, Figure 5 and Figure 6 As shown in the figure, a deep learning method for single tree crown segmentation based on multimodal data mainly involves the theory of forest biological characteristics and semantic segmentation technology in computer science. It makes up for the limitations of single modality data in current crown segmentation, provides support for the application of crown segmentation in subtropical forests with multiple canopies and multiple tree species, and provides model support for large-scale, automated, and high-precision forest resource surveys.

[0042] A deep learning method for single tree crown segmentation based on multimodal data is proposed. Based on high-resolution GF-2 imagery, LiDAR point cloud features, and ultra-high-resolution RGB multimodal data, a vectorized crown post-processing strategy is integrated to propose a deep learning model, namely the Multimodal Crown Segmentation Fusion Network (MTCDFNet). By integrating the forest structure, spectral, and texture characteristics of multimodal remote sensing data, it can achieve accurate crown segmentation in multi-canopy, multi-species subtropical forests.

[0043] Deep learning model: Fusion of GF-2, LiDAR, and ultra-high-resolution RGB data, as shown below:

[0044] The multimodal crown segmentation fusion network adopts an encoder-decoder architecture. The encoder is used to extract abstract and distinguishable features, and the decoder uses the extracted features for accurate crown prediction.

[0045] The multimodal crown segmentation fusion network mainly forms three branch structures in the encoder:

[0046] One is for texture and other feature extraction of ultra-high resolution RGB (UHR, 0.0623m) images;

[0047] One is for spectral feature extraction of high-resolution (1m) GF-2 images;

[0048] A forest structure feature extraction for LiDAR point cloud data;

[0049] The features extracted by the three branches are complementary.

[0050] Subsequently, the multimodal crown segmentation fusion network constructs a three-branch feature fusion (TBFF) module to integrate the multi-level features extracted by the three branches into a joint feature space to improve the discriminative feature ability of crown recognition.

[0051] The three-branch feature fusion (TBFF) module generates channel and spatial attention maps from two consecutive sub-modules, namely channel attention and spatial attention modules.

[0052] Considering the diversity of crown shapes, sizes, and colors in high-density, multi-canopy mixed forest systems, obtaining multi-scale discriminative features of the three branches is the key to accurately extracting the crown.

[0053] Previous studies have demonstrated that the ASPP module can effectively capture contextual information at multiple scales, and the enlarged receptive field is beneficial for detecting and segmenting large objects.

[0054] The context-aware module (CAM) (Figure 1(B)) consists of a convolutional layer with a kernel size of 1 that extracts specific features from the input data, asymmetric convolutions with dilations of 1, 3, and 5 that allow the convolutional layer to capture more contextual information over a larger area without excessively increasing the number of parameters, and a global average pooling layer that provides a holistic view of the features by capturing the global context of the feature map.

[0055] Deeper layers of the encoder are able to capture high-level features, which will help identify more discriminative and representative features of the tree crown. However, the downsampling processes involved in building deeper networks, such as pooling and strided convolution, may lead to the loss of fine details and spatial information. On the contrary, although the low-level features (i.e., edges, gradients, textures, and colors) extracted from shallow layers lack sufficient abstraction, maintaining better resolution and spatial details from deep layers will be beneficial for accurately predicting the pixel-level position of the tree crown.

[0056] In order to enhance the information integration of low-level and high-level features or abstractions, a gated attention mechanism is adopted as a multi-level fusion (MLF) module ( Figure 1C ).

[0057] The multi-level fusion (MLF) module utilizes high-level features to guide low-level features, highlighting important spatial details while effectively filtering out irrelevant noise.

[0058] Considering the differences in appearance and intrinsic properties of the crown at different developmental stages, a recurrent residual block (RR Conv) with a recursive residual structure is introduced (Figure 1(D)) to integrate residual connections and recursive connections within the CNN, thereby improving the network's ability to capture temporal dependencies across different stages and alleviating the gradient vanishing problem caused by networks with too many layers.

[0059] The three branch structures are as follows:

[0060] 1. RGB-based branch structure:

[0061] Due to its ultra-high spatial resolution and low cost, UHR images provide rich spatial information, which helps to understand the background and interactions between different tree crowns. Therefore, they are increasingly used in tree crown segmentation tasks.

[0062] Previous studies have demonstrated that CNNs can efficiently extract spatial features from UHR images. However, CNNs with shallow structures and small receptive fields primarily focus on local details and are more sensitive to noise and variation. To address this issue, He et al. (2016) proposed a residual module that increases the number of CNN layers, achieving better performance in object segmentation. However, deeper CNNs typically have more convolutional layers and neurons, resulting in more parameters that need to be learned during training. To strike a balance between model depth and computational efficiency, this branched structure uses ResNet-50 to encode UHR images and output multi-layer spatial features, excluding the final pooling layer and fully connected layer.

[0063] Therefore, this branch can obtain the global semantic information of the crown through deep features, and obtain the local details of the crown through shallow features.

[0064] 2. Branched structures based on LiDAR or spectroscopy:

[0065] Like UHR images, lidar-derived CHM images and other feature maps or spectral images are also used for tree classification and segmentation, providing detailed forest structure or spectral information. Currently, multi-filter convolutional neural networks have been effectively used to analyze spectral and CHM images. Because traditional CNNs use fixed filter sizes and have a limited field of view, ensemble methods that incorporate multi-scale techniques have proven to offer significant advantages in segmentation tasks.

[0066] Therefore, this branch introduces four CNNs with different filters (i.e., 1×1, 3×3, 5×5, and 7×7) for data fusion and semantic segmentation.

[0067] In addition, a DownConv sequence module is used to learn multi-layer spatial features from the fused multi-filter CNN output features extracted from the Spectral or LiDAR encoder. The DownConv module has three Conv2D layers, each of which is accompanied by a BatchNorm layer to stabilize the training process.

[0068] In addition, the first Conv2D layer in the DownConv block uses a stride of 2 to halve the spatial resolution of the input feature map. To address the spatial resolution inconsistency between GF-2 images, LiDAR-derived feature maps, and UHR data, the feature maps obtained from the LiDAR and Spectral branches are upsampled using the nearest neighbor interpolation method to ensure the same resolution as the RGB branch-derived feature maps. The LiDAR encoder and the spectral encoder have the same structure.

[0069] In Figure 1(A), Figure 1(B), Figure 1(C), and Figure 1(D), Res-1 to Res-4 represent the four residual blocks of ReNet50.

[0070] TBFF, MLF, CAM, and RR Conv represent the three-branch fusion module, multi-level fusion module, context-aware module, and cyclic residual convolution block, respectively. The number of cycles indicates the number of cycles of the rectangle highlighted in light blue.

[0071] Figure 2(a) shows the LiDAR encoder and Figure 2(b) shows the multi-level downsampling module structure.

[0072] The LiDAR encoder and the spectral encoder have the same structure. X0 to XT represent the input images of the LiDAR-derived CHM image and other feature maps. Similarly, for the spectral encoder, X0 to XT represent the input images of the GF-2-based image.

[0073] Figure 3 The main structure of the three-branch feature fusion (TBFF) module.

[0074] Xspectral, Xlidar, and Xrgb represent the multimodal features extracted by the spectral branch, lidar branch, and RGB branch, respectively. C and X S denote the input feature maps for channel and spatial attention modules, respectively.

[0075] Three branch fusion modules:

[0076] The three branches of the Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) primarily extract various types of features from RGB, GF-2, and LiDAR data. Subsequently, complementary information, such as crown edges and shapes, tree species, crown height, density, and vertical distribution, is combined to achieve more accurate crown segmentation with greater resilience to noise and variations.

[0077] However, studies have shown that the importance of features varies depending on the canopy's properties. Specifically, accurately segmenting canopies with unique and complex spatial characteristics often relies heavily on UHR imagery. Conversely, distinguishing canopies with similar textural features but different spectral or vertical characteristics requires GF-2 imagery or LiDAR-based feature maps.

[0078] Therefore, previous multimodal feature fusion strategies ignore the importance of different features in different crown attributes and introduce unimportant feature noise into the segmentation model, which potentially affects the performance of crown segmentation.

[0079] The present invention includes channel attention and spatial attention modules ( Figure 3) are sequentially integrated into the TBFF fusion module to assign appropriate weights to multi-channel input features, enabling the model to focus on the most relevant features and suppress irrelevant noise. Specifically, the channel attention module assigns weights to different channels of the input feature map and emphasizes the key channels for crown segmentation by multiplying them with the channel attention weights.

[0080] At the same time, the spatial attention module assigns weights to different spatial locations within the feature map, highlighting the most relevant spatial regions for segmenting the crown by multiplying the channel-weighted feature map with the spatial attention weights. In other words, the module can learn "which features are important" through the channel attention mechanism and subsequently use the spatial attention mechanism to focus on "where the feature map is important" during training.

[0081] Example 2: As shown in Figure 1, Figure 2, Figure 3 , Figure 4, Figure 5 and Figure 6 As shown in the figure, a deep learning method for single tree crown segmentation based on multimodal data is proposed. The implementation area is a mixed Chinese fir forest in Shanxia Experimental Forest Farm in Jiangxi Province. The area has a high species diversity, among which Chinese fir (Cunninghamia Lanceolata), nanmu (Machilus Pauhoi) and Schima Superba are the main species, forming a multi-layer mixed forest with overlapping crowns, different properties and sizes.

[0082] Implementation Design:

[0083] Ablation experiments of the Multimodal Crown Division Fusion Network (MTCDFNet):

[0084] In order to evaluate the performance of Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) in extracting tree crowns using multimodal remote sensing data, the three-branch structure of Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) was divided into three single-branch networks.

[0085] (1) RTCNet, a network that learns texture features from RGB images.

[0086] (2) STCNet, a network for learning spectral features from GF-2 images.

[0087] (3) LTCNet, a network that learns structural features from LiDAR point cloud data.

[0088] Each network was trained separately using feature maps derived from RGB images, GF-2 images, and LiDAR point cloud data. The performance of these three networks was then compared with that of the Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) to evaluate the effectiveness of utilizing data from various models.

[0089] An effective fusion strategy that integrates the complementary information of multi-channel data is the key to improving the accuracy of crown segmentation. Several fusion strategies have been proposed, namely early fusion, mid-term fusion, and late fusion (Cai et al., 2023; Shi et al., 2021; Yu et al., 2023).

[0090] The multimodal tree crown segmentation fusion network (MTCDFNet) model proposed in this paper adopts an intermediate fusion strategy. It extracts features from RGB, spectral and LiDAR data respectively through a three-branch structure and fuses them using the TBFF module.

[0091] To evaluate the performance of the mid-term fusion strategy, early fusion and late fusion were performed for comparison ( Figure 6 ).

[0092] For the early fusion strategy, the GF-2 and LiDAR based feature maps are resampled to the same resolution RGB (0.0623m) and then combined into a single multi-band image.

[0093] For the late fusion strategy, RTCNet, STCNet, and LTCNet are trained on RGB, spectral, and LiDAR point cloud data, respectively, and the final prediction is produced by averaging the outputs of these models.

[0094] Comparative test with existing models:

[0095] To further evaluate the effectiveness of the Multimodal Crown Segmentation Fusion Network (MTCDFNet) in segmenting crowns, we compared it with four models currently widely used in object segmentation research: UNet (Guo et al., 2021; Ronneberger et al., 2015), AUNet (Yi et al., 2023), RAUNet (Cai et al., 2023), and ResUNet (Cao and Zhang, 2020; Diakogiannis et al., 2020). Among them, UNet is a popular U-shaped structure that is widely used in semantic segmentation tasks.

[0096] ResUNet and AUNet are both encoder-decoder models that combine residual blocks with cross-attention mechanisms and ASPP modules, respectively. Meanwhile, RAUNet uses gated attention modules and residual blocks to accurately detect objects with clear boundaries at the detailed level.

[0097] Migration experiments with different forest characteristics:

[0098] Crown size and shape are often influenced by forest characteristics such as stand density, leading to significant differences in crown properties between mixed forests and single-species plantations.

[0099] In order to evaluate the transferability and universality of the Multimodal Crown Division Fusion Network (MTCDFNet) between different forest types, the mixed fir forest in Jiangxi Shanxia Experimental Forest Farm (Area A) was selected as the original training area, and the pure fir plantation in Huangfengqiao State-owned Forest Farm (Area B) was selected as the target migration area for testing. Based on the percentage of adjacent crown overlap in the selected implementation area, Area B was divided into three forest levels: dense, medium, and sparse (b1, b2, b3). Among them, three key experiments were carried out:

[0100] Including M1: To evaluate the performance of the multimodal crown segmentation fusion network (MTCDFNet) in pure Chinese fir plantations with different forest densities, the multimodal crown segmentation fusion network (MTCDFNet) of pure Chinese fir plantations in migration area B was randomly initialized and trained using limited data from dense b1, medium b2, and sparse b3 forests respectively;

[0101] Including M2: To evaluate the generalization ability of the Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) on different forest types and densities, the Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) was pre-trained with data from a mixed Chinese fir forest in region A, and then the model was applied to dense b1, medium b2, and sparse b3 densities of a pure Chinese fir plantation in region B;

[0102] Including M3: In order to determine the improvement performance of the pre-trained model of the sampling combined with fine-tuning method on the prediction results, the multimodal crown division fusion network (MTCDFNet) was first pre-trained using data from the mixed Chinese fir forest in area A, and then fine-tuned in the dense b1, medium b2 and sparse b3 forests of the Chinese fir plantation pure forest in area B using limited data.

[0103] Implementation steps:

[0104] All models were trained for 2000 epochs to achieve convergence in ablation and comparative experiments. In order to speed up the training process, the Adam optimizer was used and the Beta value was set to 0.9% and 0.999. At the same time, in order to enhance the generalization and optimization capabilities of the model, a multi-step learning rate decay strategy was adopted and the initial learning rate was set to 1e-2. In addition, several data augmentation techniques were applied, including random horizontal and vertical flipping, random resizing, and random rotation of 45, 135, 225, and 315 degrees, which effectively expanded the training dataset. All experiments used the TensorFlow (2.8.0) architecture and adopted NVIDIA GeForce RTX 3070 (8GB VRAM), CUDA version 11.4, Intel R Core TM The system was operated on an i7-10700K (3.80GHz), 128GB of memory, and Ubuntu 18.04.6LTS.

[0105] Evaluation Metrics:

[0106] The performance of crown segmentation is evaluated using pixel-based metrics, including precision, recall, and the crown F1 score. Crown and Matthew's correlation coefficient (MCC) ( Diakogiannis et al., 2020 ; Shendryk et al., 2016 ; Waldner and Diakogiannis, 2020 ).

[0107] The calculation formulas for these four evaluation indicators are as follows:

[0108]

[0109]

[0110]

[0111]

[0112] Where Tp, Tn, Fp, and Fn represent the number of pixels correctly classified as crown, pixels classified as background, pixels misclassified as crown, and pixels misclassified as background, respectively. Precision corresponds to commission errors, while recall corresponds to omission errors. Precision and recall are combined by calculating the harmonic mean to assess the overall accuracy of crown delineation. The Matthew correlation coefficient (MCC) measures the consistency between the predicted and reference crowns, particularly when dealing with imbalanced data.

[0113] The edge-based method and the object-based method were used to evaluate the accuracy of crown segmentation at the boundary and object levels respectively. Three edge-based evaluation metrics were used: completeness (Com), correctness (Corr), and the F1 score of the boundary. edge To evaluate the accuracy of crown boundary extraction. Completeness quantifies the proportion of accurately detected reference crown boundaries, while Correctness evaluates the proportion of incorrectly detected boundaries in the predicted crown. The F1 score F edge The harmonic mean of completeness and correctness provides a robust measure of boundary delineation accuracy. The three evaluation metrics are calculated as follows:

[0114]

[0115]

[0116]

[0117] To further complete the object-scale crown evaluation, three object-based evaluation metrics are used: over-segmentation error (OS), under-segmentation error (US), and total segmentation error (TS) (Cai et al., 2023; Waldner and Diakogiannis, 2020; Yang et al., 2015). Over-segmentation error (OS) and under-segmentation error (US) quantify the errors of over-segmentation and under-segmentation, respectively. Total segmentation error (TS) provides an overall evaluation of these two segmentation errors, providing a check on boundary connectivity for generating closed boundaries of the crown. The calculation formulas for these three evaluation metrics are as follows:

[0118]

[0119]

[0120]

[0121] Where P represents a set of predicted tree crowns with n tree crowns {P1, P2, ..., P n}, and R represents a reference crown set with m crowns {R1, R2, ..., R m}. OS and US represent the errors of over-segmentation and under-segmentation respectively. I When overlapping with multiple reference crowns, P I The error is calculated as the weighted sum of over-segmentation and under-segmentation errors, determined by the area of ​​the overlapping region.

[0122] Implementation results:

[0123] Ablation experiment results of Multimodal Crown Division Fusion Network (MTCDFNet):

[0124] Table 1 shows the performance of different models in the crown extraction of the implementation area. Among the various models, the multimodal crown segmentation fusion network (MTCDFNet) shows superior performance in terms of Matthew correlation coefficient MCC, crown F1 score F Crown and the F1 score of the boundary edge The highest values ​​of 0.960, 0.962 and 0.810 were achieved respectively, and the lowest total segmentation error TS was 0.101. In addition, compared with other models, the multimodal tree crown segmentation fusion network (MTCDFNet) showed smaller standard deviations on these indicators, highlighting its robustness and efficiency in accurately segmenting tree crowns using multimodal data. When comparing different fusion strategies involving LiDAR, RGB and GF-2 data, the mid- and early fusion strategies used by the multimodal tree crown segmentation fusion network (MTCDFNet) significantly outperformed the late fusion strategy and the single-branch model (i.e., LTCNet, RTCNet, STCNet) in terms of range, boundary and object level indicators. It is worth noting that the mid-term fusion strategy achieved the highest performance in tree crown segmentation. Among them, the mid-term and early fusion strategies showed high accuracy in extracting the "range", "boundary" and "object" of the tree crown, with Matthew's correlation coefficient MCC scores of 0.960 and 0.957 respectively, and the F1 score F of the tree crown Crown The values ​​are 0.962 and 0.959 respectively, and the F1 score F of the boundary edge The values ​​are 0.810 and 0.780 respectively, and the total segmentation error TS is low, which is 0.101 and 0.144 respectively.

[0125] Table 1. Tree crown segmentation results of different models and fusion strategies in the implementation area. The values ​​in Table 1 represent the mean and standard deviation of each indicator, and the bold values ​​highlight the highest accuracy of various models.

[0126] Table 1:

[0127]

[0128] Comparative experimental results of the Multimodal Crown Division Fusion Network (MTCDFNet):

[0129] Figure 5 shows the comparative experimental results of the Multimodal Crown Segmentation Fusion Network (MTCDFNet) and four other existing models in the implementation area: ResUNet, RAUNet, AUNet and UNet. The results show that the Multimodal Crown Segmentation Fusion Network (MTCDFNet) surpasses these four models in all evaluation indicators, indicating that it has good crown extraction capabilities. Specifically, compared with the Multimodal Crown Segmentation Fusion Network (MTCDFNet), these models show Matthew's correlation coefficient MCC and crown F1 score F Crown It dropped by more than 0.060, and the total segmentation error TS increased by nearly 0.020 ( Figure 5(a)-Figure 5(j) ).

[0130] Figure 5. Comparison results of MTCDFNet, ResUNet, RAUNet, AUNet and UNet models. ( Figure 5a - Figure 5j ) indicates the evaluation results of the crown extraction in the implementation area, showing the Matthew correlation coefficient MCC, the crown F1 score F Crown , the F1 score of the boundary edge Evaluation results for total segmentation error (TS), precision (Precision), recall (Recall), completeness (Completeness), correctness (Correctness), over-segmentation error (OS), and under-segmentation error (US). (S1-S4) are selected from the implementation area to illustrate the detailed errors of tree crown rendering using MTCDFNet, ResUNet, RAUNet, AUNet, and UNet.

[0131] In addition, four subregions were randomly selected within the implementation to compare the specific errors that occurred during the tree crown identification process. The results showed that the Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) effectively identified the tree crowns in all subregions with minimal misclassification and omission errors. However, the performance of the other models varied depending on the forest conditions. Overall, the other four models showed satisfactory performance in subregions (S2) and (S3), which are characterized by dense forests with significant overlap and few small trees or shrubs in the understory. However, there were significant errors in the extraction process in subregions (S1) and (S4), especially misclassification errors. The forest understory in subregions (S1) and (S4) exhibited a multi-layered structure, especially with small trees near the road. Smaller trees have similar spectral and textural characteristics to the surrounding shrubs or herbaceous plants, posing a significant challenge to ResUNet, RAUNet, AUNet, and UNet, which only use single RGB data. In contrast, the multimodal tree crown segmentation fusion network (MTCDFNet) proposed in this paper effectively utilizes the complementarity of the structural, texture and spectral features of LiDAR, GF-2 and RGB data, thereby greatly reducing the misclassification error.

[0132] Transferability of Multimodal Crown Division Fusion Network (MTCDFNet) across different forest types and densities:

[0133] Table 2 presents the experimental results of migration using three methods, M1, M2, and M3, based on the Multimodal Crown Segmentation Fusion Network (MTCDFNet), in a pure fir plantation in Region B, Hunan Province. M1 represents the migration method using a randomly initialized MCDFNet. M2 and M3 represent pre-trained MCDFNet models with and without fine-tuning, respectively. In the pure fir plantation in Region B, the accuracy of crown segmentation varied depending on forest density. Specifically, the MCDFNet performed better in crown segmentation for dense forests (b1) and medium-dense forests (b2) with the same canopy layer and a single tree species. However, the accuracy of crown segmentation for all metrics in sparse forests (b3) was the lowest. In addition to forest density, the migration method also significantly impacted the performance of MCDFNet in crown extraction. Specifically, the pre-trained Multimodal Crown Segmentation Fusion Network (MTCDFNet) (M3) with fine-tuning achieved the most satisfactory results, showing high accuracy in almost all evaluation indicators among the three transfer methods at different forest densities. However, the pre-trained Multimodal Crown Segmentation Fusion Network (MTCDFNet) (M2) without fine-tuning showed the worst performance and lowest accuracy in all indicators. In addition, the randomly initialized Multimodal Crown Segmentation Fusion Network (MTCDFNet) (M1) performed significantly better than M2 in crown delineation, but slightly lower than M3. Overall, the implementation results show that the Multimodal Crown Segmentation Fusion Network (MTCDFNet) model has good transferability and universality across different forest types and densities.

[0134] Table 2

[0135]

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A deep learning method for single tree crown segmentation based on multimodal data, characterized in that: Contains the following steps: Based on high-resolution GF-2 images, LiDAR point cloud features, and ultra-high-resolution RGB multimodal data, a vectorized crown post-processing strategy is integrated to propose a deep learning model, namely the Multimodal Crown Segmentation Fusion Network MTCDFNet. By integrating the forest structure, spectrum, and texture characteristics of multimodal remote sensing data, the accurate segmentation of tree crowns in multi-canopy, multi-species subtropical forests is achieved. The Multimodal Crown Segmentation Fusion Network adopts an encoder-decoder architecture. The encoder is used to extract abstract, distinguishable features, and the decoder uses the extracted features for accurate crown prediction. The multimodal crown segmentation fusion network forms a three-branch structure in the encoder: One is for texture feature extraction of ultra-high-resolution RGB (UHR) 0.0623m images; One is for spectral feature extraction of high-resolution 1m GF-2 images; A forest structure feature extraction for LiDAR point cloud data; The features extracted by the three branches are complementary. The multimodal crown segmentation fusion network constructs three-branch fusion module 0 to three-branch fusion module N, which is used to integrate the multi-level features extracted by the three branches into a joint feature space. The three-branch fusion module generates channel and spatial attention maps for two consecutive sub-modules, the channel attention module and the spatial attention module. The context-aware module includes a convolutional layer with a kernel size of 1 that extracts specific features from the input data, asymmetric convolutions with expansions of 1, 3, and 5 that allow the convolutional layer to capture more contextual information over a larger area without excessively increasing the number of parameters, and a global average pooling layer that provides a holistic view of the features by capturing the global context of the feature map. The gated attention mechanism is used as a multi-level fusion module. The multi-level fusion module uses high-level features to guide low-level features, highlight important spatial details, and filter out irrelevant noise. To account for the differences in appearance and intrinsic properties of tree crowns at different developmental stages, we introduce a recurrent residual block (RR Conv) with a recursive residual structure. This integrates the residual connection and recursive connection within the CNN to improve the network's ability to capture temporal dependencies across different stages while alleviating the gradient vanishing caused by networks with too many layers. Among them: the output of the three-branch fusion module 0 corresponds to the input of the multi-level fusion module N, The output of the three-branch fusion module 1 corresponds to the input of the multi-level fusion module N-1; ... The output of the three-branch fusion module N-1 corresponds to the input of the multi-level fusion module 1, The output of the three-branch fusion module N corresponds to the input of the context-aware module, The three-branch fusion module N is connected in series with the context perception module and the cyclic residual convolution module 1. The cyclic residual convolution module 1 and the multi-level fusion module 1, the cyclic residual convolution module 2, the multi-level fusion module 2, the cyclic residual convolution module 3... the multi-level fusion module N and the cyclic residual convolution module N+1 are connected in series.

2. The deep learning method for single tree crown segmentation based on multimodal data according to claim 1, characterized in that: The three branch structures include: RGB-based branch structure: CNN efficiently extracts spatial features from UHR images, uses ResNet-50 to encode UHR images and outputs multi-layer spatial features, excluding the last pooling layer and fully connected layer; LiDAR or spectral-based branch structure: CHM images and other feature maps or spectral images obtained by lidar are used for tree classification and segmentation, providing detailed forest structure or spectral information. Four CNNs with different filters, namely 1×1, 3×3, 5×5 and 7×7, are introduced for data fusion and semantic segmentation.

3. The deep learning method for single tree crown segmentation based on multimodal data according to claim 1, characterized in that: The multimodal crown segmentation fusion network MTCDFNet extracts various types of features from RGB, GF-2 and LiDAR data, and combines the edge and shape of the crown, tree species type, crown height, density and vertical distribution supplementary information to achieve more accurate crown segmentation.

4. The deep learning method for single tree crown segmentation based on multimodal data according to claim 1, characterized in that: The channel attention module and the spatial attention module are integrated into the multi-level fusion module MLF, which assigns appropriate weights to the multi-channel input features, can focus on the most relevant features and suppress irrelevant noise. The channel attention module weights the different channels of the input feature map, and emphasizes the key channels for crown segmentation by multiplying them with the channel attention weights. The spatial attention module assigns weights to different spatial positions within the feature map, and highlights the most relevant spatial areas for segmenting the crown by multiplying the channel-weighted feature map with the spatial attention weights. It learns "which features are important" through the channel attention mechanism, and then uses the spatial attention mechanism to focus on "where the feature map is important" during training.

Images