Single tree crown segmentation deep learning method based on multi-modal data

Through the deep learning method of multimodal data fusion, a multimodal canopy division fusion network is constructed, which solves the accuracy of canopy segmentation in subtropical forests, and realizes high-precision canopy segmentation in multi-canopy and multi-tree environments. Survey on large-scale forest resources.

CN119048748AActive Publication Date: 2024-11-29RES INST OF FOREST RESOURCE INFORMATION TECHN CHINESE ACADEMY OF FORESTRY

Patent Information

Application Number
CN202410968398.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-29
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The existing canopy segmentation method has accuracy challenges in subtropical forests with multiple canopy and multi-tree species. In particular, similar appearance and environmental background factors of the canopy affect the segmentation effect, and traditional models are difficult to achieve accurate division in complex scenarios.

Method used

Using a deep learning method of multimodal data fusion, a multimodal crown division fusion network (MTCDFNet) is built through high-score image GF-2, lidar LiDAR point cloud and ultra-high resolution RGB data, using encoder and decoder Architecture, channel attention and spatial attention modules, recursive residual structures, extract and fuse multi-level features for accurate segmentation of the canopy.

Benefits of technology

The accurate segmentation of the canopy is achieved in subtropical forests with multiple canopy and multiple tree species, which improves the accuracy and accuracy of forest resource inventory, reduces segmentation errors, and is suitable for large-area and automated forest resource surveys.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048748A_ABST
    Figure CN119048748A_ABST
Patent Text Reader

Abstract

The invention discloses a single tree crown segmentation deep learning method based on multi-modal data, and belongs to the technical field of artificial intelligence in forestry scientific research. Based on a high-resolution image GF-2, a laser radar Li DAR point cloud feature and ultrahigh-resolution RGB multi-modal data, a vectorization tree crown post-processing strategy is fused, a deep learning model, namely a multi-modal tree crown division fusion network (MTCDFNet) is proposed, and a deep learning model is constructed by integrating a forest structure, spectrum and texture features of multi-modal remote sensing data. Accurate segmentation of the crowns in the multi-canopy and multi-tree-species subtropical forest is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology in forestry science research and relates to a single tree crown segmentation deep learning method based on multimodal data. Background Art

[0002] Accurate tree crown spatial information is crucial for forest management measures and forest health monitoring and assessment. The quality of single tree crown segmentation is closely related to the estimation of basic forest parameters and tree species classification in single tree scale forest resource inventory. However, the current crown segmentation method has room for improvement in application data and model algorithms.

[0003] In terms of application data: Traditional remote sensing-based methods mainly rely on single-modal data, namely CHM or RGB images, focusing on single features such as texture, spectrum or forest structure, while ignoring the fusion of multi-modal data features. At the same time, the complexity of forest characteristics, namely stand density, crown characteristics and background environment, poses challenges to single-modal data methods, especially in subtropical forests dominated by multiple canopies and multiple tree species. For example, (i) closely adjacent or overlapping crowns, as well as changes in crown size, shape and color may affect the accuracy of segmentation, especially when trees have similar crown appearance; (ii) environmental background factors, such as the presence of shrubs, grass or other low-lying vegetation, may also affect the performance of the training model and cause false positives in the prediction results; (iii) limitations in spatial or spectral resolution may hinder the effectiveness of deep learning models in identifying small crowns or distinguishing variability within crown types.

[0004] In terms of model algorithms: the main segmentation algorithms or models currently used, such as valley-following, region-growing and watershed segmentation, require manual setting of thresholds, and the parameters have specific scene limitations, making it difficult to achieve accurate crown division in complex subtropical forests; the currently widely used convolutional neural network (CNN) has the performance of extracting representative and distinctive features from the original input. It is a robust and effective segmentation method for describing crown UAV-RGB images, but the segmentation results of this method are prone to produce discontinuous and open crown boundaries, which affects the accuracy of crown segmentation to a certain extent. Summary of the invention

[0005] In view of the problems in the prior art, the present invention provides a deep learning method for single tree crown segmentation based on multimodal data.

[0006] A deep learning method for single tree crown segmentation based on multimodal data includes the following steps:

[0007] Based on high-resolution GF-2 images, LiDAR point cloud features and ultra-high resolution RGB multimodal data, and integrating vectorized crown post-processing strategies, a deep learning model, namely the Multimodal Crown Segmentation Fusion Network (MTCDFNet), is proposed. By integrating the forest structure, spectrum and texture characteristics of multimodal remote sensing data, accurate crown segmentation in multi-canopy and multi-species subtropical forests is achieved.

[0008] The multimodal crown segmentation fusion network adopts an encoder-decoder architecture. The encoder is used to extract abstract and distinguishable features, and the decoder uses the extracted features for accurate crown prediction. The multimodal crown segmentation fusion network forms three branch structures in the encoder: one is used for texture feature extraction of ultra-high resolution RGB (UHR, 0.0623m) images; one is used for spectral feature extraction of high-resolution (1m) GF-2 images; and one is used for forest structure feature extraction of LiDAR point cloud data. The features extracted by the three branches are complementary, and the multimodal crown segmentation fusion network constructs a three-branch feature The fusion (TBFF) module is used to integrate the multi-level features extracted by the three branches into a joint feature space. The three-branch feature fusion (TBFF) module generates channel and spatial attention maps for two consecutive sub-modules, namely the channel attention module and the spatial attention module. The context-aware module (CAM) includes a convolutional layer with a kernel size of 1 to extract specific features from the input data, asymmetric convolutions with 1, 3 and 5 expansions allow the convolutional layer to capture more contextual information over a larger area without excessively increasing the parameters, and a global average pooling layer to provide a holistic view of the features by capturing the global context of the feature map. A gated attention mechanism is used as the multi-level fusion (MLF) module. The multi-level fusion (MLF) module uses high-level features to guide low-level features, highlight important spatial details, filter out irrelevant noise, and the differences in the appearance and intrinsic properties of the crown at different developmental stages. The recurrent residual block (RR Conv) with a recursive residual structure is introduced to integrate the residual connection and recursive connection within the CNN to improve the network's ability to capture temporal dependencies across different stages, while alleviating the gradient disappearance caused by the network with too many layers.

[0009] The three branch structures include: RGB-based branch structure: CNN efficiently extracts spatial features from UHR images, uses ResNet-50 to encode UHR images and outputs multi-layer spatial features, excluding the last pooling layer and fully connected layer; LiDAR or spectrum-based branch structure: CHM images and other feature maps or spectral images acquired by lidar are used for tree classification and segmentation, providing detailed forest structure or spectral information; CNN introducing four different filters (i.e., 1×1, 3×3, 5×5, and 7×7) is used for data fusion and semantic segmentation.

[0010] The Multimodal Crown Segmentation Fusion Network (MTCDFNet) extracts various types of features from RGB, GF-2 and LiDAR data, and combines the crown edge and shape, tree species type, crown height, density and vertical distribution complementary information together to achieve more accurate crown segmentation.

[0011] The channel attention module and the spatial attention module are integrated into the TBFF fusion module to assign appropriate weights to multi-channel input features, focusing on the most relevant features and suppressing irrelevant noise. The channel attention module weights different channels of the input feature map and emphasizes the key channels for crown segmentation by multiplying them with the channel attention weights. The spatial attention module assigns weights to different spatial positions within the feature map and highlights the most relevant spatial regions for segmenting the crown by multiplying the channel-weighted feature map with the spatial attention weights. The channel attention mechanism is used to learn "which features are important" and then the spatial attention mechanism is used to focus on "where the feature map is important" during training.

[0012] The advantages of the present invention are: the multimodal crown segmentation fusion network (MTCDFNet) realizes accurate segmentation of tree crowns in multi-canopy, multi-species subtropical forests by integrating the forest structure, spectrum and texture characteristics of multimodal remote sensing data. Its accuracy fully meets the application requirements of large-area, single-tree scale forest resource inventory.

[0013] As shown in Figure 7(a) and Figure 7(b), in order to determine whether the multimodal crown segmentation fusion network (MTCDFNet) is effective in acquiring crown features from UHR, LiDAR, and GF-2 data, 20 crowns were randomly selected from 6 sub-regions, and their completeness and correctness, as well as over-segmentation and under-segmentation errors, were visualized at the individual boundary level and object level.

[0014] The implementation results show that these 20 trees can effectively segment various sparse and dense scenes. At the boundary and object scales, the F1 score F edgeThe values ​​of the Completeness and Correctness indicators are mainly distributed above 0.750, while at the object scale, the values ​​of the total segmentation error TS, over-segmentation error OS and under-segmentation error US are mostly lower than 0.100.

[0015] In addition, the Z-score indicator was selected to describe the distance between each crown and the mean value, in units of standard deviation. The results show that P2, P13, and P20 are small trees under the forest surrounded by large trees, and their Z-scores are at the F1 score F1 of the border. edge , Completeness and Correctness indicators are lower than -1.500, while the total segmentation error TS, over-segmentation error OS and under-segmentation error US indicators are higher than 1.200. For the total segmentation error TS and under-segmentation error US, the Z-score value of P16 exceeds 1.500, and the F1 score F edge , Completeness and Correctness indicators range from 0.1 to 0.80. For the total segmentation error TS and under-segmentation error US, the Z-score of P20 is distributed between -0.229 and 0.303, while the F1 score F edge , Completeness and Correctness indicators are lower than -1.100. The F1 score of other diagnostic trees at the boundary is F edge The Z-scores for the indicators range from -1.000 to 1.000 for completeness, correctness, total segmentation error TS, over-segmentation error OS, and under-segmentation error US. From the above statistical results, it is observed that larger trees in the upper forest layer can be effectively identified. However, smaller trees in dense conditions under the forest are not easily discovered or depicted, most likely because they are misclassified as part of the surrounding or neighboring larger trees. Overall, the implementation results show that the Multimodal Crown Segmentation Fusion Network (MTCDFNet) model can effectively learn multi-channel features of various forest scenes, making it highly effective in segmenting the crown. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. As shown in the figure:

[0017] Figure 1(A) The multimodal tree crown segmentation fusion network MTCDFNet of the present invention.

[0018] FIG. 1(B) Context-aware module CAM of the present invention.

[0019] FIG1(C) shows a multi-level fusion module MLF of the present invention.

[0020] FIG1(D) shows the cyclic residual convolution module RR_Conv of the present invention.

[0021] FIG. 2( a ) shows the basic structure of the LiDAR encoder of the present invention.

[0022] FIG2( b ) shows the multi-level downsampling module structure of the present invention.

[0023] Figure 3 It is the main structure of the three branch feature fusion (TBFF) module of the present invention. Xspectral, Xlidar and Xrgb represent the multi-modal features extracted by the spectral branch, the lidar branch and the RGB branch respectively.

[0024] FIG. 4( a ) is a diagram of a multi-channel data fusion strategy (mid-term fusion strategy) of the present invention.

[0025] FIG4( b ) is a diagram of a multi-channel data fusion strategy (early fusion strategy) of the present invention.

[0026] FIG4( c ) is a diagram of the multi-channel data fusion strategy (late fusion strategy) of the present invention.

[0027] Figure 5(a) Matthew correlation coefficient of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0028] Figure 5(b) F1 score of the crown of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0029] Figure 5(c) shows the F1 scores of the boundaries of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0030] Figure 5(d) shows the total segmentation error of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0031] Figure 5(e) shows the accuracy of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0032] Figure 5(f) Recall rate of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0033] Figure 5(g) shows the completeness of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNET models of the present invention.

[0034] Figure 5(h) The correctness of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0035] Figure 5(i) Over-segmentation error of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0036] Figure 5(j) Under-segmentation error of the comparison results of the MTCDFNet, ResUNet, RAUNet, AUNet and UNet models of the present invention.

[0037] Figure 6 (S1-S4) represents sub-regions 1-4, which are used to illustrate the detailed errors of drawing tree crowns using MTCDFNet, ResUNet, RAUNet, AUNet and UNet.

[0038] Figure 7(a) One of the 20 tree crown segmentation accuracy verifications proposed in different sparse and dense scenarios in the implementation area.

[0039] Figure 7(b) is the second verification of the segmentation accuracy of 20 tree crowns selected in different sparse and dense scenarios in the implementation area. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Example 1: As shown in Figure 1, Figure 2, Figure 3 , Figure 4, Figure 5 and Figure 6 As shown in the figure, a deep learning method for single tree crown segmentation based on multimodal data mainly involves the theory of forest biological characteristics and semantic segmentation technology in computer science. It makes up for the limitations of single modality data in current crown segmentation, provides support for the application of crown segmentation of subtropical forests with multiple canopies and multiple tree species, and provides model support for large-scale, automated and high-precision forest resource surveys.

[0042] A deep learning method for single tree crown segmentation based on multimodal data is proposed. Based on high-resolution image GF-2, LiDAR point cloud features and ultra-high resolution RGB multimodal data, a vectorized crown post-processing strategy is integrated to propose a deep learning model, namely the Multimodal Crown Segmentation Fusion Network (MTCDFNet). By integrating the forest structure, spectral and texture characteristics of multimodal remote sensing data, accurate crown segmentation in multi-canopy and multi-species subtropical forests is achieved.

[0043] Deep learning model: Fusion of GF-2, LiDAR and ultra-high resolution RGB data, as shown below:

[0044] The multimodal crown segmentation fusion network adopts an encoder-decoder architecture. The encoder is used to extract abstract and distinguishable features, and the decoder uses the extracted features for accurate crown prediction.

[0045] The multimodal crown segmentation fusion network mainly forms three branch structures in the encoder:

[0046] One is for texture and other feature extraction of ultra-high resolution RGB (UHR, 0.0623m) images;

[0047] One is for spectral feature extraction of high-resolution (1m) GF-2 images;

[0048] A forest structure feature extraction for LiDAR point cloud data;

[0049] The features extracted by the three branches are complementary.

[0050] Subsequently, the multimodal crown segmentation fusion network constructs a three-branch feature fusion (TBFF) module to integrate the multi-level features extracted by the three branches into a joint feature space to improve the distinguishing feature ability of crown recognition.

[0051] The three-branch feature fusion (TBFF) module generates channel and spatial attention maps from two consecutive sub-modules, namely, channel attention and spatial attention modules.

[0052] Considering the diversity of crown shapes, sizes, and colors in high-density, multi-canopy mixed forest systems, obtaining multi-scale discriminative features of the three branches is the key to accurately extracting the crown.

[0053] Previous studies have demonstrated that the ASPP module can effectively capture contextual information at multiple scales, and the enlarged receptive field is beneficial for detecting and segmenting large objects.

[0054] The context-aware module (CAM) (Figure 1(B)) consists of a convolutional layer with kernel size 1 that extracts specific features from the input data, asymmetric convolutions with dilations of 1, 3, and 5 that allow the convolutional layer to capture more contextual information over a larger area without excessively increasing the parameters, and a global average pooling layer that provides a holistic view of the features by capturing the global context of the feature map.

[0055] The deeper layers of the encoder are able to capture high-level features, which will help identify more discriminative and representative features of the tree crown. However, the downsampling processes involved in building deeper networks, such as pooling and strided convolution, may lead to the loss of fine details and spatial information. In contrast, although the low-level features (i.e., edges, gradients, textures, and colors) extracted from shallow layers lack sufficient abstraction, maintaining better resolution and spatial details from deep layers will be beneficial for accurately predicting the pixel-level position of the tree crown.

[0056] In order to enhance the information integration of low-level and high-level features or abstractions, a gated attention mechanism is adopted as a multi-level fusion (MLF) module ( Figure 1C ).

[0057] The multi-level fusion (MLF) module utilizes high-level features to guide low-level features, highlighting important spatial details while effectively filtering out irrelevant noise.

[0058] Considering the differences in the appearance and intrinsic properties of the crown at different developmental stages, a recurrent residual block (RR Conv) with a recursive residual structure is introduced (Figure 1(D)) to integrate the residual connection and recursive connection within the CNN, thereby improving the network's ability to capture temporal dependencies across different stages and alleviating the gradient vanishing problem caused by networks with too many layers.

[0059] The three branch structures are as follows:

[0060] 1. RGB-based branch structure:

[0061] UHR images are increasingly used in tree crown segmentation tasks due to their ultra-high spatial resolution and low cost, providing rich spatial information that helps to understand the background and interactions between different tree crowns.

[0062] Previous studies have shown that CNN can efficiently extract spatial features from UHR images. However, CNNs with shallow structures and small receptive fields mainly focus on local details and are more sensitive to noise and changes. To solve this problem, He Kaiming et al. (2016) proposed a residual module to increase the number of CNN layers and achieve better performance in object segmentation. However, deeper CNNs usually have more convolutional layers and neurons, resulting in more parameters that need to be learned during training. In order to strike a balance between model depth and computational efficiency, this branch structure uses ResNet-50 to encode UHR images and output multi-layer spatial features, excluding the last pooling layer and fully connected layer.

[0063] Therefore, this branch can obtain the global semantic information of the crown through deep features, and obtain the local details of the crown through shallow features.

[0064] 2. Branched structures based on LiDAR or spectroscopy:

[0065] Like UHR images, CHM images and other feature maps or spectral images acquired by LiDAR are also used for tree classification and segmentation, which can provide detailed forest structure or spectral information. Currently, multi-filter convolutional neural networks have been effectively used to analyze spectral and CHM images. Since traditional CNNs use fixed filter sizes and produce limited field of view, integrated methods combined with multi-scale techniques have been shown to have obvious advantages in segmentation task practice.

[0066] Therefore, this branch introduces four CNNs with different filters (i.e., 1×1, 3×3, 5×5, and 7×7) for data fusion and semantic segmentation.

[0067] In addition, DownConv sequence modules are used to learn multi-layer spatial features from the fused multi-filter CNN output features extracted from the Spectral or LiDAR encoder. The DownConv module has three Conv2D layers, each of which is accompanied by a BatchNorm layer to stabilize the training process.

[0068] In addition, the first Conv2D layer in the DownConv block uses a stride of 2 to halve the spatial resolution of the input feature map. To address the inconsistency in spatial resolution between GF-2 images, LiDAR-derived feature maps, and UHR data, the feature maps obtained from the LiDAR and Spectral branches are upsampled using nearest neighbor interpolation to ensure the same resolution as the feature maps derived from the RGB branch. The LiDAR encoder and the spectral encoder have the same structure

[0069] In Figure 1(A), Figure 1(B), Figure 1(C), and Figure 1(D), Res-1 to Res-4 represent the four residual blocks of ReNet50.

[0070] TBFF, MLF, CAM, and RR Conv represent the three-branch fusion module, multi-level fusion module, context-aware module, and cyclic residual convolution block, respectively. The number of cycles indicates the number of cycles of the cuboids highlighted in light blue.

[0071] The LiDAR encoder in Figure 2(a) and the multi-level downsampling module structure in Figure 2(b).

[0072] The LiDAR encoder and the spectral encoder have the same structure. X0 to XT represent the input images of the LiDAR derived CHM images and other feature maps. Similarly, for the spectral encoder, X0 to XT represent the input images of the GF-2 based images.

[0073] Figure 3 The main structure of the three branch feature fusion (TBFF) module.

[0074] Xspectral, Xlidar, and Xrgb represent the multi-modal features extracted by the spectral branch, lidar branch, and RGB branch, respectively. C and X S denote the input feature maps for channel and spatial attention modules, respectively.

[0075] Three branch fusion modules:

[0076] The three branches of the Multimodal Crown Segmentation Fusion Network (MTCDFNet) mainly extract various types of features from RGB, GF-2 and LiDAR data. Subsequently, complementary information such as the edge and shape of the crown, tree species type, crown height, density and vertical distribution are combined to achieve more accurate crown segmentation with stronger adaptability to noise and changes.

[0077] However, studies have shown that the importance of features varies depending on the canopy properties. Specifically, accurately segmenting canopies with unique and complex spatial features often relies heavily on UHR images. In contrast, distinguishing canopies with similar textural features but different spectral or vertical characteristics requires GF-2 images or LiDAR-based feature maps.

[0078] Therefore, previous multimodal feature fusion strategies ignore the importance of different features in different crown attributes and introduce unimportant feature noise into the segmentation model, which potentially affects the performance of crown segmentation.

[0079] The present invention includes a channel attention module and a spatial attention module ( Figure 3) are sequentially integrated into the TBFF fusion module to assign appropriate weights to multi-channel input features, allowing the model to focus on the most relevant features and suppress irrelevant noise. Specifically, the channel attention module weights different channels of the input feature map and emphasizes the key channels for crown segmentation by multiplying them with the channel attention weights.

[0080] At the same time, the spatial attention module assigns weights to different spatial locations within the feature map, highlighting the most relevant spatial regions for segmenting the crown by multiplying the channel-weighted feature map with the spatial attention weights. That is, the module can learn "which features are important" through the channel attention mechanism, and then use the spatial attention mechanism to focus on "where the feature map is important" during training.

[0081] Embodiment 2: As shown in Figure 1, Figure 2, Figure 3 , Figure 4, Figure 5 and Figure 6 As shown in the figure, a deep learning method for single tree crown segmentation based on multimodal data is used. The implementation area is a mixed Chinese fir forest in Shanxia Experimental Forest Farm in Jiangxi Province. The area has a high species diversity, among which Chinese fir (Cunninghamia Lanceolata), nanmu (Machilus Pauhoi) and Schima Superba are the main species, forming a multi-layer mixed forest with overlapping crowns, different properties and sizes.

[0082] Implementation Design:

[0083] Ablation experiments of Multimodal Crown Division Fusion Network (MTCDFNet):

[0084] In order to evaluate the performance of Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) in extracting tree crowns using multimodal remote sensing data, the three branch structures of Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) were divided into three single-branch networks.

[0085] (1) RTCNet, a network that learns texture features from RGB images.

[0086] (2) STCNet, a network that learns spectral features from GF-2 images.

[0087] (3) LTCNet, a network that learns structural features from LiDAR point cloud data.

[0088] Each network was trained separately using feature maps derived from RGB images, GF-2 images, and LiDAR point cloud data. The performance of the three networks was then compared with that of the Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) to evaluate the effectiveness of utilizing data from various models.

[0089] An effective fusion strategy that integrates complementary information of multi-channel data is the key to improving the accuracy of crown segmentation. Several fusion strategies have been proposed in previous studies, namely early fusion, mid-term fusion, and late fusion (Cai et al., 2023; Shi et al., 2021; Yu et al., 2023).

[0090] The multimodal tree crown segmentation fusion network (MTCDFNet) model proposed in this paper adopts an intermediate fusion strategy. It extracts features from RGB, Spectral and LiDAR data respectively through three branch structures, and uses the TBFF module for fusion.

[0091] To evaluate the performance of the mid-term fusion strategy, early fusion and late fusion were performed for comparison ( Figure 6 ).

[0092] For the early fusion strategy, the GF-2 and LiDAR based feature maps are resampled to the same resolution RGB (0.0623m) and then combined into a single multi-band image.

[0093] For the late fusion strategy, RTCNet, STCNet, and LTCNet are trained on RGB, spectral, and LiDAR point cloud data, respectively, and the final prediction is produced by averaging the outputs of these models.

[0094] Comparative test with existing models:

[0095] To further evaluate the effectiveness of the Multimodal Crown Segmentation Fusion Network (MTCDFNet) in segmenting the crown, it is compared with four models currently widely used in object segmentation research: UNet (Guo et al., 2021; Ronneberger et al., 2015), AUNet (Yi et al., 2023), RAUNet (Cai et al., 2023) and ResUNet (Cao and Zhang, 2020; Diakogiannis et al., 2020). Among them, UNet is a popular U-shaped structure that is widely used in semantic segmentation tasks.

[0096] ResUNet and AUNet are both encoder-decoder models that combine residual blocks with cross-attention mechanisms and ASPP modules, respectively. Meanwhile, RAUNet uses gated attention modules and residual blocks to accurately detect objects with clear boundaries at the detail level.

[0097] Migration experiments with different forest characteristics:

[0098] Crown size and shape are often influenced by forest characteristics such as stand density, leading to significant differences in crown properties between mixed forests and single-species plantations.

[0099] In order to evaluate the transferability and universality of the multimodal crown division fusion network (MTCDFNet) between different forest types, the mixed fir forest in Jiangxi Shanxia Experimental Forest Farm (Area A) was selected as the original training area, and the artificial pure fir forest in Huangfengqiao State-owned Forest Farm (Area B) was selected as the target migration area for testing. According to the percentage of adjacent crown overlap in the selected implementation area, Area B was divided into three forest levels: dense, medium and sparse (b1, b2, b3). Among them, three key experiments were mainly carried out:

[0100] Including M1: To evaluate the performance of the multimodal crown segmentation fusion network (MTCDFNet) in pure fir plantations with different forest densities, the multimodal crown segmentation fusion network (MTCDFNet) of pure fir plantations in migration area B was randomly initialized and trained using limited data from dense b1, medium b2, and sparse b3 forests respectively;

[0101] Including M2: To evaluate the generalization ability of the multimodal crown segmentation fusion network (MTCDFNet) on different forest types and densities, the multimodal crown segmentation fusion network (MTCDFNet) was pre-trained with data from mixed Chinese fir forests in region A, and then the model was applied to dense b1, medium b2, and sparse b3 densities of pure Chinese fir plantations in region B;

[0102] Including M3: In order to determine the improvement performance of the pre-trained model of the sampling combined with fine-tuning method on the prediction results, the multimodal crown division fusion network (MTCDFNet) was first pre-trained using data from the mixed Chinese fir forest in area A, and then fine-tuned in the dense b1, medium b2 and sparse b3 forests of the pure Chinese fir plantation in area B using limited data.

[0103] Implementation steps:

[0104] All models were trained for 2000 epochs to achieve convergence in ablation and comparative experiments. In order to speed up the training process, the Adam optimizer was used, and the Beta value was set to 0.9% and 0.999. At the same time, in order to enhance the generalization and optimization capabilities of the model, a multi-step learning rate decay strategy was adopted, and the initial learning rate was set to 1e-2. In addition, several data augmentation techniques were applied, including random horizontal and vertical flipping, random resizing, and random rotation of 45, 135, 225, and 315 degrees, which effectively expanded the training dataset. All experiments used the TensorFlow (2.8.0) architecture, using NVIDIA GeForce RTX 3070 (8GB VRAM), CUDA version 11.4, Intel R Core TM i7-10700K (3.80GHz), 128GB of RAM, and operated on a system running Ubuntu 18.04.6LTS.

[0105] Evaluation Metrics:

[0106] The performance of crown segmentation is evaluated using pixel-based metrics, including precision, recall, and the crown F1 score. Crown and Matthew’s correlation coefficient (MCC) ( Diakogiannis et al., 2020 ; Shendryk et al., 2016 ; Waldner and Diakogiannis, 2020 ).

[0107] The calculation formulas for these four evaluation indicators are as follows:

[0108]

[0109]

[0110]

[0111]

[0112] Where Tp, Tn, Fp and Fn represent the number of pixels correctly classified as crown, pixels classified as background, pixels misclassified as crown and pixels misclassified as background, respectively. Precision corresponds to commission error, while recall corresponds to omission errors. The overall accuracy of crown delineation is evaluated by combining precision and recall by calculating the harmonic mean. The Matthew correlation coefficient (MCC) measures the consistency between the predicted and reference crowns, especially when dealing with unbalanced data.

[0113] The edge-based method and the object-based method were used to evaluate the accuracy of crown segmentation at the boundary and object levels, respectively. Three edge-based evaluation metrics were used: completeness (Com), correctness (Corr), and the boundary F1 score F edge To evaluate the accuracy of crown boundary extraction. Completeness quantifies the proportion of accurately detected reference crown boundaries, while Correctness evaluates the proportion of incorrectly detected boundaries in the predicted crown. The F1 score of the boundary F edge The harmonic mean representing completeness and correctness provides a robust measure of the accuracy of boundary delineation. The calculation formulas for these three evaluation indicators are as follows:

[0114]

[0115]

[0116]

[0117] To further complete the object-scale crown evaluation, three object-based evaluation indicators are used: over-segmentation error OS, under-segmentation error US and total segmentation error TS (Cai et al., 2023; Waldner and Diakogiannis, 2020; Yang et al., 2015). The over-segmentation error OS and under-segmentation error US quantify the errors of over-segmentation and under-segmentation, respectively. The total segmentation error TS provides an overall evaluation of the two segmentation errors and provides a detection of boundary connectivity for generating the closed boundary of the crown. The calculation formulas of these three evaluation indicators are as follows:

[0118]

[0119]

[0120]

[0121] Where P represents a predicted tree crown set with n tree crowns {P1, P2, ..., P n}, and R represents a reference crown set with m crowns {R1, R2, ..., R m}. OS and US represent the errors of over-segmentation and under-segmentation, respectively. I When overlapping with multiple reference crowns, P I The error is calculated as the weighted sum of over-segmentation and under-segmentation errors, determined by the area of ​​the overlapping region.

[0122] Implementation results:

[0123] Ablation experiment results of Multimodal Crown Division Fusion Network (MTCDFNet):

[0124] Table 1 shows the performance of different models in the crown extraction of the implementation area. Among the various models, the multimodal crown segmentation fusion network (MTCDFNet) showed superior performance in terms of Matthew correlation coefficient MCC, crown F1 score F Crown and the F1 score of the boundary edge The highest values ​​of 0.960, 0.962 and 0.810 were achieved respectively, and the lowest total segmentation error TS was 0.101. In addition, compared with other models, the multimodal crown segmentation fusion network (MTCDFNet) showed smaller standard deviations in these indicators, highlighting its robustness and efficiency in accurately segmenting the crown using multimodal data. When comparing different fusion strategies involving LiDAR, RGB and GF-2 data, the mid- and early fusion strategies used by the multimodal crown segmentation fusion network (MTCDFNet) significantly outperformed the late fusion strategy and the single-branch model (i.e., LTCNet, RTCNet, STCNet) in terms of range, boundary and object level indicators. It is worth noting that the mid-term fusion strategy achieved the highest performance in crown segmentation. Among them, the mid-term and early fusion strategies showed high accuracy in extracting the "range", "boundary" and "object" of the crown, with Matthew's correlation coefficient MCC scores of 0.960 and 0.957 respectively, and the F1 score F of the crown Crown The values ​​are 0.962 and 0.959 respectively, and the F1 score F of the boundary edge The values ​​are 0.810 and 0.780 respectively, and the total segmentation error TS is low, which are 0.101 and 0.144 respectively.

[0125] Table 1. Tree crown segmentation results of different models and fusion strategies in the implementation area. The values ​​in Table 1 represent the mean and standard deviation of each indicator, and the bold values ​​highlight the highest accuracy of various models.

[0126] Table 1:

[0127]

[0128] Comparative experimental results of the multimodal crown segmentation fusion network (MTCDFNet):

[0129] Figure 5 shows the comparative experimental results of the multimodal crown segmentation fusion network (MTCDFNet) and four other existing models in the implementation area: ResUNet, RAUNet, AUNet and UNet. The results show that the multimodal crown segmentation fusion network (MTCDFNet) exceeds these four models in all evaluation indicators, indicating that it has good crown extraction capabilities. Specifically, compared with the multimodal crown segmentation fusion network (MTCDFNet), these models show Matthew correlation coefficient MCC and crown F1 score F Crown It dropped by more than 0.060, and the total segmentation error TS increased by nearly 0.020 ( Figure 5(a)-Figure 5(j) ).

[0130] Figure 5. Comparison results of MTCDFNet, ResUNet, RAUNet, AUNet and UNet models. ( Figure 5a - Figure 5j ) indicates the evaluation results of the crown extraction in the implementation area, showing the Matthew correlation coefficient MCC, the crown F1 score F Crown , the F1 score of the boundary edge , total segmentation error TS, precision Precision, recall Recall, completeness Completeness, correctness Correctness, over-segmentation error OS and under-segmentation error US. (S1-S4) are selected from the implementation area to illustrate the detailed errors of drawing tree crowns using MTCDFNet, ResUNet, RAUNet, AUNet and UNet.

[0131] In addition, four sub-regions were randomly selected within the implementation to compare the specific errors that occurred during the process of identifying the tree crown. The results show that the Multimodal Tree Crown Segmentation Fusion Network (MTCDFNet) effectively identified the tree crowns in all sub-regions with minimal misclassification and omission errors. However, the performance of other models varied depending on the forest conditions. Overall, the other four models showed satisfactory performance in sub-regions (S2) and (S3), which are characterized by dense forests, significant overlaps, and fewer small trees or shrubs under the forest. However, there were significant errors in the extraction process of sub-regions (S1) and (S4), especially misclassification errors. The forest understory in sub-regions (S1) and (S4) showed a multi-layered structure, especially with small trees near the road. Smaller trees have similar spectral and textural characteristics to the surrounding shrubs or herbaceous plants, which poses a significant challenge to ResUNet, RAUNet, AUNet, and UNet, which only adopt single RGB data. In contrast, the multimodal tree crown segmentation fusion network (MTCDFNet) proposed in this paper effectively utilizes the complementarity of the structure, texture and spectral features of Li DAR, GF-2 and RGB data, thereby greatly reducing the misclassification error.

[0132] Transferability of Multimodal Crown Division Fusion Network (MTCDFNet) across different forest types and densities:

[0133] Table 2 shows the migration test results of three methods M1, M2 and M3 based on the multimodal crown segmentation fusion network (MTCDFNet) in the pure artificial Chinese fir forest in Area B, Hunan Province. M1 represents the migration method of the multimodal crown segmentation fusion network (MTCDFNet) with random initialization. M2 and M3 represent the pre-trained multimodal crown segmentation fusion network (MTCDFNet) models without fine-tuning strategy and with fine-tuning strategy, respectively. In the pure artificial Chinese fir forest in Area B, the accuracy of crown segmentation varies with forest density. Specifically, the multimodal crown segmentation fusion network (MTCDFNet) has better performance in crown segmentation of dense forest b1 and medium-dense forest b2 with the same canopy and single tree species. However, the accuracy of each index of crown segmentation in sparse forest b3 is the lowest. In addition to forest density, the migration method also has a significant impact on the performance of crown extraction by the multimodal crown segmentation fusion network (MTCDFNet). Specifically, the pre-trained Multimodal Crown Segmentation Fusion Network (MTCDFNet) (M3) with fine-tuning achieved the most satisfactory results, showing high accuracy in almost all evaluation indicators among the three transfer methods in different forest densities. However, the pre-trained Multimodal Crown Segmentation Fusion Network (MTCDFNet) (M2) without fine-tuning showed the worst performance and the lowest accuracy in all indicators. In addition, the randomly initialized Multimodal Crown Segmentation Fusion Network (MTCDFNet) (M1) performed significantly better than M2 in crown delineation, but slightly lower than M3. Overall, the implementation results show that the Multimodal Crown Segmentation Fusion Network (MTCDFNet) model has good transferability and universality in different forest types and densities.

[0134] Table 2

[0135]

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep learning method for single tree crown segmentation based on multimodal data, characterized in that: Contains the following steps: Based on high-resolution GF-2 images, LiDAR point cloud features and ultra-high resolution RGB multimodal data, and integrating vectorized crown post-processing strategies, a deep learning model, namely the Multimodal Crown Segmentation Fusion Network (MTCDFNet), is proposed. By integrating the forest structure, spectrum and texture characteristics of multimodal remote sensing data, accurate crown segmentation in multi-canopy and multi-species subtropical forests is achieved.

2. According to claim 1, a deep learning method for single tree crown segmentation based on multimodal data is characterized in that: The multimodal crown segmentation fusion network adopts an encoder-decoder architecture. The encoder is used to extract abstract and distinguishable features, and the decoder uses the extracted features for accurate crown prediction. The multimodal crown segmentation fusion network forms a three-branch structure in the encoder: One is for texture feature extraction of ultra-high resolution RGB (UHR, 0.0623m) images; One is for spectral feature extraction of high-resolution (1m) GF-2 images; A forest structure feature extraction for LiDAR point cloud data; The features extracted by the three branches are complementary. The multimodal crown segmentation fusion network constructs a three-branch feature fusion (TBFF) module to integrate the multi-level features extracted from the three branches into a joint feature space. The three-branch feature fusion (TBFF) module generates channel and spatial attention maps for two consecutive sub-modules, namely the channel attention module and the spatial attention module. Context-Aware Module (CAM), including convolutional layers with kernel size 1 to extract specific features from the input data, asymmetric convolutions with 1, 3, and 5 dilations that allow the convolutional layers to capture more contextual information over a larger area without excessively increasing the parameters, and global average pooling layers that provide a holistic view of the features by capturing the global context of the feature map. The gated attention mechanism is used as the multi-level fusion (MLF) module, which uses high-level features to guide low-level features, highlight important spatial details, and filter out irrelevant noise. In view of the differences in appearance and intrinsic properties of tree crowns at different developmental stages, a recurrent residual block (RR Conv) with a recursive residual structure is introduced to integrate the residual connection and recursive connection within CNN, thereby improving the network's ability to capture temporal dependencies across different stages and alleviating the gradient vanishing caused by networks with too many layers.

3. The deep learning method for single tree crown segmentation based on multimodal data according to claim 2 is characterized in that: The three branch structures include: The RGB-based branch structure: CNN efficiently extracts spatial features from UHR images, uses ResNet-50 to encode UHR images and outputs multi-layer spatial features, excluding the last pooling layer and fully connected layer. Branched structure based on LiDAR or spectrum: CHM images and other feature maps or spectral images acquired by LiDAR are used for tree classification and segmentation, providing detailed forest structure or spectral information. CNN with four different filters (i.e., 1×1, 3×3, 5×5, and 7×7) is used for data fusion and semantic segmentation.

4. The deep learning method for single tree crown segmentation based on multimodal data according to claim 2 is characterized in that: The Multimodal Crown Segmentation Fusion Network (MTCDFNet) extracts various types of features from RGB, GF-2 and LiDAR data, and combines the crown edge and shape, tree species type, crown height, density and vertical distribution complementary information together to achieve more accurate crown segmentation.

5. The deep learning method for single tree crown segmentation based on multimodal data according to claim 2, characterized in that: The channel attention module and the spatial attention module are integrated into the TBFF fusion module to assign appropriate weights to multi-channel input features, focusing on the most relevant features and suppressing irrelevant noise. The channel attention module weights different channels of the input feature map and emphasizes the key channels for crown segmentation by multiplying them with the channel attention weights. The spatial attention module assigns weights to different spatial positions within the feature map and highlights the most relevant spatial regions for segmenting the crown by multiplying the channel-weighted feature map with the spatial attention weights. It learns "which features are important" through the channel attention mechanism and subsequently uses the spatial attention mechanism to focus on "where the feature map is important" during training.

Citation Information

Patent Citations

  • Tree species classification method based on multi-source simultaneous high-resolution remote sensing data

    CN105354534A

  • Tree species classification method based on an unmanned aerial vehicle hyperspectral image and LiDAR point cloud

    CN109492563A

  • Automatic bridge crack detection system fusing cyclic residual convolution and context extractor network

    CN112884747A

  • Hyperspectral image and LiDAR data collaborative classification method

    CN114708455A

  • Single tree segmentation method based on fused LiDAR point cloud data

    CN115100232A

Cited By

  • Forest landscape visual quality evaluation method and system based on point cloud data and machine learning

    CN120107761A

  • Automatic color correction method for mass remote sensing images of sub-meter satellite

    CN120411262A

  • Method and system for identifying forest stand tree species based on multi-source remote sensing data fusion

    CN120544052A

  • A forest stand tree species identification method and system based on multi-source remote sensing data fusion

    CN120544052B

  • Forest crown vertical structure feature inversion method and large-area forest classification method and system

    CN120744702A