A method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network

Abstract

The present application relates to the technical field of extracting tree parameter information from remote sensing images, and specifically to a method for crown segmentation and tree height estimation based on feature interaction and aggregation networks, comprising the following steps: generating a digital orthophoto and a digital elevation model of a tree area; constructing a crown segmentation and tree height estimation model based on a feature interaction and aggregation network. The crown segmentation and tree height estimation model based on the feature interaction and aggregation network comprises a dual-channel composite network based on transfer learning, a feature interaction module FIM, a feature aggregation module FAM, and a multi-level feature fusion decoder module; performing crown depiction and tree height marking on the digital orthophoto and digital elevation model data, generating a training set, a validation set, and a test set, and training the model; and using the trained model to perform crown segmentation and tree height estimation. The present invention can accurately and efficiently extract large-area crown and tree height information from complex terrain.

Description

Technical Field

[0001] The present application relates to the technical field of tree parameter information extraction from remote sensing images, and in particular to a tree crown segmentation and tree height estimation method based on feature interaction and aggregation network. Background Art

[0002] Accurately and quickly obtaining information on tree crown area, height, and location is essential for monitoring tree health and quickly surveying forest resources. However, existing technical methods face challenges in automatically and efficiently obtaining information on tree growth. The terrain in southern China is mainly hilly and mountainous, with large terrain undulations and dense tree growth, making it difficult to dynamically monitor large areas of trees. Traditional field survey methods can provide reliable crown information, but are constrained by factors such as limited measurement range, long intervals, and high costs, and cannot meet the requirements for accurate forest monitoring. In order to achieve high-precision and efficient crown segmentation and tree height estimation that is insensitive to terrain undulations, research on how to extract crown and tree height information from high-resolution and low-cost drone aerial photography data has become a bottleneck in the field of automated extraction of crown and tree height for large areas of forest.

[0003] In recent years, deep learning has achieved great success in the field of image processing and has been widely used in tree crown segmentation and canopy height estimation. Existing semantic segmentation methods for tree crowns usually use high-resolution drone RGB images as the input of the network model and use a single network model to extract the features of the tree crown from the image. The key to this type of method is that the network model extracts rich spatial information (position, shape) and semantic information (category, attribute) from the RGB image, and has a good crown extraction effect in a forest environment with a simple background. However, deep learning methods that only use RGB images have some limitations: (1) they cannot extract the height features of the tree crown and are difficult to apply to the canopy height extraction task; (2) they are difficult to distinguish shrubs and weeds with similar spatial information to the tree crown in complex mountain scenes, resulting in low crown extraction accuracy of the network model. In response to the above limitations, some researchers have fused multiple data such as RGB images, multispectral images, digital surface models (DSMs), canopy height models (CHMs) by channel superposition, and then used deep learning methods to extract features of multi-source data from the fused data. This type of method can use the features of other data sources to increase the difference between the crown and the background, and introduce additional feature information of the crown (such as canopy height), thereby improving the accuracy of crown segmentation and canopy height estimation. However, these methods destroy the original data structure through channel combination, making it difficult for the network model to extract complementary features from different data sources and unable to calibrate different data features. On the other hand, in the process of feature extraction of single-channel network models, the fused multi-channel data will be seriously disturbed by different modal features, thereby inhibiting the ability of the network model to effectively extract high-quality canopy height features from data sources such as CHM and DSM. In addition, for the task of crown height extraction, existing deep learning methods rely heavily on the accuracy of CHM. CHM is usually derived from the difference between DSM and digital terrain model (DTM), and the accuracy of DTM is highly dependent on the accuracy of ground point cloud classification. Therefore, exploring a model that can achieve deep interaction and aggregation of multi-source data is of great value for accurately extracting tree crowns and tree heights in complex mountainous terrain using DSM and RGB images.

[0004] The invention patent with publication number CN115049920A discloses a large-area crop extraction method based on GEE combined with growth period attributes and object-oriented. The invention method improves the utilization rate and calculation efficiency of image bands, and has the advantages of small input data volume and low sample dependence. However, this method cannot achieve end-to-end crop extraction. The accuracy of the object-oriented segmentation method is too dependent on high-quality parameters, and it is difficult to ensure the accuracy of crop extraction under complex backgrounds; the invention patent with publication number CN115995005A discloses a crop extraction method based on single-period high-resolution remote sensing images. The invention discloses a method and device for extracting crops from single-period high-resolution remote sensing images using a convolutional neural network, thereby improving the efficiency of crop extraction. However, this method is not suitable for crop extraction with large terrain undulations and complex fruit tree crown backgrounds, and it is difficult to meet the needs of more refined crown segmentation. The invention patent application with publication number CN114170510A discloses a method and system for intelligent remote sensing extraction of crops based on transfer learning. The invention method effectively controls the differences in phenological characteristics of similar crops caused by regional differences, but does not solve the problem of more refined crown extraction. The invention patent application with publication number C The invention patent application N117690013A discloses a method for quickly estimating the tree height of a desert oasis shelterbelt. The invention method uses the measured data of trees and the net primary productivity to establish a functional relationship with the tree height to estimate the shelterbelt tree height. It has certain innovation, but the functional relationship constructed by using the two variables of net primary productivity and measured data has poor robustness and is difficult to be applied to the estimation of the tree height of a large-scale shelterbelt. The invention patent application with publication number CN112948764A discloses a method for quickly calculating the tree height of a tree. By establishing a tree height-diameter-at-breast-height model for urban greening tree species, Estimating the height of trees. This invention method effectively reduces the cost of urban tree surveys, but it requires measuring the tree's diameter at breast height to estimate the tree's height, and cannot achieve automated estimation of tree height; the invention patent with publication number CN107832681A discloses a method for estimating the height of a single tree in a forest by combining LiDAR point cloud and synchronous remote sensing images. A three-dimensional tree height model is constructed through point cloud and image data to estimate the height value of the canopy apex, which effectively improves the accuracy of estimating the height of a single tree in a forest stand. However, this invention method is too dependent on LiDAR point cloud data and cannot be widely used in large-scale complex scenes. The invention patent with publication number CN116543165A discloses a remote sensing image fruit tree segmentation method based on a dual-channel composite neural network. This invention method can extract fruit tree canopy information from complex terrain and background, but cannot estimate large-scale tree height information from complex terrain in mountainous areas. Summary of the invention

[0005] The purpose of the present invention is to provide a tree crown segmentation and tree height estimation method based on feature interaction and aggregation network, extract complementary features from multi-source data to correct the information of different data sources, so as to promote the fusion of multi-source data features and realize accurate and efficient extraction of tree crown and tree height information in complex terrain scenes.

[0006] The technical solution adopted by the present invention is:

[0007] A tree crown segmentation and tree height estimation method based on feature interaction and aggregation network includes the following steps:

[0008] S1: Generate digital orthophoto DOM and digital elevation model DEM of tree area using drone images;

[0009] S2: constructing a tree crown segmentation and tree height estimation model based on feature interaction and aggregation network, wherein the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network comprises: a dual-channel composite network based on transfer learning that can simultaneously extract multi-source data information, a feature interaction module FIM, a feature aggregation module FAM and a multi-level feature fusion decoder module;

[0010] The dual-channel composite network based on transfer learning includes a tree geometric structure feature extraction network branch based on digital orthophoto DOM and a tree height feature extraction network branch based on digital elevation model DEM;

[0011] The tree geometric structure feature extraction network branch based on the digital orthophoto DOM is an attention mechanism network Transformer, which is used to extract the geometric structure feature information of the tree;

[0012] The tree height feature extraction network branch based on the digital elevation model DEM is also an attention mechanism network Transformer, which is used to extract the height feature information of the trees;

[0013] The feature interaction module FIM is a triple feature interaction module of position, channel and space, which is used to extract the complementary features of the digital orthophoto DOM and the digital elevation model DEM, and realize the deep interaction of the tree geometric structure feature information and the tree height feature information;

[0014] The feature aggregation module FAM is an aggregation module of tree geometric structure feature information and tree height feature information, which is used to achieve the fusion of tree geometric structure feature information and tree height feature information after deep interaction, and obtain a weighted feature map containing tree geometric structure feature information and height feature information;

[0015] The multi-level feature fusion decoder module is a multi-level feature map interactive fusion module based on the cross attention mechanism, which is used to realize the mutual fusion of feature maps of different scales and restore the resolution of feature maps to obtain the final crown segmentation and tree height estimation results;

[0016] S3: using digital orthophoto DOM and digital elevation model DEM to depict tree crowns and mark tree heights, obtaining label data for tree crown segmentation and tree height estimation, and dividing the digital orthophoto DOM, digital high-rise model DEM and their depicted and marked corresponding labels in proportion as the training set, validation set and test set of the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network;

[0017] S4: using the training set and the validation set to train the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network, and calculating the loss between the tree crown segmentation and tree height estimation results obtained by the multi-level feature fusion decoder module and the real tree crown and tree height labels, and finally back-propagating the loss to optimize the model parameters, completing the model training, and testing the model using the test set;

[0018] S5: Use the trained tree crown segmentation and tree height estimation model based on feature interaction and aggregation network to perform tree crown segmentation and tree height estimation on the digital orthophoto DOM and digital elevation model DEM generated by drone images.

[0019] Furthermore, the specific method of step S1 is: performing photogrammetry and aerial triangulation processing on the drone image to obtain a digital orthophoto DOM, a digital elevation model DEM and a three-dimensional dense point cloud.

[0020] Furthermore, the attention mechanism network Transformer includes four stages, each stage consists of an overlapping image block embedding layer and a Transformer module, the beginning of each stage is a layer of overlapping image block embedding layer, and the overlapping image block embedding layer of the first stage, the second stage, the third stage and the fourth stage are followed by stacked 3 layers, 3 layers, 6 layers and 3 layers of Transformer modules respectively; the overlapping image block embedding layer includes a layer of 3×3 convolutional layer and a layer of normalization layer, and the Transformer module includes a layer of efficient multi-head attention mechanism layer and a layer of feedforward neural network layer FNL.

[0021] Furthermore, the feedforward neural network layer FNL includes 1 layer of multilayer perceptron module, 1 layer of 3×3 convolution layer, 1 GELU activation function and 1 layer of multilayer perceptron module stacked in sequence; the multilayer perceptron module includes 1 layer of linear mapping layer, 1 ReLU activation function, 1 layer of linear mapping layer and 1 Sigmoid function connected in sequence.

[0022] Furthermore, the feature interaction module FIM includes a position feature interaction module, a channel feature interaction module and a spatial feature interaction module. The position feature interaction module includes a parallel horizontal average pooling layer and a vertical average pooling layer, followed by a feature extraction module A and two parallel Sigmoid activation functions. The feature extraction module A is composed of a 1×1 convolutional layer, a batch normalization layer BN, and a ReLU activation function stacked in sequence; the channel feature interaction module includes a parallel global maximum pooling layer GMP and a global average pooling layer GAP, followed by a multi-layer perceptron module; the spatial feature interaction module is a feature extraction module B, and the feature extraction module B includes a sequentially stacked 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer and a Sigmoid function.

[0023] Furthermore, the feature aggregation module FAM includes 4 1×1 convolutional layers, 1 3×3 depth-separable convolutional layer, 1 ReLU6 activation function, 1 Softmax activation function, 1 Spilt function and 1 batch normalization layer BN; wherein the 3 1×1 convolutional layers are arranged in parallel, and the middle 1×1 convolutional layer is sequentially stacked with 1 3×3 depth-separable convolutional layer, 1 ReLU6 activation function, 1 1×1 convolutional layer and 1 Softmax activation function, and the output of the Softmax activation function is processed by a Spilt function and then multiplied element-by-element with the outputs of the other 2 1×1 convolutional layers, and the two outputs are added element-by-element and input into the batch normalization layer BN for processing.

[0024] Furthermore, the multi-level feature fusion decoder module includes three parallel cross-attention modules, three parallel MLP modules, followed by one MLP module and two upsampling modules, and the three cross-attention modules interact through two upsampling operations; the MLP module is composed of a layer of normalization layers, a layer of linear mapping layers and a GELU activation function superposition; the upsampling module is composed of a 1×1 convolution layer, a ReLU activation function and an upsampling layer connected in sequence.

[0025] Furthermore, the cross-attention module includes 2 parallel cross-attention layers, 2 parallel feedforward neural network layers FFN, followed by a feature mapping module and a reshape operation; the feature mapping module is composed of a linear mapping function, a 3×3 depth-separable convolution layer, a GELU activation function and a linear mapping function stacked in sequence; the feedforward neural network layer FFN includes a linear mapping layer, a 3×3 depth-separable convolution layer and a linear mapping layer connected in sequence.

[0026] The multi-level feature fusion decoder module realizes deep interactive fusion of feature maps of different scales through the cross-attention module, maintains the consistency of contextual information of feature maps of different scales, uses the MLP module to change the size and number of channels of the output feature map of the cross-attention module, and uses the upsampling layer to restore the feature map to the size of the input image. Finally, the error is calculated with the real tree crown and tree height labels to obtain the model loss. Finally, the loss is back-propagated to optimize the model parameters and complete the model training.

[0027] Furthermore, when constructing the training set, validation set and test set of the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network, the digital orthophoto DOM and digital elevation model DEM data obtained in step S1 are subjected to tree crown depiction and tree height marking to obtain the tree crown and tree height labels of each tree, the digital orthophoto DOM and digital elevation model DEM and the corresponding labels are segmented into 256×256 pixel sizes, and new image sets and label sets are generated by horizontal flipping, vertical flipping and 90° rotation transformation; finally, the total digital orthophoto, digital elevation model and labeled image obtained are divided into a ratio of 6:2:2 as the training set, validation set and test set of the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network.

[0028] The beneficial technical effects of the present invention are:

[0029] (1) The present invention constructs a tree crown segmentation and tree height estimation method based on feature interaction and aggregation network, which is used to extract the tree crown and tree height from drone images of complex terrain. This method can use the correlation of position, channel and space to achieve deep interaction of different data features, so that the two data streams of the network can pay more attention to complementary features and reduce the feature differences between different data. At the same time, this method aggregates the features of the two data streams separately through the feature aggregation module, and uses the soft attention mechanism to efficiently extract feature information from the two data streams that is beneficial to tree crown segmentation, thereby enhancing the output features of the model. In addition, this method can realize information interaction of multi-level features and enhance the model's perception ability of multi-scale targets, solving the problem of insufficient extraction of complementary features from different data sources under multi-source data input conditions, and improving the utilization rate of multi-source data information;

[0030] (2) In order to eliminate the influence of terrain undulation, the digital orthophoto DOM representing the geometric structure characteristics of the tree crown and the digital elevation model DEM representing the tree height characteristics are simultaneously input into the constructed tree crown segmentation and tree height estimation model based on feature interaction and aggregation network, which improves the tree crown segmentation and tree height estimation performance of the model in complex terrain. The tree crown segmentation and tree height estimation strategy of the present invention has achieved good results in forest information extraction and can meet the needs of tree crown extraction and tree height estimation in complex terrain scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of a method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network in an embodiment of the present invention;

[0032] Figure 2 It is a structural schematic diagram of a tree crown segmentation and tree height estimation model based on feature interaction and aggregation network in an embodiment of the present invention;

[0033] Figure 3 It is a structural diagram of a feature interaction module FIM in an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the structure of a feature aggregation module FAM in an embodiment of the present invention;

[0035] Figure 5 Schematic diagram of the structure of a multi-level feature fusion decoder module in an embodiment of the present invention;

[0036] Figure 6 Schematic diagram of the structure of the cross attention module in an embodiment of the present invention;

[0037] Figure 7 This is a schematic diagram of the tree crown and tree height results extracted by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0039] Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with ordinary skills in the field described in this application. The "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0040] like Figure 1 A tree crown segmentation and tree height estimation method based on feature interaction and aggregation network shown in FIG. includes the following steps:

[0041] S1: Generate digital orthophoto DOM and digital elevation model DEM of tree area using drone images. The specific method is as follows:

[0042] Perform photogrammetry and aerial triangulation processing on UAV images to obtain digital orthophotos DOM, digital elevation models DEM and three-dimensional dense point clouds.

[0043] S2: constructing a tree crown segmentation and tree height estimation model based on feature interaction and aggregation network, wherein the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network comprises: a dual-channel composite network based on transfer learning that can simultaneously extract multi-source data information, a feature interaction module FIM, a feature aggregation module FAM and a multi-level feature fusion decoder module;

[0044] like Figure 2 As shown, the dual-channel composite network based on transfer learning includes a tree geometry feature extraction network branch based on digital orthophoto DOM and a tree height feature extraction network branch based on digital elevation model DEM. The digital orthophoto DOM and the digital elevation model DEM are respectively input into the tree geometry feature extraction network branch based on digital orthophoto DOM and the tree height feature extraction network branch based on digital elevation model DEM for feature extraction. The tree geometry feature extraction network branch based on digital orthophoto DOM extracts four different scale feature maps X with rich local detail information and high-level semantic information, namely, feature map X1, feature map X2, feature map X3 and feature map X4. The tree height feature extraction network branch based on digital elevation model DEM extracts four different scale feature maps H with rich tree height information, namely, feature map H1, feature map H2, feature map H3 and feature map H4. Then, the feature maps extracted by the tree geometric structure feature extraction network branch based on the digital orthophoto DOM and the tree height feature extraction network branch based on the digital elevation model DEM are respectively input into the feature interaction module FIM for deep information interaction, and eight calibrated feature maps are obtained, namely, feature map X rec1 , feature map X rec2 , feature map X rec3 , feature map X rec4 , feature map H rec1 , feature map H rec2 , feature map H rec3 And the feature map H rec4. Next, the eight calibrated feature maps are input into the feature aggregation module FAM (where feature maps of the same scale are input into the module together), and four feature maps of different scales T1, T2, T3 and T4 that aggregate geometric structure feature information and height feature information are obtained. Finally, the four feature maps T1, T2, T3 and T4 are input into the multi-level feature fusion decoder module, namely the MLFA-Decoder module, to obtain the results of crown segmentation and tree height estimation, and the loss is calculated between the results and the true value, and the loss is back-propagated in the network model to optimize the model parameters of the entire network.

[0045] The tree geometric structure feature extraction network branch based on the digital orthophoto DOM and the tree height feature extraction network branch based on the digital elevation model DEM are both attention mechanism networks Transformer, and the two feature extraction network branches are used to extract the geometric structure feature information of the trees and the height feature information of the trees, respectively. In an embodiment of the present invention, the attention mechanism network Transformer includes four stages, each stage is composed of an overlapping image block embedding layer and a Transformer module. The beginning part of each stage is a layer of overlapping image block embedding layer. The overlapping image block embedding layer of the first stage, the second stage, the third stage and the fourth stage are respectively followed by stacked 3 layers, 3 layers, 6 layers and 3 layers of Transformer modules; the overlapping image block embedding layer includes a 3×3 convolution layer and a layer of normalization layer, and the overlapping image block embedding layer is used to retain more spatial information and local continuity to alleviate the problem of canopy edge information loss; the Transformer module includes a layer of efficient multi-head attention mechanism layer and a layer of feedforward neural network layer FNL. The efficient multi-head attention mechanism layer is used to extract global information, and the feedforward neural network layer FNL is used to enhance the local feature expression of the model; the feedforward neural network layer FNL includes a layer of multi-layer perceptron module, a layer of 3×3 convolution layer, a GELU activation function and a layer of multi-layer perceptron module stacked in sequence. The feature map sizes output by the four stages of the attention mechanism network Transformer are respectively , , and ,in, H and Ware the length and width of the image input to the network; among them, the feature maps output by the first, second, third and fourth stages of the attention mechanism network Transformer of the tree geometric structure feature extraction network branch based on the digital orthophoto DOM are X1, X2, X3 and X4 respectively, and the feature maps output by the first, second, third and fourth stages of the attention mechanism network Transformer of the tree height feature extraction network branch based on the digital elevation model DEM are H1, H2, H3 and H4 respectively.

[0046] The feature interaction module FIM is a triple feature interaction module of position, channel and space, which is used to extract the complementary features of the digital orthophoto DOM and the digital elevation model DEM, and realize the deep interaction of tree geometric structure feature information and tree height feature information. In an embodiment of the present invention, the feature interaction module FIM includes 1 position feature interaction module, 1 channel feature interaction module and 1 space feature interaction module. The position feature interaction module includes 1 parallel horizontal average pooling layer and 1 vertical average pooling layer, followed by 1 feature extraction module A and 2 parallel Sigmoid activation functions. The feature extraction module A is composed of 1 layer of 1×1 convolution layer, 1 layer of batch normalization layer BN and 1 ReLU activation function stacked in sequence. The channel feature interaction module includes 1 global maximum pooling layer GMP, 1 global average pooling layer GAP and 1 multi-layer perceptron module. The spatial feature interaction module is a feature extraction module B, which includes a 1×1 convolution layer, a ReLU activation function, a 1×1 convolution layer, and a Sigmoid function stacked in sequence. The multilayer perceptron module includes a linear mapping layer, a ReLU activation function, a linear mapping layer, and a Sigmoid function connected in sequence.

[0047] like Figure 3As shown in the figure, first, the feature interaction module FIM pairs the feature maps X and H output by the tree geometric structure feature extraction network branch based on the digital orthophoto DOM and the tree height feature extraction network branch based on the digital elevation model DEM at the same scale and inputs them into the position feature interaction module in pairs. The feature maps are processed by the horizontal average pooling layer and the vertical average pooling layer respectively, so that the feature maps are reduced in dimension in the horizontal and vertical directions respectively, and the location information of the trees is extracted. Each input feature map obtains two feature maps after dimension reduction. Then, a channel splicing operation, i.e., Concat operation, is used to splice the two pairs of feature maps after dimension reduction to obtain a feature map with the number of channels increased by four times. Next, the concatenated feature map is input into the feature extraction module A, and the number of channels of the feature map is reduced and the feature map is normalized through 1 layer of 1×1 convolution layer, 1 layer of batch normalization layer BN and 1 ReLU activation function; then, the normalized feature map is divided into two feature maps with the same number of channels by an equal division operation, and the two feature maps are respectively input into the corresponding 2 Sigmoid activation functions for processing, and four feature maps XP with position weight information are obtained after another division h , Feature Map XP w , Feature map HP h And the characteristic map HP w , where the feature map XP h And the characteristic map HP h is the feature map in the horizontal direction, feature map XP w And the characteristic map HP w is the feature map in the vertical direction. Next, the feature map XP h and Feature Map XP w Multiply the position feature element by element with the input feature map X of the feature interaction module FIM, and transform the feature map HP h And the characteristic map HP w The position feature is multiplied element by element with the input feature map H of the feature interaction module FIM to obtain two feature maps XP and HP that have been corrected for position features.

[0048] Secondly, the feature maps X and H output by the tree geometry feature extraction network branch based on the digital orthophoto DOM and the tree height feature extraction network branch based on the digital elevation model DEM are paired at the same scale and input into the channel feature interaction module in pairs. The feature maps X and H are first spliced ​​using a channel splicing operation, and then the spliced ​​feature maps are processed using the global maximum pooling layer GMP and the global average pooling layer GAP to capture the channel characteristics of the trees and convert the feature maps into four vectors with a length equal to the number of channels. Then, the four vectors are spliced ​​using a channel splicing operation to obtain a vector with four times the number of channels. Next, the concatenated vector is input into a multilayer perceptron module, which compresses the vector length through a linear mapping layer and improves the nonlinear expression ability of the model through a ReLU activation function; then, the vector is input into a linear mapping layer to further reduce the length of the vector, and finally a Sigmoid activation function is used to further improve the expression ability of the model, and the vector is converted into a two-dimensional feature map to obtain two feature maps XC1 and HC1 with channel weight information. After that, the feature map XC1 is multiplied element-by-element by the input feature map X of the feature interaction module FIM, and the feature map HC1 is multiplied element-by-element by the input feature map H of the feature interaction module FIM to obtain two feature maps XC and HC after channel feature correction.

[0049] Finally, the feature maps X and H output by the tree geometric structure feature extraction network branch based on the digital orthophoto DOM and the tree height feature extraction network branch based on the digital elevation model DEM are paired at the same scale and input into the spatial feature interaction module in pairs. The spatial features of the trees are extracted using 1 layer of 1×1 convolution layer, 1 ReLU activation function, 1 layer of 1×1 convolution layer and 1 Sigmoid function in the feature extraction module B. First, the feature map X and the feature map H are spliced ​​using a channel splicing operation, and then the spliced ​​feature map is input into a 1×1 convolution layer to compress the number of channels of the feature map, and a ReLU activation function is used to improve the nonlinear expression ability of the model. Next, the feature map is input into a 1×1 convolution layer to further compress the number of channels of the feature map, and a Sigmoid function is used to further improve the expression ability of the model, and two feature maps XS1 and HS1 with spatial weight information are obtained. After that, the feature map XS1 is multiplied element-by-element by the input feature map X of the feature interaction module FIM, and the feature map HS1 is multiplied element-by-element by the input feature map H of the feature interaction module FIM to obtain two feature maps XS and HS after spatial feature correction. Then, the feature map XP after position feature correction, the feature map XC after channel feature correction, the feature map XS after spatial feature correction, and the input feature map X of the feature interaction module FIM are added element-by-element to obtain the feature map X rec Similarly, the feature map HP after position feature correction, the feature map HC after channel feature correction, the feature map HS after spatial feature correction, and the input feature map H of the feature interaction module FIM are added element by element to obtain the feature map H rec Since the tree geometry feature extraction network branch based on the digital orthophoto DOM and the tree height feature extraction network branch based on the digital elevation model DEM output feature maps X1, X2, X3, X4 and feature maps H1, H2, H3, H4 of four different scales respectively, the feature maps of the same scale output by the two feature extraction network branches must be processed by the feature interaction module FIM at the same time. Therefore, the feature map X1 and the feature map H1 are input into the feature interaction module FIM for processing to obtain the feature map X1. rec1 And the feature map H rec1 At the same time, the feature map X rec1 Add element by element to feature map X1 and input it into feature map X2 for processing. rec1 The feature map X2 is added element by element to the feature map H1 and input into the feature map H2 for processing to enhance the feature expression ability of the model; the processed feature map X2 and the feature map H2 are then input into the feature interaction module FIM for processing to obtain the feature map X rec2 And the feature map H rec2Similarly, the feature map X rec2 Add element by element to feature map X2 and input it into feature map X3 for processing. rec2 The feature map X3 is added element by element with the feature map H2 and input into the feature map H3 for processing; the processed feature map X3 and the feature map H3 are then input into the feature interaction module FIM for processing to obtain the feature map X rec3 And the feature map H rec3 Similarly, the feature map X rec3 Add element by element to feature map X3 and input it into feature map X4 for processing. rec3 The feature map X4 and the feature map H3 are added element by element and input into the feature map H4 for processing; the processed feature map X4 and the feature map H4 are then input into the feature interaction module FIM for processing to obtain the feature map X rec4 And the feature map H rec4 The final feature interaction module FIM outputs the feature map X rec1 , X rec2 , X rec3 , X rec4 And the feature map H rec1 , H rec2 , H rec3 , H rec4 Among them, the feature map X rec1 And the feature map H rec1 is a feature map of the same scale, feature map X rec2 And the feature map H rec2 is a feature map of the same scale, feature map X rec3 And the feature map H rec3 is a feature map of the same scale, feature map X rec4 And the feature map H rec4 are feature maps of the same scale.

[0050] The feature aggregation module FAM is an aggregation module for tree geometric structure feature information and tree height feature information, and is used to realize the fusion of tree geometric structure feature information and height feature information after deep interaction, and obtain a weighted feature map containing tree geometric structure feature information and height feature information. In an embodiment of the present invention, the feature aggregation module FAM includes 4 layers of 1×1 convolutional layers, 1 layer of 3×3 depth-separable convolutional layer, 1 ReLU6 activation function, 1 Softmax activation function, 1 Spilt function and 1 batch normalization layer BN. Specifically, the 3 layers of 1×1 convolutional layers are arranged in parallel, and the middle 1×1 convolutional layer is followed by 1 layer of 3×3 depth-separable convolutional layer, 1 ReLU6 activation function, 1 layer of 1×1 convolutional layer and 1 Softmax activation function. The output of the Softmax activation function is processed by a Spilt function and then multiplied element-by-element with the output of the other 2 layers of 1×1 convolutional layers. The two outputs are added element-by-element and input into the batch normalization layer BN for processing. The feature map of the same scale output by the feature interaction module FIM is input into the feature aggregation module FAM to aggregate the corrected feature map X rec And the feature map H rec .like Figure 4 As shown, the FAM module first uses a channel concatenation operation to transform the feature map X rec And the feature map H rec Then, we use one 1×1 convolutional layer and one 3×3 depth-separable convolutional layer to aggregate the features in the concatenated feature map, and use one ReLU6 activation function to improve the nonlinear expression ability of the model. Next, we input the feature map into one 1×1 convolutional layer to compress the number of channels in the feature map, and input the obtained feature map into the Softmax activation function and the Spilt function to divide the feature map into two equal parts with the same number of channels, and obtain two feature maps F. s1 and F s2 At the same time, the feature aggregation module FAM module uses 1 layer of 1×1 convolutional layer to compress the feature map X rec And the feature map H rec The number of channels, and the feature map F processed by the Spilt function s1 and feature map F s2 The feature maps are obtained by performing element-by-element multiplication with the feature maps after the number of channels is compressed by directly passing through a 1×1 convolution layer. f x and feature map f h After that, the feature map f x and feature map f hThe elements are added one by one, and a batch normalization layer BN is used to normalize the data to obtain the aggregated feature map T. Since the feature interaction module FIM outputs four pairs of feature maps of the same scale, after being processed by the feature aggregation module FAM, four feature maps of different scales T1, T2, T3 and T4 can be obtained.

[0051] The multi-level feature fusion decoder module is a multi-level feature map interactive fusion module based on the cross-attention mechanism, which is used to realize the mutual fusion of feature maps of different scales and restore the resolution of feature maps to obtain the final crown segmentation and tree height estimation results. In an embodiment of the present invention, the multi-level feature fusion decoder module includes three parallel cross-attention modules, three parallel MLP modules, followed by one MLP module and two upsampling modules. The three cross-attention modules interact with each other through two upsampling operations; the MLP module is composed of a layer of normalization layers, a linear mapping layer and a GELU activation function superimposed, and the upsampling module is composed of a 1×1 convolution layer, a ReLU activation function and an upsampling layer connected in sequence. Figure 5 As shown in the figure, the multi-level feature fusion decoder module inputs the four feature maps T1, T2, T3 and T4 obtained by the feature aggregation module FAM into the cross attention module to realize the deep interaction of adjacent feature maps, so as to achieve the purpose of mutual fusion between feature maps of different scales. First, the feature map T4 is upsampled, and then the upsampled feature map T4 and the feature map T3 are input into the first cross attention module to generate a fused feature map F1; the feature map T2 and the upsampled fused feature map F1 are input into the second cross attention module to generate a fused feature map F2; the feature map T1 and the upsampled fused feature map F2 are input into the third cross attention module to generate a fused feature map F3. Next, the upsampled feature map T4 is converted into a vector using the flatten function, and is element-wise added to the fused feature map F1 to obtain the feature map F a1 , and input it into the first MLP module for feature reconstruction to obtain the feature map M1. Then, the feature map F a1 After upsampling, perform element-by-element addition operation with the fusion feature map F2 to obtain the feature map F a2 , and input it into the second MLP module for feature reconstruction to obtain feature map M2. Next, the feature map F a2 After upsampling, perform element-by-element addition operation with the fusion feature map F3 to obtain the feature map F a3, and input it into the third MLP module for feature reconstruction to obtain feature map M3. Then, the upsampled feature map T4, the upsampled feature map M1, the upsampled feature map M2 and the feature map M3 are channel-joined and input into an MLP module for feature fusion. Finally, the fused feature map is input into two upsampling modules for upsampling to restore the resolution of the feature map, so that the resolution of the feature map is the same as the resolution of the input image of the crown segmentation and tree height estimation model based on feature interaction and aggregation network, and the crown segmentation and tree height estimation results are obtained.

[0052] The cross attention module includes two parallel cross attention layers, two parallel feedforward neural network layers FFN, followed by a feature mapping module and a reshape operation. The feature mapping module consists of a linear mapping function, a 3×3 depth-separable convolution layer, a GELU activation function and a linear mapping function stacked in sequence. The feedforward neural network layer FFN includes a linear mapping layer, a 3×3 depth-separable convolution layer and a linear mapping layer connected in sequence. The structure of the cross attention module is as follows Figure 6As shown in the figure, for two feature maps T4 and T3 of different scales with sizes of C1×H1×W1 and C2×H2×W2 (C2=2C1, H2=2H1, W2=2W1, H1, H2 and W1, W2 are the length and width of the input image in the network respectively, and C1, C2 are the number of channels of the input image in the network), the two cross-attention layers first use the flatten function to convert the two feature maps into vectors, and use the channel projection operation to divide the vectors into three component vectors Q, K and V. Then, the component vectors K and V of the two feature maps are respectively multiplied element by element to obtain the two feature vectors corresponding to the two feature maps, and then the two feature vectors are cross-multiplied with each other's component vectors Q to obtain two weighted feature vectors. Next, the two weighted feature vectors are respectively input into two feedforward neural network layers FFN, that is, they are sequentially input into a linear mapping layer, a 3×3 depth-separable convolution layer and a linear mapping layer to improve the nonlinear expression ability of the model. Next, the feature vector output by the feedforward neural network layer FFN is added to the weighted feature vector input by the feedforward neural network layer FFN element by element to improve the training speed of the network. After that, the two added feature vectors are channel concatenated and then input into a linear mapping layer, a 3×3 depth-separable convolution layer, a GELU activation function and a linear mapping layer of the feature mapping module to achieve the fusion of the two feature vectors, and the fused feature vector is reshaped to convert the fused feature vector from a vector form to a matrix form to obtain a fused feature map F1. After that, the fused feature map F1 is upsampled, and the upsampled fused feature map F1 is deeply interactively fused with the feature map T2 in the same way as the above operation to obtain the fused feature map F2; similarly, the fused feature map F2 is upsampled and deeply interactively fused with the feature map T1 to obtain the fused feature map F3.

[0053] S3: Use digital orthophoto DOM and digital elevation model DEM to depict the tree crown and mark the tree height, obtain the label data of tree crown segmentation and tree height estimation, and divide the digital orthophoto DOM, digital high-rise model DEM and the corresponding labels of their depiction and marking in a ratio of 6:2:2 as the training set, validation set and test set of the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network. The embodiment of the present invention uses ArcGIS software to depict the tree crown labels of digital orthophoto DOM and digital elevation model DEM, and the height of the tree crown depicted by the accurate tree height mark extracted in step S1 to obtain the label image of tree crown segmentation and tree height estimation. Then, the digital orthophoto DOM, digital elevation model DEM and label image are segmented into 256×256 pixel sizes, and new images and labels are generated by horizontal flipping, vertical flipping and 90° rotation transformation. Finally, the original image and the transformed image and label are divided in a ratio of 6:2:2 as the training set, validation set and test set of the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network.

[0054] S4: Use the training set and the validation set to train the crown segmentation and tree height estimation model based on feature interaction and aggregation network, and calculate the loss of the crown segmentation and tree height estimation results obtained by the multi-level feature fusion decoder module and the real crown and tree height labels. Finally, back-propagate the loss, optimize the model parameters, complete the model training, and test the model using the test set. The loss calculation formula is as follows:

[0055] ,

[0056] in, G is the total number of label categories, y i,g is a symbolic function, if i The pixels belong to g label categories, then y i,g Equal to 1, otherwise y i,g is equal to 0; Indicates i The pixel is predicted to be g The possibility of label categories, CE represents the cross entropy loss.

[0057] S5: Use the trained tree crown segmentation and tree height estimation model based on feature interaction and aggregation network to perform tree crown segmentation and tree height estimation on the digital orthophoto DOM and digital elevation model DEM generated by drone images.

[0058] The embodiment of the present invention uses the trained tree crown segmentation and tree height estimation model based on feature interaction and aggregation network to perform tree crown segmentation and tree height estimation tests on the test set obtained in step S3, and performs accuracy evaluation on the test results and the true labels. The evaluation index formula for tree crown segmentation is as follows:

[0059] ;

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] in, OA is the overall accuracy, Precision For accuracy, Recall is the recall rate, F 1 for F 1 rating, QUR is the average intersection-union ratio; g Indicates g Tag categories; G Indicates the total number of label categories; P Indicates the correctly predicted pixel; N Indicates the pixel point that is incorrectly predicted; TP represents the correctly predicted canopy pixel; FP Indicates the wrongly predicted pixel where the background is predicted as the canopy; TN Represents the correctly predicted background pixels; FN Indicates incorrectly predicted pixels where the canopy was predicted as background.

[0065] The evaluation index formula for tree height estimation is as follows:

[0066] ,

[0067] ,

[0068] ,

[0069] Among them, MAE is the mean absolute error, RMSE is the root mean square error, R 2 is the coefficient of determination, D The number of tree heights extracted for the model, For the actual tree height, is the predicted tree height, is the average tree height of the test dataset.

[0070] Four existing representative deep learning convolutional neural networks, such as the fully convolutional neural network FCN, the U-shaped structure network U-Net, the high-resolution contextual information extraction network HRCNet_W48 and the deep dense image labeling network DeepLab-V3+, and three existing advanced transformation neural networks, such as the cross-shaped window transformation neural network CSwin-base, the U-shaped structure transformation network TransUNet and the segmentation transformation network SegFormer-B2, are selected for comparative analysis with the tree crown segmentation and tree height estimation method described in the embodiment of the present invention. The obtained tree crown segmentation and tree height estimation results are shown in Table 1.

[0071] Table 1 Evaluation results of tree crown segmentation and tree height estimation accuracy of eight methods:

[0072]

[0073] The method of the present invention achieved the highest accuracy in crown segmentation (e.g., mIoU=95.08%), and compared with other methods, the maximum improvement of mIoU was 7.32%. Compared with networks such as DeepLabV3+, CSwin-Base, and SegFormer-B2, the method of the present invention showed higher accuracy in crown segmentation and tree height estimation, with mIoU increased by 3.12%, 3.06%, and 3.47%, respectively, and RMSE errors reduced by 25.4%, 22.4%, and 26.9%, respectively. This shows that the method of the present invention can accurately and efficiently achieve crown segmentation and tree height estimation in complex terrain, showing superior performance.

[0074] Figure 7 This is a schematic diagram of the results of tree crown segmentation and tree height estimation according to an embodiment of the present invention. Figure 7 (a) is a drone image of a large citrus tree planting area taken in an embodiment of the present invention. Figure 7 (b) is the result of crown segmentation and tree height estimation obtained by the method of the present invention. The advantage of the method of the present invention is that it can accurately and efficiently extract crown information and estimate the height of each tree from a complex mountain environment, and realize the 3D information extraction of citrus trees in a large area. It can be seen that the embodiment of the present invention can accurately and quickly realize large-area crown extraction and tree height estimation.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network, characterized in that: The steps include: S1: Generate digital orthophoto DOM and digital elevation model DEM of tree areas using drone images; S2: constructing a tree crown segmentation and tree height estimation model based on feature interaction and aggregation network, wherein the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network comprises: a dual-channel composite network based on transfer learning that can simultaneously extract multi-source data information, a feature interaction module FIM, a feature aggregation module FAM and a multi-level feature fusion decoder module; The dual-channel composite network based on transfer learning includes a tree geometric structure feature extraction network branch based on the digital orthophoto DOM and a tree height feature extraction network branch based on the digital elevation model DEM; the tree geometric structure feature extraction network branch based on the digital orthophoto DOM and the tree height feature extraction network branch based on the digital elevation model DEM are both attention mechanism networks Transformer, and the two feature extraction network branches are used to extract the geometric structure feature information of the trees and the height feature information of the trees respectively; The feature interaction module FIM is a triple feature interaction module of position, channel and space, which is used to extract the complementary features of the digital orthophoto DOM and the digital elevation model DEM, and realize the deep interaction of tree geometric structure feature information and tree height feature information; The feature aggregation module FAM is an aggregation module of tree geometric structure feature information and tree height feature information, which is used to achieve the fusion of tree geometric structure feature information and tree height feature information after deep interaction, and obtain a weighted feature map containing tree geometric structure feature information and height feature information; The multi-level feature fusion decoder module is a multi-level feature interactive fusion module based on the cross attention mechanism, which is used to realize the mutual fusion of feature maps of different scales and restore the resolution of feature maps to obtain the final crown segmentation and tree height estimation results; S3: using digital orthophoto DOM and digital elevation model DEM to depict tree crowns and mark tree heights, obtaining label data for tree crown segmentation and tree height estimation, and dividing the digital orthophoto DOM, digital high-rise model DEM and their depicted and marked corresponding labels in proportion as the training set, validation set and test set of the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network; S4: using the training set and the validation set to train the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network, and calculating the loss between the tree crown segmentation and tree height estimation results obtained by the multi-level feature fusion decoder module and the real tree crown and tree height labels, and finally back-propagating the loss to optimize the model parameters, completing the model training, and testing the model using the test set; S5: Use the trained tree crown segmentation and tree height estimation model based on feature interaction and aggregation network to perform tree crown segmentation and tree height estimation on the digital orthophoto DOM and digital elevation model DEM generated by drone images.

2. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 1, characterized in that: The specific method of step S1 is: performing photogrammetry aerial triangulation processing on the drone image to obtain a digital orthophoto DOM, a digital elevation model DEM and a three-dimensional dense point cloud.

3. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 1, characterized in that: The attention mechanism network Transformer includes four stages, each stage consists of an overlapping image block embedding layer and a Transformer module. The beginning part of each stage is a layer of overlapping image block embedding layer. The overlapping image block embedding layer of the first stage, the second stage, the third stage and the fourth stage are followed by stacked 3 layers, 3 layers, 6 layers and 3 layers of Transformer modules respectively; the overlapping image block embedding layer includes a layer of 3×3 convolutional layer and a layer of normalization layer, and the Transformer module includes a layer of efficient multi-head attention mechanism layer and a layer of feedforward neural network layer FNL.

4. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 3 is characterized in that: The feedforward neural network layer FNL includes 1 layer of multilayer perceptron module, 1 layer of 3×3 convolution layer, 1 GELU activation function and 1 layer of multilayer perceptron module stacked in sequence; the multilayer perceptron module includes 1 layer of linear mapping layer, 1 ReLU activation function, 1 layer of linear mapping layer and 1 Sigmoid function connected in sequence.

5. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 4, characterized in that: The feature interaction module FIM includes a position feature interaction module, a channel feature interaction module and a spatial feature interaction module. The position feature interaction module includes a parallel horizontal average pooling layer and a vertical average pooling layer, followed by a feature extraction module A and two parallel Sigmoid activation functions. The feature extraction module A is composed of a 1×1 convolution layer, a batch normalization layer BN, and a ReLU activation function stacked in sequence; the channel feature interaction module includes a parallel global maximum pooling layer GMP and a global average pooling layer GAP, followed by a multi-layer perceptron module; The spatial feature interaction module is a feature extraction module B, which includes a 1×1 convolution layer, a ReLU activation function, a 1×1 convolution layer and a Sigmoid function stacked in sequence.

6. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 5, characterized in that: The feature aggregation module FAM includes 4 1×1 convolutional layers, 1 3×3 depth-separable convolutional layer, 1 ReLU6 activation function, 1 Softmax activation function, 1 Spilt function and 1 batch normalization layer BN; wherein 3 1×1 convolutional layers are arranged in parallel, and the middle 1×1 convolutional layer is sequentially stacked with 1 3×3 depth-separable convolutional layer, 1 ReLU6 activation function, 1×1 convolutional layer and 1 Softmax activation function. The output of the Softmax activation function is processed by a Spilt function and then multiplied element-by-element with the outputs of the other 2 1×1 convolutional layers. The two outputs are added element-by-element and input into the batch normalization layer BN for processing.

7. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 6, characterized in that: The multi-level feature fusion decoder module includes three parallel cross-attention modules, three parallel MLP modules, followed by one MLP module and two upsampling modules. The three cross-attention modules interact with each other through two upsampling operations; the MLP module is composed of a layer of normalization layers, a layer of linear mapping layers and a GELU activation function superposition; the upsampling module is composed of a 1×1 convolution layer, a ReLU activation function and an upsampling layer connected in sequence.

8. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 7, characterized in that: The cross attention module includes two parallel cross attention layers, two parallel feedforward neural network layers FFN, followed by a feature mapping module and a reshape operation; the feature mapping module is composed of a linear mapping function, a 3×3 depth-separable convolution layer, a GELU activation function and a linear mapping function stacked in sequence; the feedforward neural network layer FFN includes a linear mapping layer, a 3×3 depth-separable convolution layer and a linear mapping layer connected in sequence.

9. The method for tree crown segmentation and tree height estimation based on feature interaction and aggregation network according to claim 1, characterized in that: The specific method of step S3 is: performing tree crown depiction and tree height marking on the digital orthophoto DOM and digital elevation model DEM data obtained in step S1 to obtain the tree crown and tree height labels of each tree, dividing the digital orthophoto and digital elevation model and the corresponding labels into 256×256 pixel sizes, and generating new image sets and label sets through horizontal flipping, vertical flipping, and 90° rotation transformation; finally, dividing the total digital orthophoto, digital elevation model and label image obtained in a ratio of 6:2:2 as the training set, verification set and test set of the tree crown segmentation and tree height estimation model based on feature interaction and aggregation network.

Images