Satellite-borne remote sensing forest tree species identification method based on multi-modal alignment hierarchical fusion
By using a multimodal alignment hierarchical fusion network, the problems of feature misalignment and semantic gap in multimodal remote sensing data in complex forest areas are solved, achieving high-precision forest tree species identification and improving the accuracy and robustness of tree species identification. It is applicable to forest resource surveys and management in different climate zones.
Patent Information
- Application Number
- CN202610561675.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-27
- Publication Date
- 2026-05-26
- Estimated Expiration
- 2046-04-27
AI Technical Summary
Existing multimodal remote sensing data identification methods fail to fully consider the differences in imaging mechanisms, spatial resolution, feature dimensions, and physical semantics among different modalities, resulting in limited tree species identification accuracy. In particular, in complex forest areas, there are problems such as feature misalignment, information distortion, and semantic gap.
A multimodal alignment hierarchical fusion network is adopted. Through a three-branch feature extraction module, an intramodal constraint module, and a cross-modal hierarchical fusion module, differential features are extracted from spaceborne multispectral data, lidar vertical structure data, and high-resolution panchromatic images, respectively. Accurate alignment and distortion correction are achieved through a cross-modal guided attention mechanism, and finally, the identification results of dominant tree species in the forest are output.
It significantly enhances the collaborative expression capability of multi-source remote sensing information in complex forest backgrounds, strengthens the identification capability of tree species with similar spectra, improves the accuracy and robustness of tree species identification at the regional scale, adapts to different forest stand structures and ecological backgrounds, and has good regional adaptability and promotion and application value.
Smart Images

Figure CN122090290A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal remote sensing data processing and analysis technology, and in particular to a method for identifying forest tree species in spaceborne remote sensing based on multimodal alignment and hierarchical fusion. Background Technology
[0002] Identification of dominant tree species in forests is an important aspect of forest resource monitoring. Existing research mainly relies on multispectral remote sensing data for tree species classification. However, in forest areas with high canopy closure and complex backgrounds, spectral information is easily affected by shadows, mixed pixels, and environmental differences, resulting in problems such as "different species sharing the same spectrum" and "same species having different spectra," which limits the accuracy of tree species identification.
[0003] To improve recognition accuracy, existing studies have incorporated high-resolution image texture features and lidar vertical structure features as auxiliary information. High-resolution images can enhance canopy texture and edge details, but they struggle to depict the vertical structure of forest stands. LiDAR can provide three-dimensional information such as tree height, canopy profile, and leaf area index; however, current tree species identification research is mostly based on airborne lidar, which suffers from limited coverage, high cost, and insufficient regional application. The application of spaceborne lidar in regional-scale tree species identification remains insufficient.
[0004] While existing multimodal tree species identification methods can fuse multi-source information such as multispectral, lidar, and high-resolution imagery, most of them employ simple stitching or general deep learning structures, failing to fully consider the differences in imaging mechanisms, spatial resolution, feature dimensions, and physical semantics among different modalities. This easily leads to problems such as feature misalignment, information distortion, and semantic gaps during the fusion process. Furthermore, they lack intramodal constraints, precise cross-modal alignment, and hierarchical fusion mechanisms, making it difficult to fully leverage the complementary advantages of multimodal information and limiting the improvement of regional dominant tree species identification performance in complex forest scenarios.
[0005] Therefore, there is an urgent need for a method to identify dominant tree species in regional forests that can perform differentiated modeling for the heterogeneity of multimodal remote sensing data and achieve accurate cross-modal alignment and hierarchical fusion. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides a multimodal alignment and hierarchical fusion method for identifying tree species in spaceborne remote sensing forests, which solves the technical problems of existing multimodal deep learning models being unable to adapt to the heterogeneity of remote sensing data and lacking accurate alignment and hierarchical integration mechanisms.
[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal alignment hierarchical fusion method for identifying forest tree species using spaceborne remote sensing, characterized in that the method includes the following steps: Acquire multimodal remote sensing data, including spaceborne multispectral data, lidar vertical structure data, and high-resolution panchromatic imagery, for complex forest scenes within the target area; A multimodal alignment hierarchical fusion network is used to perform differentiated modeling on multimodal remote sensing data to extract multimodal forest stand features. A two-stage fusion strategy is adopted to achieve cross-modal hierarchical interactive fusion of multimodal forest stand features, and finally output the forest dominant tree species identification results. The multimodal alignment hierarchical fusion network includes: A three-branch feature extraction module is used to simultaneously extract differentiated stand spectral features, vertical structure features, and canopy spatial texture features from spaceborne multispectral data, lidar vertical structure data, and high-resolution panchromatic images. In addition, an intramodal constraint module is used to perform global-local joint optimization of multimodal forest stand features, which enhances the expression of local details while maintaining global semantic consistency; Furthermore, a cross-modal hierarchical fusion module is developed that uses high-resolution canopy spatial texture features as a guide, performs precise alignment and distortion correction of forest stand spectral and vertical structure features based on a cross-modal guided attention mechanism, and then completes the deep interactive fusion of spectral, texture and vertical structure features through hybrid attention blocks.
[0008] Furthermore, the three-branch feature extraction module includes: A branch for extracting MSI features of forest stand spectral characteristics is developed by using depthwise separable convolution (DSC) and squeeze-excitation mechanism (SE-Block) to enhance band independence and channel discriminability. In addition, by introducing Vertical Separation Convolution (VSC) as a mechanism for extracting vertical direction information of forest stands, a LiDAR vertical structure feature extraction branch is used to extract differentiated vertical structure information of forest stands from the inverted spaceborne lidar GEDI data to obtain vertical structure features. Furthermore, a scale-adaptive deformable convolutional SADC is used to dynamically capture canopy edges and spatial texture details. An adaptive average pooling layer is introduced after the scale-adaptive deformable convolutional SADC to achieve a PAN spatial detail extraction branch that obtains canopy spatial texture features in a unified scale with other modal features.
[0009] Furthermore, the depthwise separable convolutional (DSC) is configured as three layers stacked sequentially, and works in conjunction with BatchNorm normalization and ReLU activation function to form a progressive spectral feature extraction mechanism, wherein: The first layer of depthwise separable convolutional DSC focuses on extracting raw spectral features, the middle layer of depthwise separable convolutional DSC captures the interaction between bands, and the third layer of depthwise separable convolutional DSC deeply integrates joint spatial spectral features. The squeeze-excitation mechanism SE-Block adaptively enhances the spectral feature representation of important frequency bands and suppresses redundant channel information through global average pooling and channel weight adjustment.
[0010] Furthermore, the MSI feature extraction branch also includes the introduction of spatial location coding (SPE) to embed pixel coordinate information and explicitly learn the spatial distribution patterns of different dominant tree species.
[0011] Furthermore, the Vertical Separable Convolution (VSC) adopts a three-layer vertically separable convolution structure, with each convolution layer followed by BatchNorm normalization and ReLU activation function, and residual connections are introduced after each convolution layer; The vertically separated convolution (VSC) introduces a squeeze-excitation mechanism (SE-Block) at the end and preserves the spatial distribution information of the vertical information feature map through spatial position encoding (SPE). The Vertical Separating Convolution (VSC) uses a vertical convolution kernel to perform convolution operations along the vertical direction of the feature map. This is done to improve the ability to extract local details of vertical structural features while reducing the number of computational parameters. The expression is as follows: In the formula, X Input feature map; Y Output feature map; For the first c Vertical convolution kernels with channels of size . K ×1; i, j To output feature map Y Spatial location coordinates; k It is the vertical position index within the convolution kernel; This represents the offset of the center position of the convolution kernel.
[0012] Furthermore, the scale-adaptive deformable convolutional SADC dynamically adjusts the receptive field by combining a learnable scale factor with a tanh nonlinear activation function to generate an adaptive scale sampling point offset, which is used to capture multi-scale features from single trees to stand patches. The generation of the sampling point offset includes: The offset on the multi-scale feature map is calculated using the scale-aware module. The calculation process is as follows: In the formula, For the first s Feature maps at various scales; H and W These are the height and width of the feature map, respectively; X Input feature map; For the firsts Channel descriptors of various scales; For the first s Learnable weight matrices at various scales; For the first s Bias terms at each scale; Indicates the first s Offset at each scale; Indicates the use of extracting the first i Convolution kernels for features at each scale; tanh It is the hyperbolic tangent activation function; The scale weights are calculated using a channel attention mechanism, which accounts for the offsets generated by feature maps at different scales. Weighted fusion is performed to obtain the sampling point offset. The calculation process is as follows: In the formula, Spatial position coordinates are The output feature map; For the attention mechanism, there is a learnable weight vector; This is a bias term for the attention mechanism; This represents the sampling point offset after the fusion scale adaptive constraint mechanism. For sampling points at x directional offset; For sampling points at y directional offset; The sampling network representing the convolution kernel, k and l Represents the sampling grid The relative coordinates of the sampling points; For the convolution kernel at position The weights; A dummy variable for scale indexing; The channel descriptor represents the corresponding scale.
[0013] Furthermore, the adaptive average pooling layer is used to automatically apply pooling kernels of different sizes according to the resolution difference to dynamically adjust the size of the feature map, as expressed in the following expression: In the formula, Spatial position coordinates are The channel is The output feature map; For the corresponding spatial coordinates The input feature map region; This indicates that the input feature map is at the location ,aisle The value of .
[0014] Furthermore, the intra-modal constraint module introduces local depthwise convolution operations into the Encoder of the Vision Transformer to replace the standard feedforward neural network layer, thereby capturing local feature changes within multimodal forest stand features while maintaining global semantic consistency; the specific process is as follows: Global feature dependencies are extracted using a multi-head self-attention mechanism, and the global spatial structure is preserved through relative position encoding, as expressed in the following expression: In the formula, The output feature map after self-attention weighting. These are the query, key, and value matrices, respectively. The feature output is enhanced with relative position information; d For attention head dimension; By introducing local depthwise convolution operations to replace the standard feedforward neural network layers, feature information from neighboring locations is fused. The expression is as follows: In the formula, For the first c Each channel is located in i The output feature values after convolution; Channel-specific convolution; i The output feature map contains the position coordinates. δ The GELU nonlinear activation function; k and k′ It is the relative position offset within the convolution kernel; This represents the offset of the center position of the convolution kernel; and For bias terms; For the first c Input feature sequences on each channel.
[0015] Furthermore, the precise alignment and distortion correction of stand spectral and vertical structural features guided by high-resolution canopy spatial texture features, based on a cross-modal guided attention mechanism, includes: Mapping canopy spatial texture features to query vectors Query The spectral and vertical structure features of the forest stand are stitched together along the channel dimension and then projected onto a linear layer. Key Vector sum Value Vector, and the query vector Query Attention is calculated to obtain an attention score, i.e.: In the formula, , ,N The sequence length of the canopy spatial texture features. The sequence lengths for the spectral and vertical structure characteristics of the forest stand; d For projection dimensions; Indicates position i The query vector; Indicates position j The key vector; This represents the attention score calculated using the scaled dot product; Introduce sparse attention Top-K Attention and base it on attention score. For each query position Only retain its relationship with Key Most relevant K each position token The expression is: In the formula, Indicates the first i Each query corresponds to Top-K Key location index; j Indicates position coordinates; exist Top-K Local execution within the region Softmax Normalization yields sparse attention weights, namely: In the formula, This represents the normalized sparse attention weights. i, j Both represent relative position coordinates in the feature map; j′ For temporary iteration indexes in the summation operation; In sparse attention weights Under the guidance of Value Dynamically weighted summation of vectors, and through Top-K The selection strategy ensures a more physically consistent spatial match between stand spectral characteristics and vertical structure characteristics. The formula for calculating the dynamic weighted summation is as follows: In the formula, It is a dual-modal feature; It is a value matrix.
[0016] Furthermore, the deep interactive fusion of spectral-texture-vertical structural features through hybrid attention blocks includes: Precise alignment and distortion-corrected bimodal features The feature maps are stitched together with the canopy spatial texture features, and then enhanced intramodally via the intramodal constraint module. Finally, they are fed into a hybrid attention block that integrates cross-modal guided attention mechanism and global self-attention mechanism for multimodal feature fusion. The global self-attention mechanism is used for the global interaction of multimodal forest stand features, stabilizing the consistent expression of global information, that is: In the formula, Output feature maps for global self-attention; These are the learnable linear projection matrices used to generate query, key, and value vectors, respectively. d For attention projection dimension; For multimodal embedding features; The cross-modal guided attention mechanism is used again, guided by canopy spatial texture features, to clarify the fine-grained synergistic relationships among multimodal stand features, thereby maximizing the mining of complementary information from cross-modal features. In the formula, These are dynamic weights used to highlight cross-modal complementary regions; P The spatial texture features of the canopy after pooling; M For spectral modal information; L This represents vertical structure information; ⊙ indicates element-wise multiplication; and These represent the key projection matrix and value projection matrix used for cross-modal feature interaction, respectively; This represents the fused and enhanced feature map obtained through a cross-modal guided attention mechanism, guided by canopy spatial texture features.
[0017] By employing the above technical solution, the present invention provides a multimodal alignment hierarchical fusion method for identifying forest tree species in spaceborne remote sensing, which has at least the following beneficial effects: Breaking through the bottleneck of multimodal heterogeneity fusion. This invention addresses the differences in imaging mechanisms, spatial resolution, and physical semantics between spaceborne multispectral, lidar, and high-resolution panchromatic images. It constructs a customized multimodal feature extraction and fusion framework, which can effectively alleviate the feature misalignment, information distortion, and semantic gap problems commonly found in existing methods, and significantly improve the collaborative expression capability of multi-source remote sensing information in complex forest backgrounds.
[0018] This invention enhances the utilization of vertical structure information and improves the ability to distinguish similar tree species. It introduces vertical structure features derived from spaceborne lidar and performs targeted modeling, effectively compensating for the inadequacy of traditional multispectral and two-dimensional texture information in expressing vertical differences in forest stands. This enhances the ability to identify spectrally similar tree species and improves the accuracy of tree species identification at the regional scale.
[0019] This invention achieves precise cross-modal alignment and hierarchical fusion, significantly improving recognition accuracy. Through a strategy of precise cross-modal alignment and hierarchical fusion, it effectively integrates spectral, textural, and vertical structural information, enhancing the accuracy and robustness of regional-scale dominant tree species identification. It has achieved superior classification results compared to existing methods in both temperate and subtropical study areas.
[0020] The synergistic gains of each module are significant, and the technical performance is stable and reliable. The three-branch feature extraction module, intramodal constraint module, and cross-modal hierarchical fusion module in this invention are respectively geared towards spectral modeling, vertical structure representation, spatial detail extraction, intramodal optimization, and cross-modal collaborative design. Ablation experiments show that removing any key module will lead to a decrease in overall accuracy, indicating that the components of this invention have clear division of labor and effective synergy, and can stably support tree species identification tasks in complex forest scenarios.
[0021] It exhibits good regional adaptability and application value. This invention achieves superior identification results compared to existing methods in both temperate and subtropical climate zones with significant differences, demonstrating its strong adaptability and generalization ability to different forest stand structures, tree species compositions, and ecological backgrounds. It can provide reliable technical support for forest resource surveys, dominant tree species mapping, ecological monitoring, and refined management. Attached Figure Description
[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a diagram of the overall network architecture of the multimodal alignment hierarchical fusion network in this invention; Figure 2 This is a diagram of the network structure of the scale-adaptive deformable convolution in this invention; Figure 3 This is a network structure diagram of the modal constraint module in this invention; Figure 4 This is a network structure diagram of the cross-modal hierarchical fusion module in this invention; Figure 5 This is an example diagram of the multimodal deep learning dataset used in this invention; Figure 6 This is a spatial comparison diagram of the classification results in temperate climate zones using different methods in this invention; Figure 7 This is a spatial comparison diagram of the classification results in the subtropical climate zone using different methods in this invention. Detailed Implementation
[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.
[0024] This embodiment addresses the challenges of existing multimodal deep learning models in adapting to the heterogeneity of remote sensing data and lacking precise alignment and hierarchical integration mechanisms. It proposes a multimodal alignment and hierarchical fusion method for identifying tree species in spaceborne remote sensing forests. For example... Figure 1 As shown, this invention designs a multimodal alignment and hierarchical fusion network (MAHFNet) specifically for the classification of dominant tree species. The MAHFNet fully considers the scene characteristics of different modal features, designs a three-branch feature extraction module, and constructs an intra-modal feature alignment (IMFA) module to mine intramodal local and global dependencies. Furthermore, it innovatively designs a two-stage fusion strategy based on a guided cross-attention (GCA) mechanism, realizing multi-level feature interaction and fusion of remote sensing features from multimodal forest stands, providing a new technical path for the accurate identification of dominant forest tree species. The method includes the following steps: This embodiment acquires satellite-borne multispectral data, lidar vertical structure data, and high-resolution panchromatic imagery of complex forest scenes within the target area. Multimodal remote sensing data is acquired using Sentinel-2 and Jilin-1 satellites. It can also selectively acquire satellite-borne multispectral imaging (MSI) and high-resolution panchromatic imagery (PAN) using the Landsat series, and lidar vertical structure data (Light Detection and Ranging, LiDAR) using airborne LiDAR.
[0025] This invention employs a multimodal alignment hierarchical fusion network to differentially model and extract multimodal forest stand features from satellite-borne multispectral data, lidar vertical structure data, and high-resolution panchromatic imagery. A two-stage fusion strategy is then used to achieve cross-modal hierarchical interactive fusion of these multimodal forest stand features, ultimately outputting the identification results of dominant forest tree species. By constructing a multimodal alignment hierarchical fusion network, this invention effectively addresses the problems of feature misalignment, semantic gaps, and insufficient fusion in tree species identification using multimodal remote sensing data, thereby improving the accuracy and stability of dominant tree species identification in complex forest scenarios at a regional scale.
[0026] In this embodiment, the constructed multimodal alignment hierarchical fusion network consists of a three-branch feature extraction module, an intramodal constraint module, and a cross-modal hierarchical fusion module, wherein: The three-branch feature extraction module is used to simultaneously extract differentiated stand spectral features, vertical structure features, and canopy spatial texture features from spaceborne multispectral data, LiDAR vertical structure data, and high-resolution panchromatic imagery. The module consists of an MSI feature extraction branch, a LiDAR vertical structure feature extraction branch, and a PAN spatial detail extraction branch.
[0027] The MSI feature extraction branch aims to extract differentiated spectral information from spaceborne multispectral data. To address the challenges of spectral variability and spatial heterogeneity in regional-scale tree species identification tasks, the MSI feature extraction branch employs a hierarchical feature extraction strategy that enhances global information representation while preserving local spectral details.
[0028] The MSI feature extraction branch utilizes Deeply Separable Convolutional (DSC) and the squeeze-activation mechanism SE-Block to enhance band independence and channel discriminativity. The MSI feature extraction branch uses DSC as the basic operator, effectively preserving sensitive information of specific spectral bands while significantly reducing the number of parameters by decoupling spatial filtering and channel transformation operations. This branch stacks three layers of DSC and combines them with BatchNorm normalization and ReLU activation functions to form a progressive spectral feature extraction mechanism, where: The first layer of depthwise separable convolutional DSC focuses on extracting original spectral features, the middle layer of depthwise separable convolutional DSC captures the interaction between bands, and the third layer of depthwise separable convolutional DSC deeply integrates joint spatial spectral features.
[0029] To further enhance the discriminative power of features, the MSI feature extraction branch introduces a squeeze-excitation mechanism, SE-Block. SE-Block adaptively enhances the spectral feature representation of important frequency bands and suppresses redundant channel information through global average pooling and channel weight adjustment. Furthermore, to preserve the spatial distribution characteristics of dominant tree species in the forest, the MSI feature extraction branch introduces Spatial Position Encoding (SPE). By embedding pixel coordinate information, the MAHFNet model can explicitly learn the spatial distribution patterns of different dominant tree species, thereby effectively improving classification accuracy.
[0030] The LiDAR vertical structure feature extraction branch extracts the vertical structural features of the forest stand through Vertical Separable Convolution (VSC), which is used to extract differentiated vertical structure information from the inverted spaceborne lidar GEDI data.
[0031] Spaceborne lidar GEDI can effectively acquire vertical structure characteristics of forest stands at the regional scale, such as canopy height, stratified leaf area index, and stratified leaf volume density. This is of great value in distinguishing tree species with similar spectral information but different vertical structures. Therefore, the design purpose of the LiDAR vertical structure feature extraction branch is to extract differentiated vertical structure information from the inverted spaceborne lidar GEDI data, providing additional information gain for the task of identifying dominant tree species in forests of different climatic zones. This branch is consistent with the MSI feature extraction branch in overall logic, but to better adapt to the vertical direction characteristics of spaceborne lidar GEDI data, this invention introduces a vertical direction information extraction mechanism into the traditional depthwise separable convolution module, and names it Vertical Separable Convolution (VSC).
[0032] Vertical Separating Convolution (VSC) is the core module of the LiDAR vertical structure information extraction branch. Compared to traditional depthwise separable convolution, the VSC designed in this invention uses a vertical convolution kernel to perform convolution operations along the vertical direction of the feature map. This reduces the number of computational parameters while improving the ability to extract local details of the vertical structure features of the forest stand. Its operation principle is as follows: In the formula, X The input feature map has a size of [size missing]. H×W×C ; Y The output feature map has the same size as the input feature map; For the first c Vertical convolution kernels with channels of size . K ×1; i, j To output feature map Y Spatial location coordinates, k It is the vertical position index within the convolution kernel; This is the offset of the center position of the convolution kernel, used to align the input and output feature maps.
[0033] Similar to the MSI feature extraction branch, the LiDAR vertical structure feature extraction branch employs a three-layer vertically separable convolutional structure. Each convolutional layer is followed by BatchNorm normalization and ReLU activation, and residual connections are introduced after each convolutional layer to alleviate the gradient vanishing problem. At the end of this branch, a squeeze-activation mechanism (SE-Block) is also introduced, and spatial distribution information of the vertical information feature map is preserved through spatial location encoding (SPE), providing stand vertical spatial semantic information for subsequent multimodal fusion.
[0034] The PAN spatial detail extraction branch is a module for extracting high-resolution spatial detail information. Its core task is to accurately capture the edge and texture details of tree species in forest stands from panchromatic images. Simultaneously, it leverages the high-resolution characteristics of the image to provide information guidance for subsequent feature alignment and fusion tasks. This branch dynamically captures canopy edges and spatial texture details through Scale-Adaptive Deformable Convolution (SADC), and introduces an adaptive average pooling layer after the SADC to achieve scale unification with other modal features.
[0035] Traditional fixed convolutional kernels extract features through a regular sampling grid, but their rigid structure makes it difficult to adapt to the irregular shapes and complex textures of forest features. Deformable convolution (DC) dynamically learns offsets to adaptively adjust the sampling position of the convolutional kernel, which can better capture the edge and texture information of forest tree species. However, the sampling offsets of traditional deformable convolution (DC) are directly generated through an additional convolutional layer. This generation method lacks scale adaptability and the receptive field is relatively fixed, which greatly limits the model's ability to extract features from forest stands at different scales in forest scenes.
[0036] To address the aforementioned issues, this invention innovatively introduces a lightweight scale-adaptive offset constraint mechanism based on the traditional deformable convolution (DC) architecture. The improved architecture is named Scale-Adaptive Deformable Convolution (SADC), and its flowchart is shown below. Figure 2 As shown, the Scale-Adaptive Deformable Convolutional Analog-Digital Convolutional (SADC) dynamically adjusts the receptive field to generate adaptive-scale sampling point offsets by using a learnable scale factor combined with a tanh nonlinear activation function. This allows for better capture of multi-scale features, ranging from individual trees to forest stand patches. This design effectively avoids the problem of fixed-scale sampling offsets and difficulty in extracting effective information increments in traditional deformable convolutions (DC) when there are large differences in resolution between PAN and MSI, and LiDAR.
[0037] The sampling point offset generation process of the scale-adaptive deformable convolutional SADC consists of two steps: First, the sampling point offset on the multi-scale feature map is calculated through the scale-aware module. The calculation process is as follows: In the formula, For the first s Feature maps at various scales; H and W These are the height and width of the feature map, respectively; X Input feature map; For the first s Channel descriptors of various scales; For the first s Learnable weight matrices at various scales; For the first s Bias terms at each scale; Indicates the first s Offset at each scale; Indicates the use of extracting the first i Convolution kernels for features at each scale; tanh It is the hyperbolic tangent activation function.
[0038] Secondly, a channel attention mechanism is used to calculate scale weights, and the offsets generated by feature maps at different scales are weighted and fused. This ensures that the sampling point offsets can effectively integrate the information from feature maps at various scales, allowing the convolutional kernel to focus more accurately and stably on regions in the high-resolution image that provide key spatial information about differences. The calculation process is as follows: In the formula, Spatial position coordinates are The output feature map; For the attention mechanism, there is a learnable weight vector; This is a bias term for the attention mechanism; This represents the sampling point offset after the fusion scale adaptive constraint mechanism. For sampling points at x directional offset; For sampling points at y directional offset; The sampling network representing the convolution kernel, k and l Represents the sampling grid The relative coordinates of the sampling points; For the convolution kernel at position The weights; A dummy variable for scale indexing; The channel descriptor represents the corresponding scale and is used to normalize the attention scores at different scales.
[0039] In forest scenes across different regions, tree species can exhibit significant differences in canopy edges and texture details. Scale-adaptive deformable convolutional SADCs can automatically adjust their convolution sampling positions under scale-adaptive constraints, providing effective spatial detail information for forest stands to aid in tree species identification.
[0040] Furthermore, in conventional multimodal fusion methods, PAN panchromatic images, due to their high resolution, tend to have excessively large feature maps, which then become the dominant modality during the fusion process, suppressing the effectiveness of other modalities. Therefore, pooling is necessary to ensure that the output feature map of the PAN branch is the same size as the feature maps of the MSI and LiDAR branches. Based on this idea, this invention introduces an adaptive average pooling layer after the scale-adaptive deformable convolutional SADC. This layer can automatically apply pooling kernels of different sizes according to resolution differences, thereby dynamically adjusting the feature map size. The calculation process is as follows: In the formula, Spatial position coordinates are The channel is The output feature map; For the corresponding spatial coordinates The range of the input feature map region is dynamically determined by the input-output size ratio; This indicates that the input feature map is at the location ,aisle The value of .
[0041] The intramodal constraint module is used to perform global-local joint optimization of multimodal forest stand features, enhancing local detail representation while maintaining global semantic consistency.
[0042] Traditional multimodal fusion methods often overlook local feature variations caused by factors such as topography, forest stand density, and forest age structure when extracting multimodal remote sensing features. This inconsistency in global information severely restricts the improvement of classification accuracy. To address this issue, this invention designs an Intramodal Constraint Module (IMFA) for intramodal forest stand feature alignment, such as... Figure 3 As shown. Its core idea is to replace the standard feedforward neural network layer by introducing local convolution operations in the Encoder of Vision Transformer (VIT), so as to effectively capture local feature changes within each modality of data while maintaining global semantic consistency.
[0043] The Vision Transformer (VIT) primarily relies on a multi-head self-attention mechanism to globally model modal information, but it struggles to accurately capture the complex local patterns and subtle differences present in remote sensing data across different modalities. To overcome this deficiency, this invention introduces a local depthwise convolution operation on top of the traditional Vision Transformer (VIT), forming a hybrid structure that can capture both global contextual information and effectively extract fine-grained local information, named the Intramodal Constraint Module (IMFA). Specifically, the Intramodal Constraint Module (IMFA) utilizes a multi-head self-attention mechanism to extract global feature dependencies and preserves the global spatial structure through relative position encoding. The calculation process is as follows: In the formula, The output feature map after self-attention weighting. These are the query, key, and value matrices, used for global attention calculation. The feature output is enhanced with relative position information; d For attention head dimension.
[0044] Subsequently, by introducing local depthwise convolution operations to replace the standard feedforward neural network layers, local correlations in the feature sequences are further mined while maintaining their nonlinear computational capabilities, effectively fusing feature information from neighboring locations. The calculation process is as follows: In the formula, For the first c Each channel is located in i The output feature values after convolution; This is a channel-specific convolution with a kernel size of [size missing]. K×K These are used for dimensionality increase and dimensionality decrease, respectively. i The output feature map contains the position coordinates. δ The GELU nonlinear activation function; k and k′ It is the relative position offset within the convolution kernel, used to define local regions; This is the offset of the center position of the convolution kernel, used to align the input and output feature maps; and For bias terms; For the first c Input feature sequences on each channel.
[0045] Through this design, the intramodal constraint module retains the global contextual understanding capability of Vision Transformer while better taking into account the expression needs of local detailed features, effectively improving the representation quality of each modal feature.
[0046] Due to sensor differences, remote sensing features from different modalities exhibit significant heterogeneity. Traditional methods typically employ simple stitching or shared weight fusion approaches, failing to consider the alignment and mapping of features between modalities, thus hindering effective interaction of multimodal forest stand remote sensing features. To address this, this invention innovatively proposes a Cross-modal Hierarchical Fusion (CMHF) module, such as... Figure 4 As shown.
[0047] The cross-modal hierarchical fusion module adopts a two-stage cross-modal fusion strategy. First, it uses high-resolution canopy spatial texture features as a guide and achieves accurate alignment and distortion correction of multimodal forest stand features based on the innovative cross-modal guided attention mechanism (GCA). Then, it completes the deep interactive fusion of spectral, texture and vertical structural features through hybrid attention blocks.
[0048] In the guided feature alignment and distortion correction corresponding to the first stage, this invention fully utilizes the high spatial resolution of panchromatic images and proposes a precise cross-modal feature alignment strategy guided by canopy spatial texture features. Specifically, this invention first maps canopy spatial texture features into query vectors. Query The spectral and vertical structure features of the forest stand are stitched together along the channel dimension and then projected onto a linear layer. Key Vector sum Value Vector, query vector generated from canopy spatial texture features Query Attention is calculated. It is obtained from the scaled dot product. Q and K Matrix attention scoring, i.e.: In the formula, , , N The sequence length of the canopy spatial texture features. The sequence lengths for the spectral and vertical structure characteristics of the forest stand; d For projection dimensions; Indicates position i The query vector; Indicates position j The key vector; This represents the attention score calculated using the scaled dot product.
[0049] To avoid irrelevant regions interfering with attention aggregation and to reduce computational overhead, this invention introduces sparse attention (Top-K Attention) on top of this cross-modal guidance. For each query position... Only retain its relationship with Key Most relevant K each position token The expression is: In the formula, Indicates the first i Each query corresponds to Top-K Key location index; j Indicates the position coordinates.
[0050] Then Top-K Local execution within the region Softmax Normalization yields sparse attention weights, namely: In the formula, This represents the normalized sparse attention weights. i, j Both represent relative position coordinates in the feature map; j′ For temporary iterative indexes in the summation operation.
[0051] In the calculated sparse attention weights Under guidance, Value The vectors are further dynamically weighted and summed, and then... Top-K The chosen strategy ensures a more physically consistent spatial match between stand spectral and vertical structure characteristics. The formula for calculating the dynamic weighted summation is: In the formula, It is a bimodal feature, obtained from the precisely aligned spectral and vertical structure features of the forest stand; For value matrices, i, j These are the relative position coordinates in the feature map.
[0052] Unlike traditional fully connected cross-modal attention, the cross-modal guided attention mechanism (GCA) introduces a sparse attention strategy, avoiding the indiscriminate aggregation of stand spectral and vertical structural features. This improves computational efficiency while suppressing noise interference caused by sensor differences and background complexity. Through this design, the cross-modal guided attention mechanism (GCA) can more accurately capture the correspondence between stand spectral and vertical structural features, achieving fine-grained cross-modal geometric alignment and distortion correction.
[0053] In the second stage of cross-modal collaborative fusion, this invention further achieves cross-modal collaborative fusion after completing precise spatial alignment. In this stage, the bimodal features precisely aligned in the previous stage are first... Feature maps are concatenated with canopy spatial texture features, then intramodal enhancement is performed via an intramodal constraint module (IMFA), and finally fed into a hybrid attention block integrating cross-modal guided attention and global self-attention mechanisms for multimodal feature hierarchical fusion. Global self-attention is used for global interaction of multimodal forest stand features, stabilizing the consistent representation of global information, i.e.: In the formula, Output feature maps for global self-attention; These are the learnable linear projection matrices used to generate query, key, and value vectors, respectively. d For attention projection dimension; This is a multimodal embedding feature.
[0054] Cross-modal guided attention is again guided by canopy spatial texture features to further clarify the fine-grained synergistic relationships among multimodal stand features, thereby maximizing the mining of complementary information from cross-modal features, namely: In the formula, These are dynamic weights used to highlight cross-modal complementary regions; P The spatial texture features of the canopy after pooling; M For spectral modal information; L This represents vertical structure information; ⊙ indicates element-wise multiplication; d For attention projection dimension; and These represent the key projection matrix and value projection matrix used for cross-modal feature interaction, respectively; This represents the fused and enhanced feature map obtained through a cross-modal guided attention mechanism, guided by canopy spatial texture features.
[0055] The alternating application of two attention mechanisms allows the fusion process to maintain the independence of modal features while dynamically emphasizing important interactions between modalities.
[0056] Through the aforementioned cross-modal precise alignment and cross-modal collaborative enhancement strategies, the cross-modal hierarchical fusion module proposed in this invention can effectively solve the problems of heterogeneous feature misalignment and feature fusion difficulties that are common in existing remote sensing multimodal deep learning methods, and has good model generalization ability.
[0057] This embodiment verifies and evaluates the accuracy of the spaceborne remote sensing forest tree species identification method proposed in this invention using field measurement data, as detailed below: 1. Field measurement dataset.
[0058] This invention conducted four field surveys of dominant tree species in the temperate and subtropical study areas in September 2022, June 2023, May 2024, and July 2024. Representative forest stands were selected based on the distribution characteristics of different dominant tree species in the forests. During the selection process, factors such as forest age structure, understory coverage, and stand density were comprehensively considered, ensuring that the proportion of a single dominant tree species in the selected stands was not less than 80%. Furthermore, to ensure spatial representativeness and accuracy of remote sensing image matching, the selected stands were all larger than 30m × 30m, and the minimum distance from the center point to the boundary was greater than 10m. The geographic coordinates of the center point of each stand were recorded using a GPS device, Mobile Mapper 6, with a positioning accuracy of 2.5m. Ultimately, this invention obtained 1200 typical tree species plots and 609 non-forest plots in the temperate study area; and 1981 typical dominant tree species plots and 831 non-forest plots in the subtropical study area, as shown in Table 1.
[0059] Table 1 Distribution of the number of sample plots of dominant tree species in the study area 2. Multimodal deep learning dataset.
[0060] Based on data acquisition and preprocessing, this invention integrates Sentinel-2 data, Jilin-1 L1 panchromatic data, and GEDI vertical structure data to construct a multimodal deep learning dataset for tree species classification. The Sentinel-2 data uses autumn imagery, containing 10 spectral bands and 5 vegetation indices. For the GEDI-derived vertical structure features, discrete GEDI spots are first interpolated and inverted, and then vertical structure features with high sensitivity to autumn tree species identification are selected for multimodal classification.
[0061] Simultaneously, this invention combines field measurement datasets with 2019 forest resource inventory data to select typical and representative areas within the temperate and subtropical study regions, creating high-quality classification labels with a spatial resolution of 10m for use in multimodal deep learning model training and classification applications, such as... Figure 5 As shown in Table 2, the constructed dataset has a size of 100×100 and consists of 22 sample blocks, covering 11 common dominant tree species and forest types in temperate and subtropical study areas. The temperate study area contains 101,570 effective tree species pixels, and the subtropical study area contains 86,396 effective tree species pixels. After subsequent data augmentation, the data size is sufficient to meet the needs of multimodal deep learning model training, validation, and inference. The dataset is divided into training and testing sets in a 6:4 ratio for model training and performance optimization.
[0062] Furthermore, to address the issue of uneven distribution of sample numbers for different tree species in real-world scenarios, this invention implements data augmentation processing on the constructed spaceborne multimodal remote sensing dataset. The data augmentation methods include random flipping, random radiometric enhancement, and mixed noise enhancement. Random flipping increases the spatial variability of the samples; random radiometric enhancement simulates the observation effects under different lighting conditions by adjusting radiometric characteristics such as brightness and contrast; and mixed noise enhancement expands the intra-class feature distribution of each tree species through weighted superposition of similar pixels and noise injection, thereby mitigating model bias caused by uneven label numbers. The combined application of these multiple data augmentation strategies effectively improves the model's adaptability to multimodal remote sensing data, further enhancing its generalization ability and stability in spaceborne forest tree species identification tasks in real-world scenarios.
[0063] Table 2. Distribution of the number of labels for each tree species in the multimodal deep learning dataset. 3. Implementation details.
[0064] This experiment was implemented in the deep learning framework PyTorch 2.1.0 and Python 3.11 environment, relying on an NVIDIA GeForce RTX 4090 graphics card (24GB VRAM) and CUDA version 11.8. During model training, a dropout rate of 0.1 was set, the Cross Entropy Loss loss function was used, and the initial learning rate was 1e-4, gradually reducing the learning rate using a cosine annealing learning rate decay strategy. The Adam optimizer was used, with a total of 500 batches, a batch size of 64, and a patch size of 16. To ensure the reproducibility of the results, the random seed was set to 14 in all experiments.
[0065] Furthermore, this invention employs multiple evaluation metrics to assess the accuracy and stability of the constructed MAHFNet model in forest dominant tree species classification tasks across different regions. These metrics include overall accuracy (OA), the Kappa coefficient, and the F1 score for each category. Overall accuracy characterizes the proportion of correctly classified tree species across all samples, reflecting the overall performance in regional-scale tree species classification tasks. The Kappa coefficient calculates the consistency between the model's predictions and the true labels based on the confusion matrix, excluding the influence of random classification factors. The F1 score, considering both precision and recall, is used to evaluate the classification performance of each category individually. These evaluation metrics effectively verify the classification performance, generalization ability, and stability of the constructed model under different categories and dataset conditions.
[0066] 4. Experimental results.
[0067] To verify the effectiveness of the MAHFNet constructed in this invention, a variety of advanced multimodal remote sensing models covering CNN, 3D convolutional, and Transformer architectures were selected for comparative analysis. Specifically, these included: MultiSenseSeg is a unified multimodal perceptual semantic segmentation model that models intramodal and intermodal relationships through modality-specific experts, an adaptive multimodal matching module, and a unified feature extraction framework, and has good modality expansion capabilities.
[0068] ISSP-Net: An interactive spatial-spectral sensing network that achieves joint modeling of multimodal spatial and spectral features through pixel-guided spatial enhancement, cross-spatial aggregation learning, and time-frequency co-spectral enhancement mechanisms.
[0069] DSTD-Net: A two-stream Transformer network that uses self-attention and co-attention modules to extract multimodal features, and combines differential attention enhancement and multi-level gating fusion strategies to achieve weighted fusion of features from different modalities.
[0070] FtransUNet: A multi-level, multi-modal fusion model that combines convolutional neural networks and visual Transformers. It enhances cross-modal representation capabilities by alternately fusing shallow and deep features and employing a multi-stage attention mechanism.
[0071] AMS-3DCNN: An adaptive multi-scale 3D convolutional network that extracts multimodal remote sensing features through multi-scale 3D convolution and enhances feature space consistency by utilizing non-local similarity sample constraints, making it suitable for complex land cover classification tasks.
[0072] By comparing with the methods described above, the technical advantages of this invention can be comprehensively evaluated in terms of multimodal feature extraction, cross-modal alignment, hierarchical fusion, and classification performance.
[0073] Tables 3 and 4 show the classification accuracy, overall accuracy (OA), Kappa coefficient, and F1 score of each category for different methods on the two datasets, with the best results highlighted in bold.
[0074] Figure 6 and Figure 7Spatial comparison of classification results of different methods in temperate and subtropical zones. Among them, (a) represents the ground truth label; (b) represents the classification result of MultiSenseSeg; (c) represents the classification result of ISSP-Net; (d) represents the classification result of DSTD-Net; (e) represents the classification result of FtransUNet; (f) represents the classification result of AMS-3DCNN model; and (g) represents the classification result of MAHFNet constructed in this invention.
[0075] Table 3 Comparison of accuracy of various models in the temperate study area Table 4 Comparison of Accuracy of Various Models in the Subtropical Study Area Experimental results show that the proposed MAHFNet achieves better recognition performance than the comparison model in both temperate and subtropical study areas, indicating that the present invention has good technical advantages in multimodal feature extraction, cross-modal alignment and hierarchical fusion, and can effectively improve the accuracy and stability of regional forest dominant tree species identification.
[0076] In the temperate study area, MAHFNet achieved the highest overall accuracy and Kappa coefficient, and obtained optimal or near-optimal classification results for some typical tree species, indicating that the present invention can effectively integrate multispectral, vertical structure, and high-resolution texture information in scenarios with relatively complete forest stands and concentrated vegetation types. In contrast, while other comparative models have certain capabilities in local feature extraction or multi-level fusion, they still have shortcomings in multimodal heterogeneous feature alignment, patch boundary preservation, and classification balance.
[0077] In subtropical study areas, the difficulty of tree species identification is further increased due to the more complex forest community structure, more diverse tree species composition, and more pronounced canopy overlap. Despite this, MAHFNet still achieved the highest overall accuracy and Kappa coefficient, and performed exceptionally well across multiple tree species categories, demonstrating that the present invention still possesses good adaptability and robustness in complex ecological contexts. In contrast, other comparative models are relatively insufficient in expressing fine-grained differential features and tree species boundary information in high-coverage scenarios, resulting in overall identification performance inferior to the present invention.
[0078] In summary, this invention effectively improves the overall accuracy of dominant tree species identification in complex forest scenarios in temperate and subtropical regions through differentiated multimodal feature extraction, intramodal constraints, and precise cross-modal alignment and hierarchical fusion, providing reliable technical support for regional-scale forest resource surveys and dominant tree species mapping.
[0079] 5. Ablation analysis.
[0080] To fully exploit the complementary information of different modalities in multimodal remote sensing data, this invention constructs the MAHFNet model. Specifically, feature extraction modules are designed for MSI, LiDAR, and PAN modalities to extract feature information from each modality. The IMFA module enhances the local detail representation and global context modeling capabilities of single-modal features by combining ViT's global attention mechanism with local convolution operations. The CMHF module, based on GCA-guided cross-modal attention mechanism, employs a two-stage hierarchical fusion strategy to achieve interactive fusion of forest stand features from different modalities.
[0081] To verify the effectiveness of each module in the multimodal feature extraction and fusion process of MAHFNet, this invention conducts ablation experiments by removing or replacing key modules to quantify the contribution of each module to the overall model performance. Specifically, this includes: replacing the MSI feature extraction branch with ordinary convolution to verify the role of depthwise separable convolution and the squeeze-excitation mechanism in spectral feature extraction; replacing the LiDAR vertical structure feature extraction branch with ordinary convolution to verify the role of vertically separable convolution in vertical structure feature extraction; replacing the PAN spatial detail extraction branch with ordinary convolution and downsampling to verify the role of deformable convolution based on scale-adaptive offset constraints in high-resolution spatial detail capture; removing the intramodal constraint module IMFA and directly using the original multimodal features for cross-modal fusion to verify the role of IMFA in intramodal feature enhancement; and removing the cross-modal hierarchical fusion module CMHF, directly concatenating the multimodal features and fusing them using a standard Transformer to verify the role of the constructed cross-modal hierarchical fusion module in multimodal alignment and cross-modal semantic association.
[0082] The ablation experiments described above were labeled with w / o MSI, w / o LiDAR, w / o PAN, w / o IMFA, and w / o CMHF, respectively, and the results are shown in Tables 5 and 6. These experiments further illustrate the functional contributions and synergistic effects of each module of this invention in the multimodal classification task of dominant forest tree species.
[0083] Table 5 Ablation Experiment Results of Each Module in the Temperate Study Area Table 6 Ablation Experiment Results of Each Module in the Subtropical Study Area The results show that each module contributes positively to the final recognition accuracy, indicating that the multimodal alignment hierarchical fusion framework constructed in this invention has good synergy and stability.
[0084] In the temperate study area, the overall accuracy decreased most significantly after replacing the MSI feature extraction branch, indicating that the multispectral branch plays a crucial role in extracting spectral information for tree species identification. Replacing the LiDAR vertical structure feature extraction branch resulted in varying degrees of accuracy decrease for all tree species, demonstrating that vertical structure information effectively enhances the ability to distinguish between different tree species. Replacing the PAN spatial detail extraction branch further reduced overall accuracy, indicating that high-resolution spatial texture information plays an important auxiliary role in representing canopy boundaries and patch details. Meanwhile, removing the intramodal constraint module IMFA weakens the intramodal local-global feature optimization capability, while removing the cross-modal hierarchical fusion module CMHF reduces cross-modal alignment and fusion effects, indicating that both play important roles in improving overall recognition performance.
[0085] In the subtropical study area, the ablation results were consistent with the overall pattern observed in the temperate study area. Replacing the MSI feature extraction branch resulted in the largest decrease in overall accuracy, indicating that multispectral information remains the primary and effective source of information for tree species identification in complex forest contexts. Replacing the LiDAR vertical structure feature extraction branch led to a significant decrease in accuracy for some broadleaf tree species, demonstrating the crucial role of vertical structure information in distinguishing tree species with similar spectra but different structures. Replacing the PAN spatial detail extraction branch further reduced overall accuracy, indicating that spatial texture information remains indispensable for representing complex canopy morphology. Furthermore, removing the intramodal constraint module IMFA and the cross-modal hierarchical fusion module CMHF both resulted in a decrease in model performance, with the removal of the cross-modal hierarchical fusion module CMHF causing a more significant decrease in accuracy, indicating that accurate cross-modal alignment and hierarchical fusion are key aspects of achieving multimodal collaborative identification in this invention.
[0086] Overall, the ablation experiment results verified the rationality of the design of each module in this invention. The MSI feature extraction branch, the LiDAR vertical structure feature extraction branch, and the PAN spatial detail extraction branch are responsible for spectral, vertical structure, and spatial texture feature extraction tasks, respectively. The intramodal constraint module IMFA is responsible for intramodal optimization, and the cross-modal hierarchical fusion module CMHF is responsible for cross-modal alignment and hierarchical fusion. All parts work together to ensure the accuracy and stability of tree species identification in complex forest scenes.
[0087] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0088] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Since the above embodiments are substantially similar to the method embodiments, their descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0089] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A multimodal alignment hierarchical fusion method for identifying forest tree species in spaceborne remote sensing, characterized in that, The method includes the following steps: Acquire multimodal remote sensing data, including spaceborne multispectral data, lidar vertical structure data, and high-resolution panchromatic imagery, for complex forest scenes within the target area; A multimodal alignment hierarchical fusion network is used to perform differentiated modeling on multimodal remote sensing data to extract multimodal forest stand features. A two-stage fusion strategy is adopted to achieve cross-modal hierarchical interactive fusion of multimodal forest stand features, and finally output the forest dominant tree species identification results. The multimodal alignment hierarchical fusion network includes: A three-branch feature extraction module is used to simultaneously extract differentiated stand spectral features, vertical structure features, and canopy spatial texture features from spaceborne multispectral data, lidar vertical structure data, and high-resolution panchromatic images. In addition, an intramodal constraint module is used to perform global-local joint optimization of multimodal forest stand features, which enhances the expression of local details while maintaining global semantic consistency; Furthermore, a cross-modal hierarchical fusion module is developed that uses high-resolution canopy spatial texture features as a guide, performs precise alignment and distortion correction of forest stand spectral and vertical structure features based on a cross-modal guided attention mechanism, and then completes the deep interactive fusion of spectral, texture and vertical structure features through hybrid attention blocks.
2. The method for identifying forest tree species using spaceborne remote sensing according to claim 1, characterized in that, The three-branch feature extraction module includes: A branch for extracting MSI features of forest stand spectral characteristics is developed by using depthwise separable convolution (DSC) and squeeze-excitation mechanism (SE-Block) to enhance band independence and channel discriminability. In addition, by introducing Vertical Separation Convolution (VSC) as a mechanism for extracting vertical direction information of forest stands, a LiDAR vertical structure feature extraction branch is used to extract differentiated vertical structure information of forest stands from the inverted spaceborne lidar GEDI data to obtain vertical structure features. Furthermore, a scale-adaptive deformable convolutional SADC is used to dynamically capture canopy edges and spatial texture details. An adaptive average pooling layer is introduced after the scale-adaptive deformable convolutional SADC to achieve a PAN spatial detail extraction branch that obtains canopy spatial texture features in a unified scale with other modal features.
3. The method for identifying forest tree species using spaceborne remote sensing according to claim 2, characterized in that, The depthwise separable convolutional (DSC) is configured as three layers stacked sequentially, and works in conjunction with BatchNorm normalization and ReLU activation function to form a progressive spectral feature extraction mechanism, wherein: The first layer of depthwise separable convolutional DSC focuses on extracting raw spectral features, the middle layer of depthwise separable convolutional DSC captures the interaction between bands, and the third layer of depthwise separable convolutional DSC deeply integrates joint spatial spectral features. The squeeze-excitation mechanism SE-Block adaptively enhances the spectral feature representation of important frequency bands and suppresses redundant channel information through global average pooling and channel weight adjustment.
4. The method for identifying forest tree species using spaceborne remote sensing according to claim 2, characterized in that, The MSI feature extraction branch also includes the introduction of spatial location coding (SPE), which is used to embed pixel coordinate information and explicitly learn the spatial distribution patterns of different dominant tree species.
5. The method for identifying forest tree species using spaceborne remote sensing according to claim 2, characterized in that, The Vertical Separable Convolution (VSC) adopts a three-layer vertically separable convolution structure, with each convolution layer followed by BatchNorm normalization and ReLU activation function, and residual connections introduced after each convolution layer. The vertically separated convolution (VSC) introduces a squeeze-excitation mechanism (SE-Block) at the end and preserves the spatial distribution information of the vertical information feature map through spatial position encoding (SPE). The Vertical Separating Convolution (VSC) uses a vertical convolution kernel to perform convolution operations along the vertical direction of the feature map. This is done to improve the ability to extract local details of vertical structural features while reducing the number of computational parameters. The expression is as follows: ; In the formula, X Input feature map; Y Output feature map; For the first c Vertical convolution kernels with channels of size . K ×1; i、 j To output feature map Y Spatial location coordinates; k It is the vertical position index within the convolution kernel; This represents the offset of the center position of the convolution kernel.
6. The method for identifying forest tree species using spaceborne remote sensing according to claim 2, characterized in that, The scale-adaptive deformable convolutional SADC dynamically adjusts the receptive field by using a learnable scale factor and a tanh nonlinear activation function to generate adaptive scale sampling point offsets, which are used to capture multi-scale features from single trees to stand patches. The generation of the sampling point offset includes: The offset on the multi-scale feature map is calculated using the scale-aware module. The calculation process is as follows: ; In the formula, For the first s Feature maps at various scales; H and W These are the height and width of the feature map, respectively; X Input feature map; For the first s Channel descriptors of various scales; For the first s Learnable weight matrices at various scales; For the first s Bias terms at each scale; Indicates the first s Offset at each scale; Indicates the use of extracting the first i Convolution kernels for features at each scale; tanh It is the hyperbolic tangent activation function; The scale weights are calculated using a channel attention mechanism, which accounts for the offsets generated by feature maps at different scales. Weighted fusion is performed to obtain the sampling point offset. The calculation process is as follows: ; In the formula, Spatial position coordinates are The output feature map; For the attention mechanism, there is a learnable weight vector; This is a bias term for the attention mechanism; This represents the sampling point offset after the fusion scale adaptive constraint mechanism. For sampling points at x directional offset; For sampling points at y directional offset; The sampling network representing the convolution kernel, k and l Represents the sampling grid The relative coordinates of the sampling points; For the convolution kernel at position The weights; A dummy variable for scale indexing; The channel descriptor represents the corresponding scale.
7. The method for identifying forest tree species using spaceborne remote sensing according to claim 2, characterized in that, The adaptive average pooling layer is used to automatically apply pooling kernels of different sizes according to the resolution difference to dynamically adjust the size of the feature map. The expression is: ; In the formula, Spatial position coordinates are The channel is The output feature map; For the corresponding spatial coordinates The input feature map region; This indicates that the input feature map is at the location ,aisle The value of .
8. The method for identifying forest tree species using spaceborne remote sensing according to claim 1, characterized in that, The intramodal constraint module introduces local depthwise convolution operations into the Encoder of the Vision Transformer to replace the standard feedforward neural network layer, thereby capturing local feature changes within multimodal forest stand features while maintaining global semantic consistency; the specific process is as follows: Global feature dependencies are extracted using a multi-head self-attention mechanism, and the global spatial structure is preserved through relative position encoding, as expressed in the following expression: ; In the formula, The output feature map after self-attention weighting. These are the query, key, and value matrices, respectively. The feature output is enhanced with relative position information; d For attention head dimension; By introducing local depthwise convolution operations to replace the standard feedforward neural network layers, feature information from neighboring locations is fused. The expression is as follows: ; In the formula, For the first c Each channel is located in i The output feature value after convolution; Channel-specific convolution; i The output feature map contains the position coordinates. δ The GELU nonlinear activation function; k and k′ It is the relative position offset within the convolution kernel; This represents the offset of the center position of the convolution kernel; and For bias terms; For the first c Input feature sequences on each channel.
9. The method for identifying forest tree species using spaceborne remote sensing according to claim 1, characterized in that, The method, guided by high-resolution canopy spatial texture features and based on a cross-modal guided attention mechanism, precisely aligns and corrects distortions in stand spectral and vertical structural features, including: Mapping canopy spatial texture features to query vectors Query The spectral and vertical structure features of the forest stand are stitched together along the channel dimension and then projected onto a linear layer. Key Vector sum Value Vector, and the query vector Query Attention is calculated to obtain an attention score, i.e.: ; In the formula, , , N The sequence length of the canopy spatial texture features. The sequence lengths for the spectral and vertical structure characteristics of the forest stand; d For projection dimensions; Indicates position i The query vector; Indicates position j The key vector; This represents the attention score calculated using the scaled dot product; Introduce sparse attention Top-K Attention and base it on attention score. For each query position Only retain its relationship with Key Most relevant K each position token The expression is: ; In the formula, Indicates the first i Each query corresponds to Top-K Key location index; j Indicates position coordinates; exist Top-K Local execution within the region Softmax Normalization yields sparse attention weights, namely: ; In the formula, This represents the normalized sparse attention weights. i, j Both represent relative position coordinates in the feature map; j′ For temporary iteration indexes in the summation operation; In sparse attention weights Under the guidance of Value Dynamically weighted summation of vectors, and through Top-K The selection strategy ensures a more physically consistent spatial match between stand spectral characteristics and vertical structure characteristics. The formula for calculating the dynamic weighted summation is as follows: ; In the formula, It is a dual-modal feature; It is a value matrix.
10. The spaceborne remote sensing forest tree species identification method according to claim 9, characterized in that, The deep interactive fusion of spectral, texture, and vertical structural features achieved through hybrid attention blocks includes: Precise alignment and distortion-corrected bimodal features The feature maps are stitched together with the canopy spatial texture features, and then enhanced intramodally via the intramodal constraint module. Finally, they are fed into a hybrid attention block that integrates cross-modal guided attention mechanism and global self-attention mechanism for multimodal feature fusion. The global self-attention mechanism is used for the global interaction of multimodal forest stand features, stabilizing the consistent expression of global information, that is: ; In the formula, Output feature maps for global self-attention; These are the learnable linear projection matrices used to generate query, key, and value vectors, respectively. d For attention projection dimension; For multimodal embedding features; The cross-modal guided attention mechanism is used again, guided by canopy spatial texture features, to clarify the fine-grained synergistic relationships among multimodal stand features, thereby maximizing the mining of complementary information from cross-modal features. ; In the formula, These are dynamic weights used to highlight cross-modal complementary regions; P The spatial texture features of the canopy after pooling; M For spectral modal information; L This represents vertical structure information; ⊙ indicates element-wise multiplication; and These represent the key projection matrix and value projection matrix used for cross-modal feature interaction, respectively; This represents the fused and enhanced feature map obtained through a cross-modal guided attention mechanism, guided by canopy spatial texture features.
Citation Information
Patent Citations
Advanced tree species identification method based on time-space-spectrum multi-domain perception and guidance
CN119919817A
Method and system for identifying forest stand tree species based on multi-source remote sensing data fusion
CN120544052A
Method and system for estimating forest carbon storage
US20250283978A1
