A deep and shallow dual-branch super-resolution method for forest hyperspectral satellite images

Abstract

The present invention relates to the field of satellite-borne hyperspectral image processing and analysis technology, solving the technical problem that existing methods fail to fully consider the huge spectral and spatial resolution differences between low-resolution hyperspectral data and high-resolution multispectral data. In particular, it relates to a deep and shallow dual-branch super-resolution method for forest hyperspectral satellite imagery, comprising constructing a dual-branch network architecture, performing feature fusion reconstruction on the features output, and obtaining high-resolution hyperspectral data with low-spectral variation characteristics that characterize forest vegetation in satellite-borne hyperspectral imagery data. The present invention effectively solves the modal difference problem between low-resolution hyperspectral and high-resolution multispectral data by learning at different feature levels and scales, and through multiple feature attention mechanisms, enables the model to pay more attention to the low-spectral variation characteristics of different forest vegetation in satellite imagery, which can meet the requirements of subsequent various forest resource precision monitoring tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite-borne hyperspectral image processing and analysis, and in particular to a deep-shallow dual-branch super-resolution method for forest hyperspectral satellite images. Background Art

[0002] Due to the characteristics of its imaging mechanism, spaceborne hyperspectral remote sensing is often limited by its inherent spatial resolution, hindering its application in regional fine-grained surface monitoring tasks. In recent years, deep learning-based hyperspectral image super-resolution technology has demonstrated promise in overcoming this obstacle. From the perspective of data sources, current hyperspectral image super-resolution methods can be divided into two categories: single hyperspectral image super-resolution and fusion-based hyperspectral image super-resolution. Single hyperspectral image super-resolution often has limitations in capturing the true details corresponding to high-resolution imagery. In contrast, fusion-based hyperspectral image super-resolution can effectively combine the complementary advantages of information from different data sources. Methods such as panchromatic fusion and multispectral fusion have achieved promising results in the field of super-resolution. Among them, super-resolution using multispectral-assisted hyperspectral imagery has become one of the most widely used methods due to its greater potential to generate more realistic high-resolution hyperspectral data.

[0003] However, existing methods focus more on the variability of overall image quality before and after super-resolution, and fail to fully consider the specific usage scenarios of subsequent data. For example, in the field of forest resource monitoring, people tend to focus more on the variability of the spectral reflectance characteristics of data pixels. This is because forests, as the most complex ecosystem on land, have very complex background conditions. In addition, due to the extremely small differences in spectral characteristics between the canopies of different tree species, the "different objects with the same spectrum" phenomenon within the forest is extremely serious, which will seriously hinder the use of satellite remote sensing images for accurate monitoring of forest resources. In contrast, for routine tasks such as land resource monitoring, due to the significant spectral differences between impervious surfaces, bare land, water bodies and other land objects, the requirements for spectral variability after super-resolution are not high. At the same time, in many hyperspectral image super-resolution tasks based on multispectral fusion, existing methods fail to fully consider the huge spectral and spatial resolution differences between low-resolution hyperspectral data (LR HSI) and high-resolution multispectral data (HR MSI). This significant difference in modality makes it difficult for existing super-resolution tasks to effectively deal with the variability of spatial and spectral characteristics of forest vegetation satellite images. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a deep and shallow dual-branch super-resolution method for forest hyperspectral satellite imagery, which solves the technical problem that the existing methods fail to fully consider the huge spectral and spatial resolution differences between low-resolution hyperspectral data and high-resolution multispectral data.

[0005] To address the above technical issues, the present invention provides the following technical solution: a deep-shallow dual-branch super-resolution method for forest hyperspectral satellite imagery. Specifically, a dual-branch network architecture is used to extract and fuse multiple cross-modal spatial and spectral features on both deep and shallow features, achieving high-precision super-resolution reconstruction of forest vegetation hyperspectral satellite remote sensing images. The method comprises the following steps:

[0006] Acquire satellite-borne hyperspectral image data for forest vegetation monitoring, including low-resolution hyperspectral data and high-resolution multispectral data;

[0007] Construct a dual-branch network architecture designed for the super-resolution task of fusion of hyperspectral and multispectral data, including a deep feature extraction branch for extracting deep encoding features from hyperspectral and multispectral data, and interactively extracting spatial spectral information at both global and local scales, effectively capturing the long-range dependencies in the spectral dimension and the diversity of local features in the spatial dimension;

[0008] Furthermore, on the one hand, a multi-scale grouping cross-modal attention module is used to group the channel dimensions of hyperspectral data to obtain spectral features of local spectral feature details, and multi-scale convolution is used for multispectral data to obtain multiple spatial features, and cross-modal fusion between the two is achieved through feature matching; on the other hand, a cross-modal feature interaction module is combined to interactively fuse spatial features and spectral features, realizing a shallow feature extraction branch that jointly represents spatial and spectral features;

[0009] The feature fusion module is used to reconstruct the deep features and shallow features output by the dual-branch network architecture to obtain high-resolution hyperspectral data with low spectral variation characteristics that characterize forest vegetation in spaceborne hyperspectral image data.

[0010] Furthermore, the deep feature extraction branch includes a lightweight residual Linformer feature extractor for extracting spatial spectral information from hyperspectral data and multispectral data respectively;

[0011] Furthermore, the hyperspectral and multispectral deep encoding features output by the lightweight residual Linformer feature extractor are combined with the hyperspectral data through basic convolution blocks to extract spatial spectral features. These features are then subjected to spatial attention and spectral attention calculations in sequence, which reduce the weights of irrelevant areas and emphasize the importance of certain band channels on a global scale. Finally, the global spatial spectral feature attention module is implemented to achieve deep feature mining and effective fusion through attention weight calculation.

[0012] Furthermore, the hyperspectral and multispectral deep encoding features output by the lightweight residual Linformer feature extractor are combined with the hyperspectral data through the basic convolutional block to extract the spatial spectral features, and then the cross attention calculation is performed on the local scale and expanded in the spatial dimension with the attention weight W attn Perform deconvolution operation to obtain high-resolution fusion feature F attn_out Local cross-scale feature attention module for feature images.

[0013] Furthermore, the lightweight residual Linformer feature extractor includes a Linformer encoder for extracting spatial spectral information, and a residual convolution module for enhancing the expression capability of spatial spectral information through convolution operations and residual connections.

[0014] Furthermore, the expression of the global spatial spectral feature attention module is:

[0015]

[0016] Among them, X represents the three feature maps output by the basic convolution block; Conv(·) represents the two-dimensional convolution operation; σ(·) represents the Sigmoid activation function; W1 and W2 are the linear transformation parameter matrices respectively; b1 and b2 are bias terms; δ(·) is the ReLU activation function; Q attn , K attn 、V attn Represents the output of the three feature maps after spatial attention and spectral attention, that is, X ss ; Represents the convolution operation; * T represents the deconvolution operation; X attn Represents the fused features obtained after deconvolution operation.

[0017] Furthermore, the calculation process of the local cross-scale feature attention module includes:

[0018] First, the input image feature F DMS 、F DHS 、F HS ∈R N×C×H×WPerform convolution feature transformation and use it as features Q, K, and V respectively. The expressions are:

[0019]

[0020] Where, Conv q (·), Conv k (·), Conv v (·) indicates that convolution operations are performed on the three input features respectively; R N×C×H×W Indicates that the characteristic shape is N×C×H×W; Indicates that the characteristic shapes are N×C mid ×H×W and N×C hs ×H×W;

[0021] Secondly, K and V are divided into several small blocks through the feature extraction block function ExtractImagePatches, and the feature blocks K and V are obtained respectively. patches ∈R N×L×C×k×k and V patches ∈R N×L×C×k×k , the expression is;

[0022] K / V patches =ExtractImagePatches(K / V,ks,stride)

[0023] Among them, L is the total number of blocks; k is the size of each small block; ks is the convolution kernel size; stride is the sliding step size; R N×L×C×k×k Indicates that the characteristic shape is N×L×C×k×k;

[0024] Finally, the feature Q is compared with the feature block K patches The initial attention score is obtained by convolution matching, and the attention score is normalized by the Softmax function to further obtain the attention weight W attn ;

[0025] The feature block V patches Expanding in spatial dimensions and attention weight W attn Perform deconvolution operation to obtain high-resolution fusion feature F attn_out Finally, the convolution operation is used to further enhance the expression ability of the fusion features and generate the local scale feature image F out The expression is:

[0026]

[0027] in, Represents the convolution operation; * T Represents the deconvolution operation; scale represents the scale of the Softmax function.

[0028] Furthermore, the calculation process of the multi-scale grouped cross-modal attention module includes:

[0029] The upsampled hyperspectral data are grouped in the channel dimension to capture the subtle differences and similarities between channels within each subgroup. The expression is:

[0030]

[0031] Among them, Interpolate represents the upsampling operation; X HSI is the hyperspectral data; G j is the hyperspectral grouping result after upsampling; M is the total number of groups;

[0032] Multi-scale convolution operations are used on multispectral data to extract spatial detail features at different receptive field sizes and perform feature splicing. The expression is:

[0033]

[0034] Among them, X MSI For hyperspectral data; is the convolution operation of different scales; s i Represents a collection of different scales; Concat(·) represents splicing in the channel dimension; Indicates features of the same scale; F multi-scale Represents the multi-scale features obtained by splicing features of different scales in the channel dimension; Indicates that the two characteristic shapes are B×D×H out ×W out 、B×ND×H out ×W out ;

[0035] For each channel group, the cross-modal attention is calculated and fused with the obtained multi-scale features. The expression is:

[0036]

[0037] in, The result of grouping is used as the feature Q in the attention calculation for convolution; K and V are the obtained multi-scale features; D is the embedding dimension; ∑ represents the sum of the elements on the embedding dimension D; σ(·) represents the Sigmoid activation function; ⊙ is the pixel-by-pixel multiplication; Q j A represents the result of convolution operation on the grouped hyperspectral results; j Indicates the attention calculation results of different grouping features; Represents the result obtained by multiplying and convolving the attention weight calculation with the V feature; To indicate that the characteristic shape is N×C j ×H out ×W out ;

[0038] Finally, after all channel groups are fused through the cross-modal attention model, the feature maps are concatenated in the channel dimension and a convolution operation is performed to remap the number of channels to that of the hyperspectral data.

[0039] Furthermore, the calculation process of the cross-modal feature interaction module includes:

[0040] The multispectral data is downsampled to the size of the hyperspectral data to facilitate feature alignment. Then, the original multispectral data, the downsampled multispectral data, and the hyperspectral data are transformed through the basic convolution block to obtain a feature map with rich detail information.

[0041] Finally, the normalized multispectral feature Q uery and hyperspectral features K ey , the cross-scale attention is calculated by tensor inner product to achieve effective fusion between features, the expression is:

[0042]

[0043] Among them, Q norm is the downsampled multispectral image feature after convolution normalization; K is the hyperspectral feature; scale is the scale factor of the Softmax normalization operation, which is used to control the sharpness of the attention distribution; ⊙ is the pixel-by-pixel multiplication; V is the multispectral feature; Conv(·) represents the two-dimensional convolution operation; Z is the normalization factor; σ(·) is the LeakyReLU activation function; A and F represent the normalized attention weight and the result of weighted fusion using the attention weight, respectively.

[0044] Furthermore, the feature fusion module gradually refines and fuses the expressions of different features through convolutional layers and nonlinear activation functions, and uses residual connections to prevent network overfitting;

[0045] Then, the feature cascade mechanism is combined to fully capture and amplify the complementary information of feature dimensions in hyperspectral data and multispectral data, thereby effectively retaining and enhancing the key low-spectral variation characteristics in the process of forest canopy feature extraction. The expression is:

[0046]

[0047] Where x is the four input features; Conv(·) represents the two-dimensional convolution operation; σ(·) is the PReLU loss function; (·)n Indicates the number of times the operation is performed. When the operation is performed for the last time, the number of output channels is adjusted to be consistent with the number of channels of the original hyperspectral. x1, x2, represent the result of one convolution operation on the four input features and the result of two convolution operations on the result of the previous step. cat 、x out They represent the result of concatenating the results of the two-step operation and the result of performing a convolution operation on the concatenated result.

[0048] Finally, the output hyperspectral features are added to the hyperspectral data upsampled by bilinear interpolation to obtain high-resolution hyperspectral data.

[0049] By means of the above technical solution, the present invention provides a deep-shallow dual-branch super-resolution method for forest hyperspectral satellite imagery, which has at least the following beneficial effects:

[0050] 1. The present invention effectively solves the modal difference problem between low-resolution hyperspectral and high-resolution multispectral data by learning at different feature levels and scales. In addition, through multiple feature attention mechanisms, the model focuses more on the low-spectral variation characteristics of different forest vegetation in satellite images. It is significantly superior to advanced models in overall consistency and spectral fidelity, and can meet the requirements of various subsequent forest resource fine-grained monitoring tasks.

[0051] 2. The present invention effectively considers the huge modal differences between satellite-borne hyperspectral data and multispectral data, and uses the complementary characteristics of feature differences between different data to focus on the differences in spatial and spectral feature information of forest canopy at different feature levels. It proposes a dual-branch network architecture designed specifically for the task of super-resolution fusion of forest satellite hyperspectral images and multispectral images. This architecture effectively overcomes the problems of insufficient information interaction and difficulty in effective feature decoupling when fusing the two types of data in traditional methods, and realizes flexible and efficient capture and complementary coordination of forest canopy features in different modalities.

[0052] 3. This paper innovatively designs a feature extractor and multiple spatial-spectral attention modules. By globally modeling spatial-spectral features and combining different types of attention modules, the model enhances the model's sensitivity to and accuracy in extracting spatial details and spectral features of forest vegetation canopies. The feature extractor effectively mines high-level abstract semantic information from low-resolution hyperspectral data and high-resolution multispectral data. The multiple spatial-spectral attention modules enhance the ability to express features at different feature levels, effectively promoting the deep fusion of forest spatial-spectral features across different data sets.

[0053] 4. The dual-branch network architecture and multiple attention modules proposed in the present invention focus on computational efficiency in their design. Through efficient feature extraction and fusion strategies, they reduce the consumption of computing resources and memory, improve the efficiency of processing large-scale, high-data-volume forest vegetation satellite-borne hyperspectral data and multispectral data fusion super-resolution, and adapt to the high-efficiency requirements in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0055] Figure 1 Flowchart of the deep-shallow dual-branch super-resolution method of the present invention;

[0056] Figure 2 This is the network structure diagram of the overall architecture of DSDMNet in the present invention;

[0057] Figure 3 This is the network structure diagram of the lightweight residual Linformer feature extractor in the present invention;

[0058] Figure 4 This is the network structure diagram of the multi-scale grouped cross-modal attention module in the present invention. DETAILED DESCRIPTION

[0059] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0060] This embodiment proposes a deep and shallow dual-branch super-resolution method for forest hyperspectral satellite images. By learning at different feature levels and scales, it effectively solves the modal difference problem between low-resolution hyperspectral and high-resolution multispectral data. In addition, through multiple feature attention mechanisms, the model pays more attention to the low-spectral variation characteristics of different forest vegetation in satellite images. It is significantly better than advanced models in overall consistency and spectral fidelity, and can meet the requirements of various subsequent forest resource fine monitoring tasks. Figure 1 As shown, the method includes the following steps:

[0061] S1. Acquire satellite-borne hyperspectral image data for forest vegetation monitoring, including low-resolution hyperspectral data (i.e., raw hyperspectral data) and high-resolution multispectral data.

[0062] S2. Construct a dual-branch network architecture designed for the super-resolution task of fusion of hyperspectral data and multispectral data, including a deep feature extraction branch for extracting deep encoding features from hyperspectral data and multispectral data, and interactively extracting spatial spectral information at the global scale and local scale respectively, effectively capturing the long-range dependencies of the spectral dimension and the local feature diversity of the spatial dimension.

[0063] In the hyperspectral and multispectral fusion super-resolution task, shallow features perform well in capturing rich spatial and spectral details, but the information extracted is often limited to the presentation of local features and low-level feature expressions, making it difficult to effectively express the more abstract and higher-level semantic differences between different forest stands in forest scenes. Specifically, different forest stands exhibit regular spatial structural patterns, crown outlines, and global spectral patterns at a macro level. These patterns require higher-order and more abstract feature expressions to be effectively captured. Therefore, this embodiment specifically designs a deep feature extraction branch. First, a feature encoder is used to extract deep encoding features from the original hyperspectral data and multispectral data, and then various cross-attention mechanisms are used to further enhance the capture and feature expression of deep semantic information. In order to effectively utilize the deeper features in hyperspectral and multispectral images, this embodiment proposes a hyperspectral / multispectral feature extractor containing deep spatial spectral information and a deep feature extraction branch based on cross-scale attention.

[0064] Specifically, the deep feature extraction branch mines high-level semantic information in the original data through a lightweight residual Linformer feature extractor, and then the global spatial-spectral feature attention module and the local cross-scale feature attention module interactively extract spatial-spectral information at the global scale and local scale, respectively. It can effectively capture the long-range dependency of the spectral dimension and the local feature diversity of the spatial dimension, thereby improving the modeling ability of spatial-spectral information.

[0065] The deep feature extraction branch includes a lightweight residual Linformer feature extractor (i.e., Figure 2 The lightweight residual Linformer feature extractor includes a Linformer encoder for extracting spatial spectral information and a residual convolution module that enhances the expression of spatial spectral information through convolution operations and residual connections, and then outputs hyperspectral and multispectral deep encoding features F Res .

[0066] like Figure 3As shown in the figure, the lightweight residual Linformer feature extractor combines the Linformer encoder with the residual convolution module for higher-level spatial spectral information extraction. The lightweight residual Linformer feature extractor first reduces the feature dimensionality of the original high-dimensional spectral data through feature embedding to generate a more compact and easier to process feature representation, and then passes it through an improved version of the Transformer, namely the Linformer structure. This structure can reduce the complexity of attention calculation, greatly improve computational efficiency, and is suitable for processing longer image sequences. Specifically, it uses a smaller projection dimension k (less than the sequence length L) to compress the key-value information during attention calculation. This optimization effectively reduces the computational complexity of the Transformer, allowing the model to more efficiently capture the global dependency between space and spectrum in image data, thereby obtaining a more efficient and deep spectral-spatial joint feature representation. Its core calculation process is as follows:

[0067]

[0068] Among them, Q, K, and V are Query, Key, and Value matrices respectively; W Q 、W K 、W V is a learnable weight parameter; Attention is attention; head i =Attention(Q i , K i , V i ), represents the attention calculated by different attention heads, i is the total number of attention heads; W O is the output mapping function; X attn represents the output result obtained by linear mapping after multi-head attention splicing; X f represents the result of flattening the spatial dimension into the sequence dimension; d h Indicates the dimension size of the features in each attention head.

[0069] After feature extraction, in order to further obtain deep features, this embodiment introduces a residual convolution module. This module enhances the expression ability of spatial spectral information through convolution operations and residual connections. Specifically, the features output by the Linformer encoder are first convolved once and twice in the channel dimension respectively, and the main features obtained are added and restored to the input dimension by convolution again. Subsequently, the features are input into multiple residual convolution modules to enhance the ability to retain and transmit information. Specifically, the feature map is activated by the LeakyReLU activation function after being processed by the convolution layer, and then combined with the residual learning mechanism, the final output is added to the input to obtain the deep information of the image features. The calculation formula is as follows:

[0070] F Res =Conv(δ(Conv(X in )))+X in

[0071] Among them, X in is the input feature; Conv(·) represents a two-dimensional convolution operation; δ(·) is the ReLU activation function. This module exploits the deep information hidden in the feature map and combines it with the lightweight residual Linformer feature extractor to enhance the ability to capture both spatial and spectral information.

[0072] In order to effectively capture the spatial spectral differences of the forest canopy, this embodiment designs a global spatial spectral feature attention (Global Spatial Spectral Feature Attention, GSSFA) to enhance the overall expression ability of feature information. The global spatial spectral feature attention module is mainly composed of a spatial attention and a spectral attention. Specifically, the global spatial spectral feature attention module extracts spatial spectral features from the hyperspectral and multispectral deep encoding features output by the lightweight residual Linformer feature extractor and the hyperspectral data through the basic convolution block, and then performs spatial attention and spectral attention calculations on these features in turn, reducing the weights of irrelevant areas and emphasizing the importance of certain band channels on a global scale, and finally realizing deep mining and effective fusion of features through attention weight calculation. As Figure 2 There is a straight line connecting the two streams between the feature extractor and the two attention modules, which means that the hyperspectral and multispectral features passing through the extractor enter the two attention modules together. The main calculation process is as follows:

[0073]

[0074] Among them, X represents the three feature maps output by the basic convolution block; Conv(·) represents the two-dimensional convolution operation; σ(·) represents the Sigmoid activation function; W1 and W2 are the linear transformation parameter matrices respectively; b1 and b2 are bias terms; δ(·) is the ReLU activation function; Q attn , K attn 、V attn Represents the output of the three feature maps after spatial attention and spectral attention, that is, X ss ; Represents the convolution operation; * T represents the deconvolution operation; X attn Represents the fused features obtained after deconvolution operation.

[0075] In order to effectively capture the local fine structure and spectral differences of the forest canopy, this embodiment uses a local cross-scale feature attention (LCFA) to focus on the spatial spectral characteristics of hyperspectral data at the local level to improve the local detail recovery capability of hyperspectral and multispectral image super-resolution reconstruction. Specifically, the local cross-scale feature attention module extracts spatial spectral features from the hyperspectral and multispectral deep encoding features output by the lightweight residual Linformer feature extractor and the hyperspectral data through the basic convolution block, and then performs cross-attention calculation on the local scale and expands the attention weight W in the spatial dimension. attn Perform deconvolution operation to obtain high-resolution fusion feature F attn_out The main calculation process is as follows:

[0076] First, the input image feature F DMS 、F DHS 、F HS ∈R N×C×H×W Perform convolution feature transformation and use it as features Q, K, and V respectively. The expressions are:

[0077]

[0078] Where, Conv q (·), Conv k (·), Conv v (·) indicates that convolution operations are performed on the three input features respectively; R N×C×H×W Indicates that the characteristic shape is N×C×H×W; Indicates that the characteristic shapes are N×C mid ×H×W and N×C hs ×H×W.

[0079] Secondly, K and V are divided into several small blocks through the feature extraction block function ExtractImagePatches, and the feature blocks K and V are obtained respectively. patches ∈R N×L×C×k×k and V patches ∈R N×L×C×k×k , the expression is;

[0080] K / V patches =ExtractImagePatches(K / V,ks,stride)

[0081] Among them, L is the total number of blocks; k is the size of each small block; ks is the convolution kernel size; stride is the sliding step size; R N×L×C×k×k This indicates that the characteristic shape is N×L×C×k×k.

[0082] Again, the feature Q is compared with the feature block K patches The initial attention score is obtained by convolution matching, and the attention score is normalized by the Softmax function to further obtain the attention weight W attn . Attention weight W attn It reflects the degree of association between each local feature and the global feature. The higher the weight value, the more important this local feature block is to the reconstruction of the target position.

[0083] Finally, the feature block V patches Expanding in spatial dimensions and attention weight W attn Perform deconvolution operation to obtain high-resolution fusion feature F attn_out Finally, the convolution operation is used to further enhance the expression ability of the fusion features and generate the local scale feature image F out , enhances the importance of forest canopy features in key areas, effectively guiding the model to focus on the spatial spectral details of forest canopies in different local areas during subsequent feature fusion. The calculation formula is as follows:

[0084]

[0085] in, Represents the convolution operation; * T Represents the deconvolution operation; scale represents the scale of the Softmax function.

[0086] In the hyperspectral and multispectral fusion super-resolution task, traditional methods usually perform deep feature encoding first to fully mine high-level abstract semantic information. However, relying solely on high-level features may lead to partial loss of fine spatial details and shallow spectral features of the image, which will seriously affect the accurate expression of forest canopy features. For example, the forest resource monitoring task essentially relies on subtle spectral feature differences (such as chlorophyll, water content, and biomass differences between different tree species) and spatial features (such as texture structure, etc.) of the forest canopy. In the super-resolution process, if we only focus on deep abstract features and ignore the direct shallow fine feature extraction of the original data, it may cause the subsequent classification model to be difficult to capture the tiny but critical difference information between different forest canopies. Therefore, this embodiment designs a shallow feature extraction branch to effectively mine the rich information of the original hyperspectral and multispectral data. This branch mainly includes two sub-branches. On the one hand, a multi-scale grouping cross-modal attention module is used to group the channel dimensions of hyperspectral data to obtain spectral features of local spectral feature details, and multi-scale convolution is used for multispectral data to obtain multiple spatial features. The cross-modal fusion between the two is achieved through feature matching. On the other hand, a cross-modal feature interaction module is combined to interactively fuse spatial features and spectral features to achieve a joint representation of spatial features and spectral features, thereby enhancing the model's ability to fuse spatial and spectral information. The details are as follows:

[0087] In the super-resolution task of fusing hyperspectral and multispectral images, traditional methods are usually unable to effectively fuse information of different resolutions and different bands, making it difficult to fully capture the spatial-spectral characteristics of the images. In particular, in the task of forest resource monitoring, accurate canopy feature extraction requires not only spatial detail information with high spatial resolution, but also rich spectral information for species differentiation. To address this problem, we designed a multiscale grouping cross-modal attention module (MGCA) to fully utilize the rich spectral information of hyperspectral images and the higher spatial detail characteristics of multispectral images. Through the cross-modal attention mechanism, we strengthen the spectral-spatial information fusion and achieve improved super-resolution reconstruction effects.

[0088] Specifically, if Figure 4 As shown. This embodiment groups the upsampled hyperspectral data in the channel dimension to capture the subtle differences and similarities between channels within each subgroup, enhance the ability to characterize local channel correlations, and reduce the model's computational complexity and memory consumption. The process can be expressed as follows:

[0089]

[0090] Among them, Interpolate represents the upsampling operation; XHSI is the hyperspectral data; G j is the hyperspectral grouping result after upsampling; M is the total number of groups.

[0091] Then, in order to effectively capture the multi-scale spatial information in the multispectral image, this embodiment uses a multi-scale convolution operation on the multispectral data to extract spatial detail features at different receptive field sizes and perform feature splicing. This is conducive to the expression of refined spatial information. The expression is:

[0092]

[0093] Among them, X MSI For hyperspectral data; is the convolution operation of different scales; s i Represents a collection of different scales; Concat(·) represents splicing in the channel dimension; Indicates features of the same scale; F multi-scale Represents the multi-scale features obtained by splicing features of different scales in the channel dimension; Indicates that the two characteristic shapes are B×D×H out ×W out 、B×ND×H out ×W out .

[0094] Then, for each channel group, cross-modal attention is calculated and fused with the obtained multi-scale features. The expression is:

[0095]

[0096] in, The result of grouping is used as the feature Q in the attention calculation for convolution; K and V are the obtained multi-scale features; D is the embedding dimension; ∑ represents the sum of the elements on the embedding dimension D; σ(·) represents the Sigmoid activation function; ⊙ is the pixel-by-pixel multiplication; Q j A represents the result of convolution operation on the grouped hyperspectral results; j Indicates the attention calculation results of different grouping features; Represents the result obtained by multiplying and convolving the attention weight calculation with the V feature; To indicate that the characteristic shape is N×C j ×H out ×W out .

[0097] Finally, after all channel groups are fused through the cross-modal attention model, the feature maps are concatenated in the channel dimension and a convolution operation is performed to remap the number of channels to that of the hyperspectral data.

[0098] In order to solve the problems of inconsistent spatial scales and large differences in spectral resolution between hyperspectral images and multispectral images, this embodiment uses a cross-modal feature interaction module (CFIM), which realizes efficient interaction and fusion between the features of the two modal data through the cross-scale attention mechanism, effectively improving the spectral reconstruction performance of the model. Specifically, this embodiment downsamples the multispectral data to the size of the hyperspectral data to facilitate feature alignment, and then performs feature transformation on the original multispectral data, the downsampled multispectral data, and the hyperspectral data through the basic convolution block to obtain a feature map with rich detail information. At the same time, in order to avoid the instability of the attention value caused by scale differences, feature normalization is performed before attention calculation, and finally the normalized multispectral feature Q is used. uery and hyperspectral features K ey , by calculating the cross-scale attention through the tensor inner product, effective fusion between features is achieved. Its main process can be expressed by the following formula:

[0099]

[0100] Among them, Q norm is the downsampled multispectral image feature after convolution normalization; K is the hyperspectral feature; scale is the scale factor of the Softmax normalization operation, which is used to control the sharpness of the attention distribution; ⊙ is the pixel-by-pixel multiplication; V is the multispectral feature; Conv(·) represents the two-dimensional convolution operation; Z is the normalization factor; σ(·) is the LeakyReLU activation function; A and F represent the normalized attention weight and the result of weighted fusion using the attention weight, respectively.

[0101] S3. A feature fusion module is used to reconstruct the deep and shallow features output by the dual-branch network architecture, generating high-resolution hyperspectral data with low spectral variation characteristics that characterize forest vegetation in spaceborne hyperspectral imagery. Specifically, low spectral variation refers to the spectral variations of the same features before and after super-resolution. The lower the spectral variation, the better the effect. Different forest vegetation can only be expressed by subtle spectral differences. The goal of forest super-resolution tasks is to preserve these subtle differences as much as possible and reduce the overall low spectral variation before and after super-resolution.

[0102] In forest scenes, since the spectral response differences between different forest canopies are weak and detailed, it is crucial to maintain and fully exploit the original spectral variability of hyperspectral data. The dual-branch network architecture proposed in this embodiment retains rich spatial texture and detail information in shallow features, and contains more abstract, high-level semantics and spectral difference characteristics in deep features. In order to ensure the effective fusion of these feature information, this embodiment uses a special feature fusion module (FFA) to effectively integrate multi-scale feature information, including low-level spatial details of the original data and deep abstract high-level spectral difference features. This shallow and deep feature fusion method helps the network integrate spectral and spatial information of different scales to achieve higher quality super-resolution reconstruction effects, thereby better serving subsequent forest resource monitoring tasks.

[0103] Specifically, the feature fusion module gradually refines and fuses the expressions of different features through convolutional layers and nonlinear activation functions (PReLU), and uses residual connections to prevent network overfitting, while alleviating the gradient vanishing problem of deep networks, effectively enhancing the ability of the two features to retain and represent spectral detail information. It then combines the feature cascade mechanism to fully capture and amplify the complementary information of the feature dimensions in hyperspectral data and multispectral data, thereby effectively retaining and enhancing the key low-spectral variation characteristics in the process of forest canopy feature extraction. The process can be represented as follows:

[0104]

[0105] Where x is the four input features; Conv(·) represents the two-dimensional convolution operation; σ(·) is the PReLU loss function; (·) n Indicates the number of times the operation is performed. When the operation is performed for the last time, the number of output channels is adjusted to be consistent with the number of channels of the original hyperspectral. x1, x2, represent the result of one convolution operation on the four input features and the result of two convolution operations on the result of the previous step. cat 、x out They respectively represent the result of splicing the results of the two-step operation and the result of performing a convolution operation on the spliced ​​result.

[0106] Finally, the output hyperspectral features are added to the hyperspectral data upsampled using bilinear interpolation to obtain high-resolution hyperspectral data. By introducing a global residual structure, the network training difficulty is reduced, effectively alleviating the vanishing gradient problem. Furthermore, this approach better preserves the rich spectral information in the original hyperspectral imagery and reduces the potential loss of spectral information during feature fusion, which is particularly important for maintaining forest spectral fidelity.

[0107] In summary, this residual connection mechanism enables the network to focus more on subtle but critical spectral difference features, thereby improving the spectral fidelity of hyperspectral super-resolution images.

[0108] This embodiment applies the above-mentioned deep-shallow dual-branch super-resolution method to multiple hyperspectral and multispectral image fusion super-resolution tasks. Figure 2 As shown, the DSDMNet (Deep-Shallow Dual-Branch Multi-Attention Network) network framework proposed in this embodiment consists of a parallel dual-branch network architecture and multiple feature attention modules. The overall design aims to effectively integrate the spatial spectral information of hyperspectral and multispectral images to improve the applicability of the model for forest resource monitoring tasks. In order to confirm that this application has shown significant advantages in limiting spectral variability, and that this advantage remains highly consistent across various indicators, this embodiment conducts an accuracy evaluation of the DSDMNet network framework, the specific contents of which are as follows:

[0109] 1. Study area and data

[0110] The study area is located in Dahuaping Town, Huoshan County, Lu'an City, Anhui Province, in the hinterland of Dabie Mountains, with a vegetation coverage rate of 84%.

[0111] The ZY-102 / E satellite, launched on December 26, 2021, carries a visible and near-infrared camera (VNIC) and a hyperspectral imager (AHSI). With a single-satellite orbital revisit period of 55 days, it provides hyperspectral imagery in 166 bands at a 30m spatial resolution. This experiment used high-quality imagery from May 4, 2022, which was radiometrically calibrated, atmospherically corrected, orthorectified, and subjected to bad band removal, resulting in a total of 151 bands.

[0112] Sentinel-2: Sentinel-2 is a high-resolution multispectral imaging satellite equipped with a multispectral imager (MSI). It consists of two polar-orbiting satellites, Sentinel 2A and Sentinel 2B, which operate in the same orbit with a phase difference of 180°. It can revisit the Earth's equatorial region every five days. In order to keep it as consistent as possible with the hyperspectral imaging date, this study used Sentinel-2 images from May 4, 2022. The data product level is L2A, which has been radiometrically and orthorectified. Due to the different resolutions of different bands in Sentinel-2, the bicubic convolution resampling method was used to resample all bands to 10 meters. Finally, a total of 12 bands, B1 to B12, were used for the experiment.

[0113] 2. Implementation Details

[0114] This experiment was implemented in the deep learning framework PyTorch 2.1.0 and Python 3.11. Model construction and optimization were performed using an NVIDIA GeForce RTX 4090 graphics card (24GB of video memory) and a CUDA 11.8 accelerator unit to fully utilize hardware resources and improve training speed and efficiency. During model training, the L1 root mean square error loss was used as the loss function, a multi-step learning rate scheduler was used, and the ADAM optimizer was employed to improve model convergence speed and stability. The initial learning rate was set to 4e-4, the total number of training batches was set to 500, and the learning rate was decayed to half every 100 epochs. The batch size was set to 10 to achieve high training efficiency and memory management. Experiments were conducted using hyperspectral images with an image size of 850×1600 and multispectral images with an image size of 2550×4800. Due to the lack of real reference images, the hyperspectral and multispectral images were uniformly downsampled three times according to the Wald protocol. The original hyperspectral images were used as reference images, and finally 48×48 hyperspectral images and 144×144 multispectral images were cropped for training. The ratio of training images to validation images was 8:2. At the same time, the pseudo-random seed was set to 1000 in all experiments to reduce accidental errors caused by the sequential input of data.

[0115] In addition, to comprehensively measure the super-resolution effect of the constructed DSDMNet, multiple evaluation metrics were used, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Erreur Relative GlobaleAdimensionnellede Synthèse (ERGAS), Spatial Consistency Coefficient (SCC), Spectral Angle Mapper (SAM), Spectral Information Divergence (SID), and Universal Image Quality Index (UIQI). PSNR, SSIM, and ERGAS can reflect the overall performance of super-resolution image quality, among which PSNR is the most commonly used metric. SCC can reflect the spatial similarity between the super-resolution image and the real image, while SAM, SID, and UIQI are used to measure the quality of the super-resolution image at the spectral level compared to the real image. Among these metrics, higher PSNR values ​​indicate better image quality, lower ERGAS, SAM, and SID values ​​indicate better image quality, and SSIM, SCC, and UIQI values ​​closer to 1 indicate better image quality. In summary, the combined use of these metrics helps verify the effectiveness of the model from multiple perspectives, but it should be noted that these metrics focus more on those that measure spectral quality.

[0116] 3. Comparison model introduction

[0117] To highlight the advantages of the DSDMNet (dual-branch network architecture) proposed in this example, a comparative evaluation of state-of-the-art methods covering CNN and Transformer architectures is conducted. The specific description is as follows:

[0118] ConSSFCNN: A CNN-based spatial and spectral fusion architecture that jointly utilizes the narrowband spectral properties in LR-HS images and the rich spatial background in HR RGB images, adds some shorter connections between the input and feature layers, and proposes the PDCon-SSF structure, which shows higher performance for HSISR.

[0119] MSDCNN: A multi-scale and multi-depth CNN network. The framework consists of a PNN and a deeper multi-scale neural network. The former network performs simple feature extraction, while the latter network contains a multi-scale feature extraction layer and builds a deep architecture, showing good performance in remote sensing full-color sharpening.

[0120] SSRNET: A new interpretable network based on CNN, which consists of three parts: Cross-Modal Message Insertion (CMMI), Spatial Reconstruction Network (SpatRN), and Spectral Reconstruction Network (SpecRN). CMMI is used to retain the most valuable information in LR HSI and HRMSI, while SpatRN and SpecRN are used to focus on reconstructing the lost spatial and spectral information under the guidance of spatial edge loss, effectively realizing the fusion of hyperspectral and multispectral.

[0121] MCT-NET: A multi-level cross-Transformer network. This network uses the Transformer model as the backbone network and designs a multi-level cross-modal interaction module to extract deep multi-scale features of HSI and MSI. It also performs cross-modal information interaction at the same scale through multi-level cross-Transformers. At the same time, a feature aggregation and reconstruction module is designed to further restore edge features. Its effect is significantly better than the traditional CNN model.

[0122] DCTransformer: A dual-cross Transformer network that uses directed pairwise multi-head cross attention to capture the interactions between different modalities, thereby promoting information transfer between modalities. It also combines SwinTransformer block-based cross attention to enhance contextual self-attention, achieving high-quality fusion of hyperspectral and multispectral imagery.

[0123] 4. Experimental Results

[0124] This example calculates and compares the average values ​​of all indicators of the validation images. In multiple hyperspectral and multispectral image fusion super-resolution tasks, DSDMNet demonstrates significant advantages in various image quality metrics. Compared with other state-of-the-art models, it demonstrates a strong ability to limit spectral variation and stability, as shown in Table 1.

[0125] Table 1 Comparison of super-resolution image quality indicators of different models

[0126] PSNR SSIM ERGAS SCC SAM SID UIQI ConSSFCNN 41.934286 0.912052 4.219024 0.952671 0.112487 5.817634 0.972852 SSRNET 47.130344 0.978782 2.298956 0.958248 0.061164 0.913872 0.991904 MSDCNN 49.797370 0.991775 1.691575 0.965438 0.033330 0.053891 0.995656 MCT-NET 51.748184 0.994640 1.339525 0.978379 0.028601 0.059198 0.997346 DCTransformer 52.402447 0.995339 1.263465 0.980911 0.027153 0.040994 0.997548 DSDMNet 53.177507 0.996006 1.197159 0.982835 0.024471 0.037029 0.997632

[0127] Specifically, in the core fidelity metric PSNR, DSDMNet surpassed all compared models with a score of 53.177507, an improvement of approximately 0.78db over the next-best DCTransformer model (52.402447), indicating that its reconstructed image has minimal pixel-level differences from the original high-resolution image. Furthermore, SSIM (0.996006), SCC (0.982835), and ERGAS (1.197159) also achieved optimal performance, improving by 0.000667, 0.001924, and 0.066306, respectively, over the DCTransformer model. This demonstrates the model's superior ability to preserve the overall image structure, preserve the fidelity of spatial details, and coordinate spectral information.

[0128] However, more importantly, among the indicators for measuring the spectral variability of images, SAM (0.024471), SID (0.037029) and UIQI (0.997632) are all significantly better than the DCTransformer model, with improvements of approximately 0.002682, 0.003965 and 0.000084 respectively, which shows that the present application has more advantages in spectral variability. At the same time, although the MCT-NET model is better than MSDCNN in the PSNR indicator of overall image quality, it is relatively inferior in the spectral variability indicator SID. This shows that MCT-NET has certain deficiencies in the stability of spectral reconstruction, and it also indirectly explains that different models perform differently in suppressing spectral variability. In contrast, the present application has demonstrated significant advantages in limiting spectral variability, and this advantage remains highly consistent across various indicators.

[0129] In summary, the proposed method, through its innovative network architecture design, achieves significant performance breakthroughs in the objective quality evaluation system for super-resolution reconstruction, demonstrating leading performance across all metrics. In particular, it demonstrates clear technical advantages in maintaining feature balance and improving spectral fidelity. This method has great application potential and can effectively support the precise monitoring of forest resources using hyperspectral satellite remote sensing.

[0130] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A deep-shallow dual-branch super-resolution method for forest hyperspectral satellite images, characterized by: The method comprises the following steps: Acquire satellite-borne hyperspectral image data for forest vegetation monitoring, including low-resolution hyperspectral data and high-resolution multispectral data; Construct a dual-branch network architecture designed for the super-resolution task of fusion of hyperspectral and multispectral data, including a deep feature extraction branch for capturing long-range dependencies in the spectral dimension and local feature diversity in the spatial dimension from hyperspectral and multispectral data; Furthermore, on the one hand, a multi-scale grouping cross-modal attention module is used to obtain spectral features and multiple spatial features of local spectral feature details; on the other hand, a shallow feature extraction branch is combined with a cross-modal feature interaction module to interactively fuse spatial features and spectral features; The deep feature extraction branch includes a lightweight residual Linformer feature extractor for extracting spatial spectral information from hyperspectral data and multispectral data respectively; Furthermore, the hyperspectral and multispectral deep encoding features output by the lightweight residual Linformer feature extractor are combined with the hyperspectral data through basic convolution blocks to extract spatial spectral features. These features are then subjected to spatial attention and spectral attention calculations in sequence, which reduce the weights of irrelevant areas and emphasize the importance of specific band channels on a global scale. Finally, the global spatial spectral feature attention module is implemented to achieve deep feature mining and effective fusion through attention weight calculation. In addition, the hyperspectral and multispectral deep encoding features output by the lightweight residual Linformer feature extractor are combined with the hyperspectral data through the basic convolution block to extract the spatial spectral features, and then the cross attention calculation is performed on the local scale and expanded in the spatial dimension with the attention weight. Perform deconvolution operation to obtain high-resolution fusion features The local cross-scale feature attention module of the feature image; The feature fusion module is used to reconstruct the deep features and shallow features output by the dual-branch network architecture to obtain high-resolution hyperspectral data with low spectral variation characteristics that characterize forest vegetation in spaceborne hyperspectral image data.

2. The deep-shallow dual-branch super-resolution method according to claim 1, characterized in that: The lightweight residual Linformer feature extractor includes a Linformer encoder for extracting spatial spectral information and a residual convolution module for enhancing the expression capability of spatial spectral information through convolution operations and residual connections.

3. The deep-shallow dual-branch super-resolution method according to claim 1, characterized in that: The expression of the global spatial spectral feature attention module is: ； Among them, X represents the three feature maps output by the basic convolution block; Represents a two-dimensional convolution operation; Represents the Sigmoid activation function; are the linear transformation parameter matrices; is the bias term; is the ReLU activation function; Represents the output of the three feature maps after spatial attention and spectral attention, that is ; Represents the convolution operation; represents the deconvolution operation; Represents the fused features obtained after deconvolution operation.

4. The deep-shallow dual-branch super-resolution method according to claim 1, characterized in that: The calculation process of the local cross-scale feature attention module includes: First, the input image features Perform convolution feature transformation and use it as feature , the expression is: ； Where, Indicates that convolution operations are performed on the three input features respectively; Indicates that the characteristic shape is N×C×H×W ; The characteristic shapes are N ×C mid ×H×W and N×C hs ×H×W ; Secondly, the feature extraction block function ExtractImagePatches Divide into several small blocks and get feature blocks respectively and , the expression is; ； in, L is the total number of blocks; k The size of each small piece; ks is the convolution kernel size; stride is the sliding step length; Indicates that the characteristic shape is N×L×C×k×k ; Finally, the features Proceed with Feature Block The initial attention score is obtained by convolution matching, and the attention weight is further obtained by normalizing the attention score through the Softmax function. ; The feature block Spreading in spatial dimensions and attention weights Perform deconvolution operation to obtain high-resolution fusion features Finally, the convolution operation is used to further enhance the expression ability of the fusion features and generate feature images at a local scale. The expression is: ； in, Represents the convolution operation; Represents the deconvolution operation; scale represents the scale of the Softmax function.

5. The deep-shallow dual-branch super-resolution method according to claim 1, characterized in that: The calculation process of the multi-scale grouped cross-modal attention module includes: The upsampled hyperspectral data are grouped in the channel dimension to capture the subtle differences and similarities between channels within each subgroup. The expression is: ； in, Represents an upsampling operation; For hyperspectral data; is the hyperspectral grouping result after upsampling; M is the total number of groups; Multi-scale convolution operations are used on multispectral data to extract spatial detail features at different receptive field sizes and perform feature splicing. The expression is: ； in, For hyperspectral data; Convolution operations of different scales; Represents a collection of different scales; Represents splicing in the channel dimension; Indicates features of the same scale; Represents the multi-scale features obtained by splicing features of different scales in the channel dimension; Indicates that the two characteristic shapes are B×D×H out ×W out 、 B×ND×H out ×W out ; For each channel group, the cross-modal attention is calculated and fused with the obtained multi-scale features. The expression is: ； in, To use the grouping results as features in attention calculation Perform convolution; K and V are the obtained multi-scale features; D is the embedding dimension; ∑ represents the sum of the elements on the embedding dimension D; Represents the Sigmoid activation function; ⊙ is pixel-by-pixel multiplication; Represents the result of convolution operation on the grouped hyperspectral results; Indicates the attention calculation results of different grouping features; Represents the result obtained by multiplying and convolving the attention weight calculation with the V feature; To represent the characteristic shape N×C j ×H out ×W out ; Finally, after all channel groups are fused through the cross-modal attention model, the feature maps are concatenated in the channel dimension and a convolution operation is performed to remap the number of channels to that of the hyperspectral data.

6. The deep-shallow dual-branch super-resolution method according to claim 1, characterized in that: The calculation process of the cross-modal feature interaction module includes: The multispectral data is downsampled to the size of the hyperspectral data to facilitate feature alignment. Then, the original multispectral data, the downsampled multispectral data, and the hyperspectral data are transformed through the basic convolution block to obtain a feature map with rich detail information. Finally, the normalized multispectral features are used and hyperspectral features , the cross-scale attention is calculated by tensor inner product to achieve effective fusion between features, the expression is: ； in, is the downsampled multispectral image feature after convolution normalization; K is the hyperspectral feature; scale is the scale factor of the Softmax normalization operation, which is used to control the sharpness of the attention distribution; ⊙ is the pixel-by-pixel multiplication; V is the multispectral feature; Represents a two-dimensional convolution operation; Z is a normalization factor; is the LeakyReLU activation function; They represent the normalized attention weights and the results of weighted fusion using attention weights, respectively.

7. The deep-shallow dual-branch super-resolution method according to claim 1, characterized in that: The feature fusion module gradually refines and fuses the expressions of different features through convolutional layers and nonlinear activation functions, and uses residual connections to prevent network overfitting; Then, the feature cascade mechanism is combined to fully capture and amplify the complementary information of feature dimensions in hyperspectral data and multispectral data, thereby effectively retaining and enhancing the key low-spectral variation characteristics in the process of forest canopy feature extraction. The expression is: ； in, The four features of the input; Represents a two-dimensional convolution operation; is the PReLU loss function; Indicates the number of times the operation is performed. When the operation is performed for the last time, the number of output channels is adjusted to be consistent with the number of channels of the original hyperspectral. , represents the result of one convolution operation on the four input features and the result of two convolution operations on the result of the previous step; They represent the result of concatenating the results of the two-step operation and the result of performing a convolution operation on the concatenated result. Finally, the output hyperspectral features are added to the hyperspectral data upsampled by bilinear interpolation to obtain high-resolution hyperspectral data.

Images