Cross-modal guided hyperspectral image classification framework and method under multiple degradation conditions

Abstract

The invention discloses a cross-modal guided hyperspectral image classification framework and method under multiple degradation conditions, and the framework utilizes the complementary information of active remote sensing data to enhance the structure perception and space consistency of degraded hyperspectral images. The framework provides multi-level and multi-scale guidance based on active remote sensing data by using a cross-modal feature pyramid guidance module, so that stable and robust cross-modal feature fusion is realized. Meanwhile, hyperspectral image wave bands are grouped through a HyperGroup Mix module to construct spectrum-space features, style variation between samples is promoted through feature exchange, and the adaptability of the model to different degradation domains is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image technology, and in particular relates to a hyperspectral image classification framework and method under multi-degradation conditions guided by cross-modal approaches. Background Technology

[0002] Hyperspectral imagery (HSI), with its continuous high spectral resolution, can detect subtle changes between extremely narrow adjacent spectral bands, thus providing finer spectral granularity than traditional multispectral imaging. This advantage makes HSI outstanding in fine-grained target recognition and complex scene classification, and it is widely used in land cover mapping, precision agriculture, mineral exploration, environmental monitoring, urban planning, and many other fields. Despite the strong application potential of HSI, its data acquisition is often susceptible to quality degradation due to various factors caused by the high spectral dimension, complex sensor architecture, and high temporal resolution of hyperspectral systems. For example, sensor aging and calibration errors can lead to fringe artifacts and dead zones; image preprocessing and transmission processes may introduce Gaussian noise, spatial blurring, and resolution mismatch; while atmospheric changes, illumination fluctuations, and terrain occlusion often cause spectral distortion or signal attenuation. Notably, these degradation sources often occur simultaneously in a non-independent and compounded manner, resulting in severe damage to the spectral consistency and spatial continuity of hyperspectral data. This multi-source degradation seriously affects the reliability and scalability of hyperspectral data-based applications in real-world scenarios. Therefore, enhancing the robustness of classification models to various degradation types while maintaining high classification accuracy under heterogeneous noise conditions has become a key challenge and is crucial for advancing intelligent interpretation in the field of remote sensing. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention proposes a hyperspectral image classification framework and method under multi-degradation conditions guided by cross-modal approaches.

[0004] The technical solution of the present invention is as follows:

[0005] A hyperspectral image classification framework under multi-degradation conditions guided by cross-modal characteristics, the hyperspectral image classification framework comprising a dual-branch structure consisting of an HSI branch and an active remote sensing branch, and a classification head consisting of linear layers;

[0006] The HSI branch, in accordance with the processing order of hyperspectral images under multiple degradation conditions, includes a first convolutional layer for extracting basic features of hyperspectral images, a first HyperGroupMix module for performing the first cross-sample style transfer and outputting the first layer of HSI feature map, a cross-modal feature pyramid guidance module for cross-modal guidance and multi-scale feature fusion and outputting HSI fused feature map, a multi-head self-attention mechanism module, and a projection layer.

[0007] The active remote sensing branch, following the processing order of active remote sensing data from the same region as the hyperspectral image, includes: a third convolutional layer for initial feature extraction and providing the HSI branch with a first active remote sensing feature map whose scale and dimension match the first layer HSI feature map; a first downsampling layer for first-level scale compression; a fourth convolutional layer for second feature extraction and providing the HSI branch with a second active remote sensing feature map whose scale and dimension match the third layer HSI feature map; a second downsampling layer for second-level scale compression and outputting the final active remote sensing feature map; a multi-head self-attention mechanism module; and a projection layer.

[0008] The HSI fused feature map and the final active remote sensing feature map are sequentially passed through the multi-head self-attention mechanism module and the projection layer of the corresponding branch, respectively, and then cross-modal feature interaction is performed. After passing through the classification head, the hyperspectral image classification is completed.

[0009] Furthermore, the cross-modal feature pyramid guidance module includes a first cross-modal feature merging module that performs initial feature fusion of the first-layer HSI feature map and an active remote sensing feature map whose scale and dimension match the first-layer HSI feature map; a second HyperGroupMix module that performs a second cross-sample style transfer and outputs a second-layer HSI feature map; a second convolutional layer that performs feature compression; a third HyperGroupMix module that performs a third cross-sample style transfer and outputs a third-layer HSI feature map; and a second cross-modal feature merging module that performs a second feature fusion of the third-layer HSI feature map and an active remote sensing feature map whose scale and dimension match the third-layer HSI feature map and outputs a fused HSI feature map.

[0010] Furthermore, the specific method for feature fusion by the cross-modal feature merging module includes:

[0011] Receive scale- and dimension-matched HSI feature maps X∈H B×C×H×W and active remote sensing feature map Y∈L B×C×H×W Where X corresponds to the HIS branch, Y corresponds to the LiDAR branch, B and C represent the batch size and number of channels, respectively, and H and W represent the spatial height and width of the feature map;

[0012] Spatial compression of HSI and active remote sensing feature maps is performed using non-overlapping 2×2 windows. This enhances local structure representation during downsampling by reshaping local spatial pixels within each window to channel dimensions. Feature block representations are obtained from both the HSI and LiDAR modes for each channel. For channel i, the feature block representation obtained from the HSI mode is denoted as the HSI feature block. The feature block representation obtained from LiDAR modes is denoted as the active remote sensing feature block. i∈{1,2,...,C};

[0013] For any channel i, connect the HSI feature block and the active remote sensing feature block along the channel dimension to form a channel-level fused feature representation. The fused feature is obtained by aggregating all channel-level fused feature representations.

[0014] Applying 1×1 convolution to compress the channel dimensions yields the final ensemble multimodal representation, i.e., the fused feature map, which is represented as follows:

[0015] Furthermore, the specific method for cross-sample style transfer performed by the HyperGroupMix module includes:

[0016] The input feature map F is divided into several subsets along the channel dimension, denoted as F = {f i |i=1,2,...,n}, where f i Let i be the i-th subset, and n be the total number of subsets;

[0017] Calculate the spectral spatial feature statistics for each subset, wherein the spectral spatial feature statistics include the channel average μ(f i ), standard deviation σ(f) i ) and median m(f i )as follows:

[0018]

[0019] Where H and W represent the height and width of the feature map, respectively, and ε is a constant;

[0020] Calculate the Gram matrix g0(f) for each feature map i ):

[0021]

[0022] Where, flat(f i ) represents the i-th subset f i The operation of flattening the spatial dimension into a tensor of shape (B, C, H × W), where I represents the identity matrix; 10 -3 I represents a higher-order perturbation term used to improve numerical stability, 10 -6 I represents a low-order perturbation term used to enhance computational robustness;

[0023] Based on Gram matrix g0(f) i The Gram matrix is ​​aligned with the centered representation of the grouped features by applying a whitening transformation, as shown in the expression:

[0024] g0(fi )=VΛV T

[0025] W = VΛ -1 / 2 V T

[0026] Where V is the eigenvector matrix, Λ is the diagonal matrix of eigenvalues, and W is the whitening matrix;

[0027] Finally, obtain the whitened feature representation g(f) i ):

[0028] g(f i )=W(f i -μ(f i ))

[0029] Randomly arranged samples f in the same batch i The whitened feature representation g(f) i ) and median m(f i Shuffle the order along the batch dimension to obtain a new random permutation of samples f. i The whitened feature representation of ′ is g(f i ′) and median m(f i ′);

[0030] Sample the mixing coefficient λ from the symmetric Beta distribution Beta(α,α) such that λ approaches 0 or 1; generate the biased first mixture characteristic statistic α using the following formula. mix Second mixed characteristic statistic ρ mix ;

[0031] α mix =λg(f i )+(1-λ)g(f i ′)

[0032] ρ mix =λm(f i )+(1-λ)m(f i ′)

[0033] Subsequently, the features of cross-sample style transfer were reconstructed using mixture feature statistics.

[0034]

[0035] Where ⊙ represents element-wise multiplication;

[0036] In the output stage, features are concatenated along the channel dimension. To restore the original input shape, the scale of the HSI feature map after cross-sample style transfer is obtained.

[0037] Furthermore, the degradation conditions of the hyperspectral image include at least one of additive Gaussian noise, cutoff line noise, convolutional blur noise, stripe noise, Poisson noise, salt-and-pepper noise, and zero-mean Gaussian noise.

[0038] A hyperspectral image classification method under multi-degradation conditions guided by cross-modal characteristics includes:

[0039] Construct and train the hyperspectral image classification framework under multi-degradation conditions guided by any of the above-mentioned methods;

[0040] The degraded hyperspectral image of the target and active remote sensing data of the same region as the hyperspectral image are input into the hyperspectral image classification framework to obtain the output result of the hyperspectral image classification framework.

[0041] Furthermore, the training method for the hyperspectral image classification framework includes:

[0042] Data preparation involves performing principal component analysis and normalization on the undegraded hyperspectral image, and extracting patches of the hyperspectral image formed by adjacent cuboids centered at each pixel, denoted as X0∈H. B×C×H×W In the formula, B and C represent the batch size and channel dimension of the patch, respectively, and H and W represent the height and width of the patch, respectively.

[0043] Active remote sensing data acquired in the same region as the hyperspectral image are normalized, and patches of the active remote sensing data are extracted in the same way, denoted as Y0∈H. B×C×H×W ;

[0044] The hyperspectral image classification framework is trained in multiple rounds using patches of hyperspectral images and patches of active remote sensing data as input data in batches. Based on the contrastive learning mechanism, the feature similarity between hyperspectral images and active remote sensing data is learned. The framework parameters are optimized until convergence using a hybrid loss function, which includes contrastive loss and cross-entropy loss.

[0045] Furthermore, the expression for the hybrid loss function is:

[0046]

[0047] In the formula, For cross-entropy loss, The weights are those corresponding to the cross-entropy loss. For comparison loss, κ is the weight corresponding to the comparison loss;

[0048] Furthermore, the expression for the contrast loss is:

[0049]

[0050] In the formula, To compare the normalized temperature-scale cross-entropy loss in the loss function, z i and z j These represent the final feature representations of the HIS branch and the active remote sensing branch, respectively; sim(u,v)=u T v represents the cosine similarity, and τ represents the temperature parameter, τ = 0.07.

[0051] Furthermore, the expression for the cross-entropy loss is:

[0052]

[0053] In the formula, C is the number of categories, p c The classification result output by the classification head in the hyperspectral image classification framework corresponds to the true label y. c The predicted probability.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] This invention proposes a cross-modal guided hyperspectral image classification framework and method under multiple degradation conditions. This framework leverages complementary information from active remote sensing data to enhance the structural awareness and spatial consistency of degraded hyperspectral images. The framework utilizes a cross-modal feature pyramid guidance module (CMFPG) to provide multi-level, multi-scale guidance based on active remote sensing data, thereby achieving stable and robust cross-modal feature fusion. Simultaneously, the HyperGroupMix module groups hyperspectral image bands to construct spectral-spatial features, and feature exchange promotes style variation among samples, significantly improving the model's adaptability to different degradation domains.

[0056] This invention employs a Cross-Modal Feature Pyramid Guided Module (CMFPG) to leverage the complementary strengths of cross-modal data (hyperspectral images and active remote sensing data) in a hierarchical and multi-scale manner. This module adaptively injects cross-modal cues into each stage of hyperspectral representation learning, thereby enhancing stable feature encoding and improving the performance of degenerate multimodal fusion. The CMFPG demonstrates significant advantages in maintaining classification reliability under adverse conditions.

[0057] This invention employs a domain-adaptive HyperGroupMix module to dynamically enhance the model's resilience under heterogeneous degradation modes. This module is a degradation-aware feature mixing module that improves domain generalization by structurally modeling spectral-spatial correlations. It divides spectral bands into semantically consistent groups, extracts statistical descriptors at the group level, and performs feature signature exchange between samples. This promotes feature diversity and facilitates cross-instance spatial-spectral style transfer, thereby improving cross-domain adaptability. This mechanism significantly mitigates performance degradation under different degradation types and provides state-of-the-art robustness in multi-degradation settings.

[0058] This invention utilizes a cross-modal guided hyperspectral image classification framework to explicitly address the performance degradation issues caused by various real-world data degradation scenarios. Unlike existing methods that rely solely on hyperspectral input, this framework integrates complementary information from active remote sensing modalities. Through a novel cross-modal feature pyramid guided module (CMFPG) and a contrastive learning scheme, this framework achieves more robust feature extraction and cross-modal fusion even with HSI data degradation. Attached Figure Description

[0059] Figure 1 This is a structural diagram of the hyperspectral image classification framework under multi-degradation conditions guided by cross-modal guidance in the embodiment;

[0060] Figure 2 This is a schematic diagram of the pyramid-shaped feature extraction path of the cross-modal feature pyramid guidance module in the embodiment;

[0061] Figure 3 This is a flowchart of the hyperspectral image classification method under multi-degradation conditions guided by cross-modal guidance in the embodiments. Detailed Implementation

[0062] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0063] Example 1:

[0064] This invention provides a cross-modal guided hyperspectral image classification framework under multiple degradation conditions, such as... Figure 1 and Figure 2 As shown, the hyperspectral image classification framework includes a two-branch structure consisting of an HSI branch and an active remote sensing branch, and a classification head consisting of linear layers;

[0065] The HSI branch, following the processing order of hyperspectral images under multiple degradation conditions, includes the first convolutional layer for extracting basic features of the hyperspectral image, the first HyperGroupMix module (HGM) for performing the first cross-sample style transfer and outputting the first layer of HSI feature map, the cross-modal feature pyramid guidance module (CMFPG) for cross-modal guidance and multi-scale feature fusion and outputting HSI fused feature map, the multi-head self-attention mechanism module (MHSA), and the projection layer.

[0066] The active remote sensing branch processes active remote sensing data from the same region as the hyperspectral image in the following order: a third convolutional layer that performs initial feature extraction and provides the HSI branch with a first active remote sensing feature map whose scale and dimension match the first layer HSI feature map; a first downsampling layer for first-level scale compression; a fourth convolutional layer that performs second feature extraction and provides the HSI branch with a second active remote sensing feature map whose scale and dimension match the third layer HSI feature map; a second downsampling layer that performs second-level scale compression and outputs the final active remote sensing feature map; a multi-head self-attention mechanism module; and a projection layer.

[0067] The HSI fused feature map and the final active remote sensing feature map are sequentially passed through the multi-head self-attention mechanism module and the projection layer of the corresponding branch, respectively, and then cross-modal feature interaction is performed. After passing through the classification head, the hyperspectral image classification is completed.

[0068] Example 2:

[0069] This embodiment is further designed based on Embodiment 1, as follows: Figure 1 and 2 As shown, in this example, the cross-modal feature pyramid guidance module includes a first cross-modal feature merging module that performs the first feature fusion of the first-layer HSI feature map and an active remote sensing feature map whose scale and dimension match the first-layer HSI feature map; a second HyperGroupMix module that performs the second cross-sample style transfer and outputs the second-layer HSI feature map; a second convolutional layer that performs feature compression; a third HyperGroupMix module that performs the third cross-sample style transfer and outputs the third-layer HSI feature map; and a second cross-modal feature merging module that performs the second feature fusion of the third-layer HSI feature map and an active remote sensing feature map whose scale and dimension match the third-layer HSI feature map and outputs the HSI fused feature map.

[0070] By stacking modules in a hierarchical manner, the Cross-Modal Feature Pyramid Guided Module (CMFPG) constructs a pyramid-shaped feature extraction path, enabling multi-scale and multi-level feature fusion. This effectively reduces the model's sensitivity to noise caused by degradation and achieves stable modeling and multi-scale semantic guidance for degraded HSI data.

[0071] Example 3:

[0072] This embodiment, based on Embodiment 2, further designs the following: The specific method for feature fusion by the cross-modal feature merging module in this example includes:

[0073] Receive scale- and dimension-matched HSI feature maps X∈H B×C×H×W and active remote sensing feature map Y∈L B×C×H×W Where X corresponds to the HIS branch, Y corresponds to the LiDAR branch, B and C represent the batch size and number of channels, respectively, and H and W represent the spatial height and width of the feature map;

[0074] Spatial compression of HSI and active remote sensing feature maps is performed using non-overlapping 2×2 windows. This enhances local structure representation during downsampling by reshaping local spatial pixels within each window to channel dimensions. Feature block representations are obtained from both the HSI and LiDAR modes for each channel. For channel i, the feature block representation obtained from the HSI mode is denoted as the HSI feature block. The feature block representation obtained from LiDAR modes is denoted as the active remote sensing feature block. i∈{1,2,...,C};

[0075] For any channel i, connect the HSI feature block and the active remote sensing feature block along the channel dimension to form a channel-level fused feature representation. This achieves spatial alignment and cross-modal feature fusion, aggregating all channel-level fusion feature representations to obtain the fused feature, which is:

[0076] Applying 1×1 convolution to compress the channel dimensions yields the final ensemble multimodal representation, i.e., the fused feature map. The ensemble multimodal representation is shown below. It effectively integrates complementary information across modalities.

[0077] Example 4:

[0078] This embodiment, based on Embodiment 2, further designs the following: The specific method for cross-sample style transfer using the HyperGroupMix module in this example includes:

[0079] This module performs grouping and mixing and statistical exchange of spectral spatial features among multiple samples, effectively perturbing style patterns in specific domains and enabling them to adapt to different degradation distributions;

[0080] The input feature map F is divided into several subsets along the channel dimension, denoted as F = {f i |i=1,2,...,n}, where f iLet be the i-th subset, and n be the total number of subsets; this grouping reduces computational complexity while preserving local spectral spatial correlation.

[0081] Subsequently, in order to model the style information embedded in the feature map, it is necessary to calculate some important spectral spatial feature statistics for each subset to obtain a stable and discriminative statistical representation; the spectral spatial feature statistics for each subset are calculated, including the channel average μ(f i ), standard deviation σ(f) i ) and median m(f i )as follows:

[0082]

[0083] Where H and W represent the height and width of the feature map, respectively, and ε is a small constant used to ensure numerical stability;

[0084] Calculate the Gram matrix g0(f) for each feature map i ):

[0085]

[0086] Here, ε is used to avoid division by zero; flat(f i ) represents the i-th subset f i The operation of flattening the spatial dimension into a tensor of shape (B,C,H×W), where I represents the identity matrix; to prevent the Gram matrix from becoming ill-conditioned during propagation, two perturbation terms are introduced: 10 -3 I represents a higher-order perturbation term used to improve numerical stability, 10 -6 I represents a low-order perturbation term used to enhance computational robustness;

[0087] Based on Gram matrix g0(f) i The Gram matrix is ​​aligned with the centered representation of the grouped features by applying a whitening transformation, as shown in the expression:

[0088] g0(f i )=VΛV T

[0089] W = VΛ -1 / 2 V T

[0090] Where V is the eigenvector matrix, Λ is the diagonal matrix of eigenvalues, and W is the whitening matrix; this operation is used to remove the relevant feature channels and normalize the second-order statistics, thereby reducing the distribution differences between samples and enhancing the stability and generalization ability of style representation.

[0091] Finally, obtain the whitened feature representation g(f)i ):

[0092] g(f i )=W(f i -μ(f i ))

[0093] Randomly arranged samples f in the same batch i The whitened feature representation g(f) i ) and median m(f i Shuffle the order along the batch dimension to obtain a new random permutation of samples f. i The whitened feature representation of ′ is g(f i ′) and median m(f i ′);

[0094] The mixing coefficient λ is sampled from the symmetric Beta distribution Beta(α,α), where α = 0.1, such that the mixing coefficient λ approaches 0 or 1; the biased first mixture characteristic statistic α is generated by the following formula. mix Second mixed characteristic statistic ρ mix To enhance the perturbation capability of the HyperGroupMix module;

[0095] α mix =λg(f i )+(1-λ)g(f i ′)

[0096] ρ mix =λm(f i )+(1-λ)m(f i ′)

[0097] Subsequently, the features of cross-sample style transfer were reconstructed using mixture feature statistics. This helps to simulate variations in sample style and enhances the generalization ability of hyperspectral models under different degradation conditions.

[0098]

[0099] Where ⊙ represents element-wise multiplication;

[0100] In the output stage, features are concatenated along the channel dimension. To restore the original input shape, the scale of the HSI feature map after cross-sample style transfer is obtained.

[0101] Example 5:

[0102] This embodiment is further designed based on Embodiment 1 in that the degradation conditions of the hyperspectral image in this example include at least one of additive Gaussian noise, cutoff line noise, convolution blur noise, stripe noise, Poisson noise, salt and pepper noise, and zero-mean Gaussian noise.

[0103] Furthermore, additive Gaussian noise, a typical type of noise, follows a Gaussian (normal) probability density distribution and is independent of the original signal. In practice, each band of the original Gaussian spectral signal is independently normalized, and then zero-mean Gaussian noise with varying variances is added. Subsequently, the data is inversely normalized to recover its original scale. The overall noise intensity is controlled by adjusting the maximum standard deviation.

[0104] Cutoff line noise, a type of strip noise, is caused by sensor malfunction or data transmission loss, resulting in the loss of entire columns of data in certain spectral bands. To simulate this effect, we randomly select one or more non-overlapping columns in each spectral band and set their values ​​to zero. The noise intensity is controlled by varying the number of damaged strips within a predetermined range.

[0105] Convolutional blur noise, typically caused by low-pass filtering, defocusing, or data transmission losses in sensors, is simulated by applying a mean convolution kernel to smooth each spectral band. The degree of blurring is adjusted by changing the kernel size.

[0106] Stripe noise, specifically uniformly wide stripe noise, can be caused by column sensor malfunctions, transmission errors, or scanning artifacts. We simulate this noise by randomly selecting multiple non-overlapping column regions of equal width within each spectral band and assigning them the same value. Similar to cutoff time noise, the sparsity and intensity of the stripes can be controlled by adjusting the number of stripes per band.

[0107] Poisson noise, due to the quantum nature of photon detection, frequently appears in low-light or nighttime HSI imaging. To simulate this, Poisson distribution noise is applied to each spectral band according to a specified signal-to-noise ratio (SNR) to simulate fluctuations in photon counts under low-light conditions.

[0108] Salt and pepper noise, a type of impulse noise typically caused by bad pixels, transmission errors, or scanner malfunctions, results in random pixels taking extreme values ​​(0 or 255). This phenomenon is simulated by randomly selecting a certain percentage of pixels in each spectral band, setting half of them to the minimum value and the other half to the maximum value, thus mimicking the "salt and pepper" effect.

[0109] Zero-mean Gaussian noise, typically caused by thermal sensor noise or electromagnetic interference during signal transmission, is simulated similarly to additive Gaussian noise, except that the same zero-mean Gaussian noise matrix is ​​uniformly added across all spectral bands. The noise intensity is controlled by adjusting the standard deviation of the distribution.

[0110] Example 6:

[0111] This invention provides a hyperspectral image classification method under multi-degradation conditions guided by cross-modal techniques, such as... Figure 3 As shown, it includes:

[0112] Construct and train the cross-modal guided hyperspectral image classification framework under multi-degradation conditions described in any of the above embodiments;

[0113] The degraded hyperspectral image of the target and active remote sensing data of the same region as the hyperspectral image are input into the hyperspectral image classification framework, and the output results of the hyperspectral image classification framework are obtained.

[0114] Example 7:

[0115] This embodiment, based on Embodiment Six, further designs the training method for the hyperspectral image classification framework as follows:

[0116] Data preparation involves performing principal component analysis and normalization on the undegraded hyperspectral image, and extracting patches of the hyperspectral image formed by adjacent cuboids centered at each pixel, denoted as X0∈H. B×C×H×W In the formula, B and C represent the batch size and channel dimension of the patch, respectively, and H and W represent the height and width of the patch, respectively.

[0117] Active remote sensing data acquired in the same region as the hyperspectral image are normalized, and patches of the active remote sensing data are extracted in the same way, denoted as Y0∈H. B×C×H×W ;

[0118] The hyperspectral image classification framework is trained in multiple rounds using patches of hyperspectral images and patches of active remote sensing data in batches. Based on the contrastive learning mechanism, the feature similarity between hyperspectral images and active remote sensing data is learned. The framework parameters are optimized until convergence using a hybrid loss function, which includes contrastive loss (NT-XentLoss) and cross-entropy loss (CELoss).

[0119] Example 8:

[0120] This embodiment, based on Embodiment Seven, is further designed in that the expression for the hybrid loss function in this example is:

[0121]

[0122] In the formula, For cross-entropy loss, The weights are those corresponding to the cross-entropy loss. For comparison loss, κ is the weight corresponding to the comparison loss; κ is generated by the softmax function from an initial weight of 0.5, with the following constraints: The weights of the two loss functions are determined by parameters. And κ dynamically adjust.

[0123] Example 9:

[0124] This embodiment, based on Embodiment Eight, is further designed in that the expression for the contrast loss in this example is:

[0125]

[0126] In the formula, To compare the normalized temperature-scale cross-entropy loss in the loss function, z i and z j These represent the final feature representations of the HIS branch and the active remote sensing branch, respectively; sim(u,v)=u T v represents the cosine similarity, and τ is the temperature parameter, τ = 0.07. During network training, the contrastive learning mechanism helps the model learn the inherent similarity between HSI and LiDAR data, thereby improving the fusion of cross-modal information.

[0127] Example 10:

[0128] This embodiment, based on Embodiment 8, is further designed in that the expression for the cross-entropy loss in this example is:

[0129]

[0130] In the formula, C is the number of categories, p c The classification result output by the classification head in the hyperspectral image classification framework corresponds to the true label y. c The predicted probability.

[0131] Example 11:

[0132] An electronic device according to the present invention includes a memory and a processor. The memory stores a computer program, and the processor is used to call and run the computer program stored in the memory to perform the method as described in any of the above embodiments.

[0133] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above embodiments.

[0134] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A hyperspectral image classification framework under multi-degradation conditions guided by cross-modal characteristics, characterized in that, The hyperspectral image classification framework includes a two-branch structure consisting of an HSI branch and an active remote sensing branch, and a classification head consisting of linear layers. The HSI branch, in accordance with the processing order of hyperspectral images under multiple degradation conditions, includes a first convolutional layer for extracting basic features of hyperspectral images, a first HyperGroupMix module for performing the first cross-sample style transfer and outputting the first layer of HSI feature map, a cross-modal feature pyramid guidance module for cross-modal guidance and multi-scale feature fusion and outputting HSI fused feature map, a multi-head self-attention mechanism module, and a projection layer. The active remote sensing branch, following the processing order of active remote sensing data from the same region as the hyperspectral image, includes: a third convolutional layer for initial feature extraction and providing the HSI branch with a first active remote sensing feature map whose scale and dimension match the first layer HSI feature map; a first downsampling layer for first-level scale compression; a fourth convolutional layer for second feature extraction and providing the HSI branch with a second active remote sensing feature map whose scale and dimension match the third layer HSI feature map; a second downsampling layer for second-level scale compression and outputting the final active remote sensing feature map; a multi-head self-attention mechanism module; and a projection layer. The HSI fused feature map and the final active remote sensing feature map are sequentially passed through the multi-head self-attention mechanism module and the projection layer of the corresponding branch, respectively, and then cross-modal feature interaction is performed. After passing through the classification head, the hyperspectral image classification is completed.

2. The hyperspectral image classification framework under multi-degradation conditions guided by cross-modal methods according to claim 1, characterized in that, The cross-modal feature pyramid guidance module includes a first cross-modal feature merging module that performs initial feature fusion of the first-layer HSI feature map and an active remote sensing feature map whose scale and dimension match the first-layer HSI feature map; a second HyperGroupMix module that performs a second cross-sample style transfer and outputs a second-layer HSI feature map; a second convolutional layer that performs feature compression; a third HyperGroupMix module that performs a third cross-sample style transfer and outputs a third-layer HSI feature map; and a second cross-modal feature merging module that performs a second feature fusion of the third-layer HSI feature map and an active remote sensing feature map whose scale and dimension match the third-layer HSI feature map and outputs a fused HSI feature map.

3. The hyperspectral image classification framework under multi-degradation conditions guided by cross-modal methods according to claim 2, characterized in that, The specific methods for feature fusion by the cross-modal feature merging module include: Receive scale- and dimension-matched HSI feature maps X∈H B×C×H×W and active remote sensing feature map Y∈L B×C×H×W Where X corresponds to the HIS branch, Y corresponds to the LiDAR branch, B and C represent the batch size and number of channels, respectively, and H and W represent the spatial height and width of the feature map; Spatial compression of HSI and active remote sensing feature maps is performed using non-overlapping 2×2 windows. This enhances local structure representation during downsampling by reshaping local spatial pixels within each window to channel dimensions. Feature block representations are obtained from both the HSI and LiDAR modes for each channel. For channel i, the feature block representation obtained from the HSI mode is denoted as the HSI feature block. The feature block representation obtained from LiDAR modes is denoted as the active remote sensing feature block. For any channel i, connect the HSI feature block and the active remote sensing feature block along the channel dimension to form a channel-level fused feature representation. The fused feature is obtained by aggregating all channel-level fused feature representations. Applying 1×1 convolution to compress the channel dimensions yields the final ensemble multimodal representation, i.e., the fused feature map, which is represented as follows:

4. The hyperspectral image classification framework under multi-degradation conditions guided by cross-modal methods according to claim 2, characterized in that, The specific methods for cross-sample style transfer in the HyperGroupMix module include: The input feature map F is divided into several subsets along the channel dimension, denoted as F = {f i |i=1,2,...,n}, where f i Let i be the i-th subset, and n be the total number of subsets; Calculate the spectral spatial feature statistics for each subset, wherein the spectral spatial feature statistics include the channel average μ(f i ), standard deviation σ(f) i ) and median m(f i )as follows: Where H and W represent the height and width of the feature map, respectively, and ε is a constant; Calculate the Gram matrix g0(f) for each feature map i ): Where, flat(f i ) represents the i-th subset f i The operation of flattening the spatial dimension into a tensor of shape (B, C, H × W), where I represents the identity matrix; 10 -3 I represents a higher-order perturbation term used to improve numerical stability, 10 -6 I represents a low-order perturbation term used to enhance computational robustness; Based on Gram matrix g0(f) i The Gram matrix is ​​aligned with the centered representation of the grouped features by applying a whitening transformation, as shown in the expression: g0(f i )=VΛV T W=VΛ -1 / 2 V T Where V is the eigenvector matrix, Λ is the diagonal matrix of eigenvalues, and W is the whitening matrix; Finally, obtain the whitened feature representation g(f) i ): g(f i )=W(f i -μ(f i )) Randomly arranged samples f in the same batch i The whitened feature representation g(f) i ) and median m(f i Shuffle the order along the batch dimension to obtain a new random permutation of samples f. i The whitened feature representation of ′ is g(f i ′) and median m(f i ′); Sample the mixing coefficient λ from the symmetric Beta distribution Beta(α,α) such that λ approaches 0 or 1; generate the biased first mixture characteristic statistic α using the following formula. mix Second mixed characteristic statistic ρ mix ; α mix =λg(f i )+(1-λ)g(f i ′) ρ mix =λm(f i )+(1-λ)m(f i ′) Subsequently, the features of cross-sample style transfer were reconstructed using mixture feature statistics. Where ⊙ represents element-wise multiplication; In the output stage, features are concatenated along the channel dimension. To restore the original input shape, the scale of the HSI feature map after cross-sample style transfer is obtained.

5. The hyperspectral image classification framework under multi-degradation conditions guided by cross-modal methods according to claim 1, characterized in that, The degradation conditions of the hyperspectral image include at least one of additive Gaussian noise, cutoff line noise, convolutional blur noise, stripe noise, Poisson noise, salt-and-pepper noise, and zero-mean Gaussian noise.

6. A hyperspectral image classification method under multi-degradation conditions guided by cross-modal characteristics, characterized in that... include: Construct and train the hyperspectral image classification framework under multi-degradation conditions guided by any one of claims 1 to 5; The degraded hyperspectral image of the target and active remote sensing data of the same region as the hyperspectral image are input into the hyperspectral image classification framework to obtain the output result of the hyperspectral image classification framework.

7. The hyperspectral image classification method under multi-degradation conditions guided by cross-modal guidance according to claim 6, characterized in that, The training method for the hyperspectral image classification framework includes: Data preparation involves performing principal component analysis and normalization on the undegraded hyperspectral image, and extracting patches of the hyperspectral image formed by adjacent cuboids centered at each pixel, denoted as X0∈H. B×C×H×W In the formula, B and C represent the batch size and channel dimension of the patch, respectively, and H and W represent the height and width of the patch, respectively. Active remote sensing data acquired in the same region as the hyperspectral image are normalized, and patches of the active remote sensing data are extracted in the same way, denoted as Y0∈H. B×C×H×W ; The hyperspectral image classification framework is trained in multiple rounds using patches of hyperspectral images and patches of active remote sensing data as input data in batches. Based on the contrastive learning mechanism, the feature similarity between hyperspectral images and active remote sensing data is learned. The framework parameters are optimized until convergence using a hybrid loss function, which includes contrastive loss and cross-entropy loss.

8. The hyperspectral image classification method under multi-degradation conditions guided by cross-modal guidance according to claim 7, characterized in that, The expression for the hybrid loss function is: In the formula, For cross-entropy loss, The weights are those corresponding to the cross-entropy loss. For comparison loss, κ is the weight corresponding to the comparison loss; 9. The hyperspectral image classification method under multi-degradation conditions guided by cross-modal analysis according to claim 8, characterized in that, The expression for the contrast loss is: In the formula, To compare the normalized temperature-scale cross-entropy loss in the loss function, z i and z j These represent the final feature representations of the HIS branch and the active remote sensing branch, respectively. sim (u,v)=u T v represents the cosine similarity, and τ represents the temperature parameter, τ = 0.

07.

10. The hyperspectral image classification method under multi-degradation conditions guided by cross-modal analysis according to claim 8, characterized in that, The expression for the cross-entropy loss is: In the formula, C is the number of categories, p c The classification result output by the classification head in the hyperspectral image classification framework corresponds to the true label y. c The predicted probability.

Images