Disturbance perception-based multi-modal remote sensing ground feature image classification method and system

Abstract

The invention discloses a multi-modal remote sensing ground feature image classification method and system based on disturbance perception, and the method comprises the steps: firstly inputting a hyperspectral image and a laser radar image, carrying out the preprocessing, carrying out the feature extraction and disturbance embedding based on the two types of images after preprocessing, and obtaining complementary features through a network backbone structure. Secondly, based on the complementary features, obtaining a corresponding probability confidence coefficient through a modal confidence coefficient evaluation module; then, based on the complementary features, obtaining a prediction probability through a classification head, combining probability confidence, constructing a disturbance sample and a loss function of direction constraint, and finally, based on the disturbance sample and the loss function, carrying out reverse training, carrying out testing, and outputting a remote sensing ground feature image classification result. According to the method, the discrimination precision of multi-modal feature fusion is remarkably improved, the discrimination capability of the model for different ground feature categories is enhanced, and the ground feature classification robustness in a complex environment is improved.

Description

A Multimodal Remote Sensing Ground Feature Image Classification Method and System Based on Perturbation Sensing Technical Field

[0001] This invention relates to the fields of image processing and deep learning technology, and is a multimodal remote sensing image classification method and system based on perturbation perception. Background Technology

[0002] In the field of remote sensing, with the continuous advancement of Earth observation technology and hardware platforms, satellite sensors are able to acquire multi-source remote sensing image data, including visible light, multispectral, hyperspectral, and lidar data. This data is widely used in environmental monitoring, vegetation mapping, and urban planning. In recent years, researchers have proposed various deep learning-based remote sensing image classification methods, but classification based on a single data source still has limitations. For example, while hyperspectral images contain rich spectral information, they suffer from severe band redundancy and complex textures, making them susceptible to minor perturbations that can lead to classification errors. Lidar data can provide spatial structure and elevation information of ground features, but it lacks spectral dimensions and is prone to occlusion or loss during acquisition.

[0003] To fully leverage the complementarity of multi-source data, researchers have proposed multimodal fusion methods at the pixel, decision, and feature levels, achieving certain performance improvements within deep learning frameworks. However, existing fusion methods still suffer from the following shortcomings: First, information redundancy and imbalance between modalities lead to insufficient fusion feature representation; second, the model is susceptible to adversarial perturbations or anomalous samples during training, especially the fine-grained features of hyperspectral modalities, which are more easily destroyed, causing discrimination failure; third, existing optimization strategies are mostly fixed in form, lacking dynamic adjustment mechanisms, resulting in low convergence efficiency and insufficient robustness.

[0004] Especially in security-sensitive remote sensing applications, even the latest convolutional neural network (CNN) and visual transformer (ViT) models are susceptible to attacks from minute perturbations that are imperceptible to the human eye, leading to serious biases in classification results. Therefore, how to simultaneously improve classification accuracy and robustness against perturbations in classification tasks that fuse hyperspectral and lidar data has become a critical technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a multimodal remote sensing ground object image classification method and system based on perturbation perception. It aims to solve issues in existing technologies such as weak ability to identify anomalous samples, susceptibility to perturbations during training, inconsistent distribution among modalities, and low convergence efficiency. This invention achieves stable modeling and robust discrimination of multimodal features through modules such as multimodal statistical consistency calibration, modality-guided perturbation embedding, modality confidence assessment, orientation-constrained perturbation generation, adaptive fusion, and dynamic adjustment, thereby improving classification performance in complex scenarios.

[0006] In one aspect, this invention provides a multimodal remote sensing image classification method based on perturbation perception, the technical solution of which is as follows:

[0007] The first step is to input hyperspectral images and LiDAR images, and then perform preprocessing. The input hyperspectral image is denoted as... The lidar image is denoted as First, the input data is normalized to stabilize the numerical range of different channels. Then, the LiDAR image is resampled and geometrically registered to align it with the hyperspectral image in terms of spatial location and resolution. Finally, using the center pixel as an anchor point, the hyperspectral image and the aligned LiDAR image are cropped into fixed-size sub-blocks. Multimodal consistency calibration is then performed on these sub-blocks. Specifically, after cropping, local statistical features are recalculated for both sub-blocks, and a learnable lightweight calibration operator is used to align the local distributions of the two modalities, resulting in a hyperspectral block with consistency calibration. and lidar block .

[0008] The second step involves feature extraction and perturbation embedding based on the two types of preprocessed images. Complementary features are then obtained through the network backbone structure. The hyperspectral patches in the preprocessed data... Feature extraction operators used in the hyperspectral branch for joint modeling of spectral and spatial dimensions Processing to obtain hyperspectral features Pre-processed lidar block Partial extraction operator of spatial structure information of ground features via LiDAR branch Processing yields the modal guiding vector. To enhance the ability to discriminate structurally sensitive directions, this invention applies guidance vectors in the feature domain. After unitization, we obtain .

[0009] Then via broadcast operator Normalized vector Extended to hyperspectral features Isomorphic tensors This involves replicating the channel values ​​of the guiding vector in the spatial dimension. Finally, the expanded tensor is weighted and summed with the hyperspectral features to form the embedded features:

[0010]

[0011] in This refers to the embedding strength coefficient. Embedding features. The features are obtained by combining local convolutional features and global attention features under modal guidance.

[0012] The third step involves obtaining the corresponding probability confidence level based on complementary features through a modality confidence assessment module. To enhance adaptability to changes in modality quality during multimodal fusion, this invention incorporates a modality confidence assessment module during feature input. The complementary features output from the hyperspectral branch are used to output modality prediction vectors via a lightweight auxiliary classification head. The hyperspectral modal prediction distribution was obtained by Softmax. ,Will The maximum value of the element in the hyperspectral probability confidence level is defined. The LiDAR branch is based on Output mode prediction vector Similarly, the modal prediction distribution is obtained. and LiDAR probability confidence . They represent the first The confidence levels of each sample in the hyperspectral and LiDAR modes will serve as the fundamental quantities for measuring modal reliability in the fusion of perturbation intensity and discrimination.

[0013] The fourth step involves obtaining predicted probabilities based on complementary features through the classification head, and constructing perturbation samples with directional constraints and a loss function by combining probability confidence. During training, complementary features are used to output discriminant vectors through the backbone classification head. ,through Obtain the predicted probability The real label is recorded as Let the classification loss function be... .

[0014] To construct a perturbation direction consistent with the structural information, the classification loss for the hyperspectral input is first calculated. gradient Agreement This means that the tensor is flattened into a vector on a sample-by-sample scale before the dot product or norm is calculated. Describing the L2 norm, To prevent small constants from being divided by zero, the broadcast tensor obtained in step three is used. As the direction of the structural subspace, the gradient is projected along this direction and normalized to obtain:

[0015]

[0016]

[0017] Using hyperspectral probability confidence Define sample-level perturbation intensity:

[0018] in This is to update the obtained perturbation intensity factor. This is the adjustment coefficient. Construct directional constraint perturbation samples:

[0019]

[0020] Next, complementary features are obtained from the original samples and perturbed samples, and then discriminant vectors are obtained through the backbone classification head. and And define the sample-level difference measure as:

[0021]

[0022] Let the basic weights be:

[0023]

[0024] in express function, To integrate the sensitivity coefficient, This is a reference threshold.

[0025] Constructing a fusion modality adjustment factor using modal confidence:

[0026]

[0027] To prevent division by zero of positive numbers. Calculate the fusion weights And generate the final discriminant vector. Reconstruction of disturbance loss:

[0028]

[0029] For the sample size, Combined with classification loss, it forms the total loss. In this way, the perturbation samples are directly used to constrain the stability of the discriminative representation, and the robustness of the decision is improved through adaptive fusion.

[0030] The fifth step involves reverse training based on the perturbation samples and the loss function, followed by testing, to output the classification results of the remote sensing ground object image. This invention proposes a dynamic scheduling mechanism based on loss variation. Let the... The total loss in rounds of training is Then, construct a feedback signal based on the rate of change of losses in two consecutive rounds:

[0031]

[0032] The above formula To prevent division by zero by small constants, the learning rate and perturbation strength factor are updated as follows:

[0033]

[0034]

[0035] in The initial learning rate, The initial disturbance intensity, This is the learning rate adjustment coefficient. This is the perturbation strength adjustment coefficient. Parameter updates follow the standard gradient descent form. This mechanism ensures that when the loss decreases, the learning rate is moderately increased and the perturbation strength is weakened to accelerate convergence; when the loss stagnates or increases, the learning rate is decreased and the perturbation strength is increased to increase exploration.

[0036] The input data undergoes the same preprocessing and embedding process to obtain the final discriminant vector. .pass Transform it into a class probability distribution Indicates the first The sample belongs to the first The predicted probability of the class. The final classification result is given by the class index corresponding to the highest probability, here. Indicates the first The predicted category of each test sample.

[0037] In another aspect, the present invention also provides a multimodal remote sensing ground object image classification system based on perturbation perception, comprising the following modules:

[0038] The preprocessing module is used to input hyperspectral images and lidar images and perform preprocessing.

[0039] The complementary feature acquisition module performs feature extraction and perturbation embedding based on the two types of preprocessed images, and obtains complementary features through the network backbone structure.

[0040] The probability confidence module, based on complementary features, obtains the corresponding probability confidence through the modal confidence evaluation module.

[0041] The perturbation sample and loss function construction module, based on complementary features, obtains the predicted probability through the classification head, and combines the probability confidence to construct the perturbation sample and loss function with orientation constraints.

[0042] The classification output module performs inverse training based on perturbation samples and a loss function, and then tests and outputs the classification results of remote sensing ground object images.

[0043] Beneficial effects of this invention:

[0044] Enhancing the robustness of ground feature classification in complex environments. Through a perturbation generation mechanism with directional constraints and confidence assessment and weighting, the perturbation direction is kept consistent with the structural information guided by the modality, effectively improving the model's stability in complex scenarios such as noise interference and blurred ground feature boundaries. Furthermore, when the confidence of a certain modality decreases, the perturbation amplitude of the corresponding sample increases, and the dependence on it during fusion weakens. This automatically biases towards more reliable modalities and discrimination paths when modal quality is unstable or anomalies exist, improving ground feature identifiability under low visibility conditions.

[0045] Improve the consistency and fusion accuracy of classification representations. Learnable alignment of hyperspectral and lidar sub-blocks is performed at the local level, significantly reducing the interference of inconsistent distribution between modes on classification. Consistency loss and adaptive fusion strategy are introduced during training to ensure that the discrimination results of clean and perturbed samples remain stable, significantly improving the discrimination accuracy of multimodal feature fusion and enhancing the model's ability to distinguish different land cover categories.

[0046] It mitigates overfitting and accelerates convergence. The dynamic scheduling mechanism can accelerate convergence when the loss decreases and enhance exploration when the loss stagnates or increases, effectively mitigating overfitting and improving the model's training efficiency and generalization ability.

[0047] This invention improves the overall performance of multimodal remote sensing land cover classification. Validated on multiple publicly available remote sensing datasets, the results show that, in scenarios fusing hyperspectral and lidar data, the classification accuracy is significantly better than baseline methods using single-modal or unperturbed constraints, especially in cases with complex land cover category boundaries and imbalanced sample distribution. Attached Figure Description

[0048] Figure 1 is a flowchart of the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] In one aspect, this invention provides a multimodal remote sensing image classification method based on perturbation perception, applicable to the fusion of hyperspectral images and lidar images for ground feature identification and classification, as shown in Figure 1. The overall system consists of data preprocessing and registration, multimodal consistency calibration, feature extraction and modality-guided embedding, modality confidence assessment, orientation-constrained perturbation generation and adaptive fusion, a dynamic scheduling mechanism based on loss variation, and an inference stage.

[0051] The first step involves inputting hyperspectral images and LiDAR images, followed by preprocessing. This embodiment uses publicly available remote sensing datasets (such as Trento and Houston), with inputs including hyperspectral images and LiDAR images, denoted as hyperspectral tensors. With LiDAR tensor First, the input data is normalized to stabilize the numerical range of different channels. For hyperspectral images, each spectral channel of each sample is scaled to the [0,1] interval using a Min–Max scaling method, i.e., for each pixel... implement:

[0052]

[0053] in To prevent small constants from being divided by zero, they are still denoted as after normalization. For LiDAR images, to reduce the impact of extreme values, truncation can be performed first by channel using percentiles, followed by Min-Max normalization. Let the lower percentile be... High percentile (e.g., 1% and 99%), for pixels First, truncate it as follows:

[0054]

[0055] in Indicates the corresponding channel quantiles, then Scale to Min–Max The normalized LiDAR image is still recorded as The LiDAR image is then resampled and geometrically registered to align it with the hyperspectral image on the same spatial grid. Let the original spatial resolution of the LiDAR be... The hyperspectral image resolution is The preprocessing section first... Resampling to the hyperspectral grid via bilinear interpolation is denoted as follows: ,have Based on this, the feature control point sets of the source and target images are... Estimate the affine transformation matrix With translation vector After outliers are removed using RANSAC, the registered LiDAR image is still recorded as... Then, using the center pixel as the anchor point, a fixed-size sub-block is cropped. Let the cropping window size be... (In this embodiment, P=11) The corresponding hyperspectral and LiDAR sub-blocks are cropped at each effective pixel position to form the subsequent basic input.

[0056] Considering that differences in local brightness, contrast, and texture scale still exist between hyperspectral and lidar data after normalization, this embodiment further introduces a learnable multimodal consistency calibration module at the sub-block level. Statistical features such as local mean, local variance, and local gradient magnitude are calculated for both types of sub-blocks within a local window, and these statistics, along with the original sub-blocks, are input into a shallow convolutional calibration network. This network performs alignment transformation on the local distributions of the two modes through several layers of small convolutions and nonlinear activations, outputting a hyperspectral block after consistency calibration. and lidar block The parameters of the calibration network are automatically updated through backpropagation during the overall training process, making the hyperspectral and LiDAR local contrast and texture intensity more consistent, thereby reducing the impact of intermodal statistical inconsistencies on subsequent deep feature extraction and perturbation modeling. Pairs of multimodal samples are obtained. and the category label corresponding to its center pixel The dataset is partitioned using stratified sampling by category, dividing the samples into training, validation, and test sets (e.g., 7:1:2), while ensuring that the proportion of different categories in each partition remains stable to reduce estimation bias caused by class imbalance.

[0057] The second step involves feature extraction and perturbation embedding based on the two types of preprocessed images. Complementary features are then obtained through the network backbone structure. The preprocessed hyperspectral blocks... Feature extraction operators used in the hyperspectral branch for joint modeling of spectral and spatial dimensions Processing yields a two-dimensional feature map. In this embodiment, This is implemented using a 3D convolutional network. Input First, by using a kernel size of 3D convolution (spectral dimension filling) , spatial dimension filling Extracting spectral-spatial joint features yields... Subsequently through The 3D convolution maps channels to the number of intermediate channels. (In this embodiment, we take) = 16), obtained Based on this, Learnable aggregation (SpecPool) is performed on the spectral dimension to compress the spectral dimension into two-dimensional spatial features, forming the final hyperspectral features. ).in ) represents the spectral dimension rearrangement and compression operator, used to aggregate high-dimensional spectral information into a two-dimensional feature representation, ultimately obtaining .

[0058] LiDAR image part of the spatial structure information extraction operator of ground features via LiDAR branch The modal guiding vector is obtained through processing. In this embodiment, It consists of two-dimensional convolution and global average pooling. Input First, the kernel size is determined. A two-dimensional convolution with padding 1, followed by GELU activation, yields spatial structure features. Subsequently Perform global average pooling (GAP) in the spatial dimension to obtain the modality guidance vector. To enhance the ability to discriminate structurally sensitive directions, this invention applies guidance vectors in the feature domain. After normalization, we get:

[0059]

[0060] in > 0 is a small constant to prevent division by zero. This is then broadcast via the operator. The normalized vector is extended to be related to the hyperspectral features. Isomorphic tensors, i.e. Finally, the expanded tensor is weighted and added to the hyperspectral features to form the embedded features:

[0061]

[0062] in This is the embedding strength coefficient. In this embodiment... A value of 0.5 is acceptable. Embedded features. The features are obtained by combining local convolutional features and global attention features under modal guidance.

[0063] The third step involves obtaining the corresponding probability confidence level based on complementary features using a modal confidence assessment module. To improve the adaptability to modal quality changes during multimodal fusion, this embodiment incorporates modal confidence assessment in the feature section. Hyperspectral branching is used in the embedding feature... Add a lightweight auxiliary classification head: First, for Global average pooling is performed in the spatial dimension to compress it into a sample-channel vector, which is then mapped through a single fully connected layer to... 3D category space, output modality prediction vector ,through The hyperspectral modal prediction distribution is obtained, and the hyperspectral probability confidence level is defined accordingly: The LiDAR branch is based on normalized guiding vectors. Modality prediction vectors are obtained through a single-layer fully connected mapping. Similarly The modal prediction distribution and LiDAR probability confidence scores are obtained: here They represent the first The confidence scores for each sample in the hyperspectral and LiDAR modes are within the range of [value missing]. Both types of confidence levels will serve as fundamental quantities for measuring modal reliability in the fusion of disturbance strength and discrimination.

[0064] The fourth step involves obtaining predicted probabilities based on complementary features through the classification head, and constructing perturbation samples with directional constraints and a loss function by combining probability confidence. During training, complementary features are used to output discriminant vectors through the backbone classification head. ,through Obtain the predicted probability The real label is recorded as , Indicates batch size, Let represent the total number of categories. Let the classification loss function be... To construct a perturbation direction consistent with the structural information, the classification loss for the hyperspectral input is first calculated. gradient Agreement This means that the tensor is flattened into a vector on a sample-by-sample scale before the dot product or norm is calculated. Describing the L2 norm, To prevent small constants from being divided by zero, the broadcast tensor obtained in step three is used. As the direction of the structural subspace, the gradient is projected along this direction and normalized to obtain:

[0065]

[0066]

[0067] Using hyperspectral confidence Define sample-level perturbation intensity:

[0068] in To update the obtained perturbation intensity factor, the scheduling module adaptively and dynamically adjusts and trims it to a preset range during the training process. This is the adjustment coefficient. Therefore, the directional constraint perturbation sample is constructed:

[0069]

[0070] Then, the original samples and the perturbed samples are passed through the same main branch to obtain the discriminant vector. and And define the sample-level difference measure as:

[0071]

[0072] Let the basic weights be:

[0073]

[0074] in express function, To integrate the sensitivity coefficient, This is a reference threshold.

[0075] Constructing a fusion modality adjustment factor using modal confidence:

[0076]

[0077] To prevent division by zero of positive numbers. The fusion weights are calculated as follows: And generate the final discriminant vector: Reconstruction disturbance loss:

[0078]

[0079] For the sample size, Combined with classification loss, it forms the total loss:

[0080]

[0081] in This is used as the consistency weight. In this way, the perturbation samples are directly used to constrain the stability of the discriminative representation, and the robustness of the decision is improved through adaptive fusion.

[0082] The fifth step involves reverse training based on the perturbation samples and the loss function, followed by testing, and outputting the remote sensing ground object image classification results. This invention introduces a dynamic scheduling mechanism based on loss variation during training, linking the learning rate and perturbation intensity to improve the stability and convergence efficiency of the optimization process. The total loss in rounds of training is At the end of each training round, utilize the losses from the previous round. Compared with the current round of losses Construct the normalized rate of change of loss: in To prevent small constants from being divided by zero. Based on the rate of change of loss. In this embodiment, the learning rate and perturbation strength factor are adaptively adjusted respectively. The learning rate is updated as follows: The initial learning rate, This is the learning rate adjustment coefficient. The update of the perturbation strength follows a design principle that is complementary to the learning rate: The initial disturbance intensity, This is the perturbation intensity adjustment coefficient. When the model training is in the phase of steadily decreasing loss... For positive, If positive, then... relatively The loss decreases slightly, thus reducing the randomness introduced by the disturbance and highlighting the role of the main path discrimination; however, when the loss stagnates or even increases, Negative, If it is negative, then... It automatically enhances the ability to explore perturbation paths, guiding the model to avoid getting trapped in local minima or regions sensitive to anomalous samples through stronger directional constraint perturbations. Parameter updates follow the standard gradient descent form: In implementation, the above update form can be directly used in an SGD-based optimizer, or it can be combined with an adaptive optimization algorithm. In the latter case, The global learning rate is considered as the outer layer, while the corrections for the first and second moments are maintained internally by the optimizer. The perturbation strength factor is updated in this step. That is, the above is used to define the sample-level perturbation intensity The base value.

[0083] No training perturbations are injected during the inference phase. The input data undergoes the same preprocessing, multimodal consistency calibration, and modality-guided embedding process as in the training phase to obtain complementary features. The final discriminant vector is calculated solely based on the clean path and the fusion module. .pass Transform it into a class probability distribution ,in Indicates the first The sample belongs to the first The predicted probability of a class is given by the index of the class corresponding to the highest probability, i.e.: Because the training process incorporates directional constraint perturbation, consistency loss, and confidence-based adaptive fusion, this embodiment can maintain discrimination stability and robustness in complex remote sensing scenarios (including noise, shadows, terrain undulations, and material changes) without adding extra computation or perturbation injection during the inference phase.

[0084] In another aspect, the present invention also provides a multimodal remote sensing ground object image classification system based on perturbation perception, comprising the following modules:

[0085] The preprocessing module is used to input hyperspectral images and lidar images and perform preprocessing.

[0086] The complementary feature acquisition module performs feature extraction and perturbation embedding based on the two types of preprocessed images, and obtains complementary features through the network backbone structure.

[0087] The probability confidence module, based on complementary features, obtains the corresponding probability confidence through the modal confidence evaluation module.

[0088] The perturbation sample and loss function construction module, based on complementary features, obtains the predicted probability through the classification head, and combines the probability confidence to construct the perturbation sample and loss function with orientation constraints.

[0089] The classification output module performs inverse training based on perturbation samples and a loss function, and then tests and outputs the classification results of remote sensing ground object images.

[0090] experiment:

[0091] The implementation environment was configured as follows: the system was deployed on a Tencent Cloud server running macOS, using the PyTorch 2.6.0 framework, with an NVIDIA A100 GPU and 40GB of VRAM, supporting multimodal parallel training. In terms of experimental setup, all models used the same training and test set partitioning strategy, with a uniform training epoch of 100 epochs, an initial learning rate of 0.0005, an Adam optimizer, and a loss function consisting of cross-entropy and a perturbation consistency term. This invention introduces a perturbation feedback path and a periodically adjusted learning rate update strategy during training. The perturbation intensity was dynamically adjusted from 0.01 to 0.05, and the periodic adjustment function used a smooth nonlinear mapping function. The outlier sample construction strategy included: injecting Gaussian noise (standard deviation of 0.02) into the test samples; randomly occluding image regions (3×3 or 5×5 in size) to simulate sensor absence or occlusion scenarios. The convergence epoch was defined as the training epoch that first achieved 95% overall accuracy.

[0092] To verify the effectiveness of the method of this invention, the publicly available remote sensing datasets Houston and Trento were selected as experimental platforms. Under a unified training strategy, several existing classification methods were compared, including multimodal fusion networks, attention-enhanced structures, and cross-resolution feature fusion models. The experimental results are shown in Table 1. Comparing the performance of the four latest classification methods, the method of this invention achieved the best performance in terms of classification accuracy, average class accuracy, and Kappa coefficient. Specifically, it achieved the highest recall rate for 9 out of 15 land cover classes, fully demonstrating the discriminative power and robustness advantages of this invention in multimodal remote sensing classification tasks.

[0093] Table 1. Comparison of Classification Effects of Different Methods

[0094]

[0095] In terms of classification accuracy, the method of this invention demonstrates significant advantages across all metrics, particularly in road and building feature recognition, achieving accuracies of 99% and 97% respectively, significantly outperforming other methods. Regarding robustness, a perturbation test set was further constructed, simulating anomalous samples by introducing Gaussian noise and occlusion regions into the original test samples. Results show that the accuracy decrease of the method on perturbation samples is controlled within 2%, indicating that the perturbation perception mechanism and adaptive scheduling strategy proposed in this invention can effectively improve the model's stability under anomalous samples. In terms of convergence efficiency, the method of this invention achieves an accuracy of over 95% by the 35th training round, verifying the advantage of the periodic adjustment mechanism in improving training efficiency.

Claims

1. A multimodal remote sensing image classification method based on perturbation perception, characterized in that, Includes the following steps: Step 1: Input hyperspectral images and lidar images, and perform preprocessing; Step 2: Based on the preprocessed images, perform feature extraction and perturbation embedding, and obtain complementary features through the network backbone structure; Step 3: Pass the complementary features through the modality confidence evaluation module to obtain the corresponding probability confidence; Step 4: Based on the complementary features, obtain the predicted probability through the classification head, and construct directionally constrained perturbation samples and loss functions by combining the probability confidence; Step 5: Based on the perturbation samples and loss functions, perform reverse training, test, and output the remote sensing ground object image classification results.

2. The multimodal remote sensing ground feature image classification method based on perturbation perception according to claim 1, characterized in that, The preprocessing is specifically implemented as follows: the input hyperspectral image is denoted as... The lidar image is denoted as First, the input data is normalized. Then, the LiDAR image is resampled and geometrically registered to align it with the hyperspectral image in terms of spatial location and resolution. Finally, the hyperspectral image and the aligned LiDAR image are cropped into fixed-size sub-blocks using the center pixel as the anchor point, and the sub-blocks are calibrated for multimodal consistency.

3. The multimodal remote sensing ground feature image classification method based on perturbation perception according to claim 2, characterized in that, The multimodal consistency calibration is specifically implemented as follows: after pruning, the local statistical features of the two sub-blocks are recalculated, and the local distributions of the two modes are aligned using a learnable lightweight calibration operator to obtain a hyperspectral block after consistency calibration. and lidar block 。 4. The multimodal remote sensing ground feature image classification method based on perturbation perception according to claim 3, characterized in that, Step 2 is specifically implemented as follows: hyperspectral blocks in the preprocessed data... Feature extraction operators used in the hyperspectral branch for joint modeling of spectral and spatial dimensions Processing to obtain hyperspectral features ； Preprocessed lidar block Ground feature spatial structure information extraction operator via LiDAR branch Processing yields the modal guiding vector. ; in the feature domain, the guiding vector After unitization, we obtain ； Then via broadcast operator Normalized vector Extended to hyperspectral features Isomorphic tensors This involves replicating the channel values ​​of the guiding vector in the spatial dimension, and finally weighting and adding the expanded tensor with the hyperspectral features to form the embedded features. Embedded features The features are obtained by combining local convolutional features and global attention features under modal guidance.

5. The multimodal remote sensing ground feature image classification method based on perturbation perception according to claim 4, characterized in that, The feature extraction operator Specifically, the implementation is as follows: Input First, spectral-spatial joint features are extracted through three-dimensional convolution to obtain... Subsequently passed The 3D convolution maps channels to the number of intermediate channels. ,get ;right Learnable aggregation is performed on the spectral dimension to compress the spectral dimension into two-dimensional spatial features, forming the final hyperspectral features. The operator for extracting spatial structure information of ground features Specifically, the implementation is as follows: Input After two-dimensional convolution and GELU activation, spatial structure features are obtained. Subsequently Perform global average pooling in the spatial dimension to obtain the modal guidance vector. ; in the feature domain, the guiding vector After unitization, we obtain 。 6. The multimodal remote sensing ground cover image classification method based on perturbation perception according to claim 5, characterized in that, Step 3 is specifically implemented as follows: the complementary features output by the hyperspectral branch are used to output mode prediction vectors through the auxiliary classification head. The hyperspectral modal prediction distribution was obtained by Softmax. ,Will The maximum value of the element in the hyperspectral probability confidence level is defined. The LiDAR branch is based on Output mode prediction vector Similarly, the modal prediction distribution is obtained. and LiDAR probability confidence 。 7. The multimodal remote sensing ground feature image classification method based on perturbation perception according to claim 6, characterized in that, The specific implementation process of step 4 is as follows: During the training process, complementary features output discriminant vectors through the backbone classification head. ,through Obtain the predicted probability The real label is recorded as Let the classification loss function be... First, calculate the classification loss for the hyperspectral input. gradient Agreement This means flattening the tensor into a vector on a sample-by-sample scale before performing dot product or norm calculations, and then using the resulting broadcast tensor. As a structural subspace direction, first... and Perform the dot product, then combine with... Performing dot product to complete gradient projection yields the features. and to Normalization yields features ； Using hyperspectral probability confidence The sample-level perturbation intensity is defined as the perturbation intensity factor multiplied by... , recorded as ， This is the adjustment coefficient; use Constructing directional constraint perturbation samples: through Features Perform element-wise weighting, then combine with... By adding elements one by one, we obtain the constrained perturbation samples. Next, complementary features are obtained from the original samples and perturbed samples, and then a discriminant vector is obtained through the backbone classification head. and and will and The difference and the 2-norm are defined as a sample-level measure of difference. Let the basic weights be... go through The function obtains, denoted as ， To integrate the sensitivity coefficient, Use the reference threshold; utilize modal confidence. Compared to and The proportion of summation is used to construct a fusion adjustment factor. ;Depend on Calculate the fusion weights And generate the final discriminant vector. ,pass and Calculate the 2-norm by subtracting the values ​​and then take the mean to obtain the disturbance loss.

8. The multimodal remote sensing ground object image classification method based on perturbation perception according to claim 7, characterized in that, The specific implementation process of step 5 is as follows: A dynamic scheduling mechanism based on loss variation is proposed, assuming the first... The total loss in rounds of training is ； Then construct a feedback signal based on the rate of change of losses in two consecutive rounds. The learning rate is updated to the initial learning rate multiplied by 1. The perturbation intensity factor is updated to the initial perturbation intensity multiplied by ， This is the learning rate adjustment coefficient. This is the disturbance intensity adjustment coefficient; parameter updates follow the gradient descent form.

9. The multimodal remote sensing ground feature image classification method based on perturbation perception according to claim 8, characterized in that, Step 5 also includes, during the inference phase, the final discrimination vector obtained from the input data; through... This will be transformed into a category probability distribution, and the final classification result will be given by the category index corresponding to the highest probability.

10. A multimodal remote sensing ground feature image classification system based on perturbation perception, used to implement the multimodal remote sensing ground feature image classification method according to any one of claims 1 to 9, characterized in that, It includes the following modules: a preprocessing module, which is used to input hyperspectral images and lidar images and perform preprocessing; and a complementary feature acquisition module, which performs feature extraction and perturbation embedding based on the two types of images after preprocessing, and obtains complementary features through the network backbone structure. The probability confidence module, based on complementary features, obtains the corresponding probability confidence through the modality confidence evaluation module; the perturbation sample and loss function construction module, based on complementary features, obtains the predicted probability through the classification head, and constructs directional constrained perturbation samples and loss functions by combining probability confidence; the classification output module, based on perturbation samples and loss functions, performs reverse training, tests, and outputs the classification results of remote sensing ground objects images.

Images