Farmland surface disturbance and desertification monitoring method and system based on multi-modal remote sensing data
By using a shared-weight Transformer backbone network and a LoRA fine-tuning mechanism for modal debiasing, combined with cross-modal debiasing attention fusion and consistency constraints, the problems of modal bias and computational overhead in multimodal remote sensing change detection are solved, achieving high-precision and robust monitoring of farmland surface disturbance and desertification.
Patent Information
- Application Number
- CN202610119795.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-28
- Publication Date
- 2026-05-05
AI Technical Summary
Existing multimodal remote sensing change detection methods suffer from modal bias, high computational overhead, and are not conducive to edge deployment and rapid transfer learning. They are particularly difficult to achieve high-precision and robust change detection in complex environments.
A method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing data is adopted. Feature extraction is performed through a Transformer backbone network with shared weights, and a modality-independent low-rank adapter is introduced for debiasing. Combined with cross-modal debiasing attention fusion and consistency constraints, the loss function is optimized to achieve a balance and consistency between optical and SAR features.
It effectively suppresses modal bias, improves feature fusion balance, ensures the reliability and robustness of change detection results, supports high-precision monitoring in complex environments, reduces computational costs, and is suitable for land resource supervision, disaster monitoring, and urban expansion analysis.
Smart Images
Figure CN121982571A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing intelligent monitoring and ecological environment supervision technology. Specifically, it relates to a method and system for monitoring farmland surface disturbance and desertification based on multimodal remote sensing data, and in particular, a multimodal remote sensing monitoring method and system based on modal debiasing fine-tuning mechanism and consistency constraints. Background Technology
[0002] With the expansion of urbanization, the frequent occurrence of natural disasters, the increasing demand for ecological environment governance and land and resources management, and the continuous enhancement of remote sensing satellite observation capabilities, remote sensing change detection technology has become one of the core means of monitoring dynamic changes on the Earth's surface and is widely used in fields such as disaster monitoring, ecological environment assessment, and urban expansion analysis.
[0003] In farmland and its surrounding areas, human activities such as illegal sand mining can cause significant changes in landform, texture, and cover type, thereby damaging arable land structure and exacerbating land degradation. These changes typically manifest as surface disturbance features in remote sensing imagery, sharing similarities with desertification evolution in spatial morphology and temporal characteristics. Therefore, they can be uniformly modeled and detected through remote sensing monitoring.
[0004] Optical remote sensing imagery boasts advantages such as high spatial resolution and rich texture and spectral differences, making it the most widely used data source for change detection. However, optical sensors are significantly affected by lighting conditions and atmospheric conditions, especially under cloud cover, rain, snow, or nighttime imaging conditions, where information is limited, making stable and continuous dynamic monitoring difficult. In contrast, synthetic aperture radar (SAR) imagery offers all-weather, all-day imaging capabilities, penetrating clouds, fog, and dust, providing a reliable supplement for change extraction in complex environments.
[0005] In recent years, with the rapid development of deep learning methods, convolutional neural networks, dual-branch architecture models, and Transformer-based remote sensing change detection models have gradually become the mainstream technologies for change detection. Some studies have begun to explore joint feature learning of optical and SAR images to leverage their complementary advantages. However, existing multimodal change detection methods still suffer from the following problems: (1) Optical images are rich in texture information and often dominate in model training, resulting in insufficient contribution of SAR features and serious modal bias problems; (2) Existing fusion processes are mostly data-driven, lacking a unified model of the physical differences between optical reflection and radar backscattering, and the output lacks consistency constraints; (3) For specific downstream tasks, training the Vision Transformer (ViT) multimodal network with all parameters is computationally expensive, which is not conducive to edge deployment or rapid transfer learning.
[0006] Therefore, the research challenge remains to find a technology that can effectively suppress modal bias, enhance the consistency between optical and SAR decisions, and possess low-cost fine-tuning capabilities to improve the reliability and practicality of surface disturbance monitoring in complex scenarios, thereby achieving high-precision and robust change detection technology in complex environments. Summary of the Invention
[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for monitoring farmland surface disturbance and desertification based on multimodal remote sensing data.
[0008] A method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing data according to the present invention includes: Step S1: Acquire remote sensing images and preprocess them to obtain multimodal dual-temporal remote sensing images; The remote sensing images include optical remote sensing images and SAR remote sensing images; Step S2: Divide the multimodal dual-temporal remote sensing image and construct a token sequence containing modality coding and temporal coding. Input the sequence into a Transformer backbone network with shared weights for feature extraction to obtain optical modality features and SAR modality features. Step S3: Introduce a mode-independent low-rank adapter into the attention module of the Transformer backbone network, and adjust the gradient contribution based on the mode bias score to perform mode debiasing on optical mode features and SAR mode features; Step S4: Construct time-series phase difference variation features for the depolarized optical mode features and SAR mode features respectively, and fuse them to obtain fused features; Step S5: Generate a change probability map based on the fused features and apply consistency constraints between optical and SAR prediction results, then output the final change detection results.
[0009] Preferably, the Transformer backbone network includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer; the model input is multimodal dual-temporal samples. and change label image ; Where t1 and t2 represent the first and second time points, respectively; and Representing optical and SAR modes respectively; The optical modal input images representing times t1 and t2, SAR modal input images representing times t1 and t2; Multimodal dual-temporal remote sensing images are mapped into an embedding space through patch partitioning and token construction. Features are extracted and fused through a multi-layer network to obtain the final change detection result.
[0010] Preferably, based on the VIT-L backbone network, a mode bias score (MBS) is defined according to the ratio of mode gradient norm differences, and this index is used as a mode bias regularization term to weight the total loss; a cross-modal debiasing attention module is constructed, which attenuates intra-modal attention and enhances cross-modal attention through a bias correction matrix; MBS is defined as:
[0011] in, and Let these represent the sets of parameters related to the optical mode and the SAR mode, respectively, with corresponding gradient norms as follows: and Furthermore, to prevent the denominator from being zero, a small constant is introduced. .
[0012] Preferably, the optical prediction change map and the SAR prediction change map are respectively The consistency constraint loss is composed of the JS divergence term and the CD divergence term, and its formalization is as follows:
[0013] in, and These are the height and width of the input image, respectively. and Representing optical and SAR modes respectively. represents the prediction category; M represents the average distribution of the optical prediction change map and the SAR prediction change map; u and v are spatial location indices, where u represents the pixel index of the image in the horizontal direction and v represents the pixel index of the image in the vertical direction.
[0014] Preferably, the total loss function is used during model training. As an optimization objective, the AdamW optimizer was used for training, and the LoRA branch weight update method of the ViT-L backbone was used to freeze some parameters of the backbone and train only the low-rank adapter parameters. Total loss function for:
[0015]
[0016]
[0017] in The loss for the transformation detection task is a joint form of binary cross-entropy loss and Dice loss; Loss due to consistency constraints; It is the modal deviation canonical loss. MBS stands for Modal Deviation Fraction; This is a probability map of change prediction, where Y is the actual change label map; JS and CD are the Jensen-Shannon divergence and contrastive divergence, respectively; where... , This is a hyperparameter used to control the regularization term.
[0018] A monitoring system for farmland surface disturbance and desertification based on multimodal remote sensing data, provided by the present invention, includes: Module M1: Acquires and preprocesses remote sensing images to obtain multimodal dual-temporal remote sensing images; The remote sensing images include optical remote sensing images and SAR remote sensing images; Module M2: Divide the multimodal dual-temporal remote sensing image and construct a token sequence containing modality coding and temporal coding. Input the sequence into a Transformer backbone network with shared weights for feature extraction to obtain optical modality features and SAR modality features. Module M3: Introduces a mode-independent low-rank adapter into the attention module of the Transformer backbone network, and adjusts the gradient contribution based on the mode bias score to perform mode debiasing on optical mode features and SAR mode features; Module M4: Constructs and fuses time-phase differential variation features of the depolarized optical modal features and SAR modal features respectively to obtain fused features; Module M5: Generates a change probability map based on fused features and applies consistency constraints between optical and SAR prediction results, outputting the final change detection results.
[0019] Preferably, the Transformer backbone network includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer; the model input is multimodal dual-temporal samples. and change label image ; Where t1 and t2 represent the first and second time points, respectively; and Representing optical and SAR modes respectively; The optical modal input images representing times t1 and t2, SAR modal input images representing times t1 and t2; Multimodal dual-temporal remote sensing images are mapped into an embedding space through patch partitioning and token construction. Features are extracted and fused through a multi-layer network to obtain the final change detection result.
[0020] Preferably, based on the VIT-L backbone network, a mode bias score (MBS) is defined according to the ratio of mode gradient norm differences, and this index is used as a mode bias regularization term to weight the total loss; a cross-modal debiasing attention module is constructed, which attenuates intra-modal attention and enhances cross-modal attention through a bias correction matrix; MBS is defined as:
[0021] in, and Let these represent the sets of parameters related to the optical mode and the SAR mode, respectively, with corresponding gradient norms as follows: and Furthermore, to prevent the denominator from being zero, a small constant is introduced. .
[0022] Preferably, the optical prediction change map and the SAR prediction change map are respectively The consistency constraint loss is composed of the JS divergence term and the CD divergence term, and its formalization is as follows:
[0023] in, and These are the height and width of the input image, respectively. and Representing optical and SAR modes respectively. represents the prediction category; M represents the average distribution of the optical prediction change map and the SAR prediction change map; u and v are spatial location indices, where u represents the pixel index of the image in the horizontal direction and v represents the pixel index of the image in the vertical direction.
[0024] Preferably, the total loss function is used during model training. As an optimization objective, the AdamW optimizer was used for training, and the LoRA branch weight update method of the ViT-L backbone was used to freeze some parameters of the backbone and train only the low-rank adapter parameters. Total loss function for:
[0025]
[0026]
[0027] in It is a joint form of binary cross-entropy (BCE) loss and Dice loss; The defined consistency constraint loss; MBS is the defined modal deviation fraction; , This is a hyperparameter used to control the regularization term.
[0028] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention proposes a method and system for monitoring farmland surface disturbance and desertification based on multimodal remote sensing. The aim is to automatically identify farmland surface disturbance and desertification evolution areas in multimodal remote sensing images from different time phases. The LoRA (Local Area Reduction) fine-tuning mechanism significantly improves the fusion balance of optical and SAR features, allowing weak modal information to be fully expressed. Simultaneously, cross-modal consistency constraints are introduced to ensure the consistency and reliability of detection results for changes in farmland or desertification areas, effectively reducing false positives and false negatives caused by environmental noise. This provides reliable remote sensing information support for farmland protection, identification of illegal sand mining disturbances, and monitoring of desertification prevention and control.
[0029] 2. This invention employs an efficient parameter fine-tuning strategy, requiring only a small number of low-rank parameters to achieve rapid model transfer learning, while maintaining stable and accurate change detection performance even in the case of modality loss. This invention can achieve high-precision and robust remote sensing change monitoring under complex imaging conditions, providing strong support for fields such as land resource supervision, disaster monitoring, and urban expansion analysis.
[0030] 3. This invention uses optical remote sensing images and SAR images acquired at different observation times in the same area as input. Through multimodal data preprocessing, patch partitioning and token construction based on the ViT-L structure, extraction of shared Transformer backbone features, modal debiasing LoRA fine-tuning, cross-modal debiasing attention fusion, and consistency constraints and multi-head prediction at the output end, it achieves high-precision identification of areas of change on the land surface. Attached Figure Description
[0031] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of the main process of this invention.
[0032] Figure 2 This is a schematic diagram illustrating the construction and input of the dataset for this invention.
[0033] Figure 3 This is a structural diagram of the model proposed in this invention.
[0034] Figure 4 This is a schematic diagram of the LoRA structure for modal debiasing. Detailed Implementation
[0035] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0036] like Figure 1 As shown, a method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing data includes: Step S1: Acquire remote sensing images and preprocess them to obtain multimodal dual-temporal remote sensing images; The remote sensing images include optical remote sensing images and SAR remote sensing images; Step S2: Divide the multimodal dual-temporal remote sensing image and construct a token sequence containing modality coding and temporal coding. Input the sequence into a Transformer backbone network with shared weights for feature extraction to obtain optical modality features and SAR modality features. Step S3: Introduce a mode-independent low-rank adapter into the attention module of the Transformer backbone network, and adjust the gradient contribution based on the mode bias score to perform mode debiasing on optical mode features and SAR mode features; Step S4: Construct time-series phase difference variation features for the depolarized optical mode features and SAR mode features respectively, and fuse them to obtain fused features; Step S5: Generate a change probability map based on the fused features and apply consistency constraints between optical and SAR prediction results, then output the final change detection results.
[0037] Specifically, in one embodiment, the above steps include: Acquire and preprocess two-phase optical and SAR images; divide the multimodal images into patches and construct a token sequence containing modality coding and temporal coding; input the sequence into a Transformer backbone network with shared weights to extract optical and SAR features; introduce a modality-independent low-rank adapter (LoRA) in the attention module and adjust the gradient contribution based on the modality bias score to suppress modality bias; construct cross-temporal differential change features and perform feature fusion by combining a cross-modal debiasing attention mechanism; generate a change probability map based on the fused features and apply consistency constraints between optical and SAR prediction results to obtain the final change detection result.
[0038] The shared-weight Transformer feature extraction backbone network is based on the ViT-L structure, including an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The model input consists of multimodal, dual-temporal samples. and change label image Multimodal and multitemporal remote sensing images are mapped to an embedding space through patch partitioning and token construction. Features are extracted through a multi-layer network and then fused and decoded to obtain change detection and prediction results.
[0039] Based on the VIT-L backbone network, a mode bias score (MBS) is defined according to the ratio of mode gradient norm differences. This index is used as a mode bias regularization term to weight the total loss, achieving a balance between optical and SAR contributions. A cross-modal debiasing attention module is constructed, which attenuates intra-modal attention and enhances cross-modal attention through a bias correction matrix, giving weaker modes higher weight in feature fusion. MBS is defined as:
[0040] in, and Let these represent the sets of parameters related to the optical mode and the SAR mode, respectively, with corresponding gradient norms as follows: and Furthermore, to prevent the denominator from being zero, a small constant is introduced. .
[0041] The optical prediction change map and the SAR prediction change map are respectively... The consistency constraint loss is constructed using the Jensen-Shannon (JS) divergence term and the contrastive (CD) divergence term, and is formalized as follows:
[0042] in, and These are the height and width of the input image, respectively. and Representing optical and SAR modes respectively. The prediction category is M. M represents the average distribution of the optical prediction change map and the SAR prediction change map; u and v are spatial location indices, where u represents the pixel index of the image in the horizontal direction and v represents the pixel index of the image in the vertical direction.
[0043] Using the total loss function As the optimization objective, the AdamW optimizer is used for training, and the LoRA branch weight update method of the ViT-L backbone is utilized to freeze most of the backbone parameters, training only the low-rank adapter parameters. This significantly reduces training costs and enables the model to be efficiently transferred to different scenarios and multiple data sources. Total loss function for:
[0044]
[0045]
[0046] in It is a joint form of binary cross-entropy (BCE) loss and Dice loss; The defined consistency constraint loss; MBS is the defined modal deviation fraction; , This is a hyperparameter used to control the regularization term.
[0047] In summary, this invention constructs an optical-SAR joint feature learning framework and introduces a modality-independent low-rank LoRA adapter into the attention mechanism of the Transformer backbone. It dynamically adjusts the gradient contribution of each modality through modality bias scores to enhance weak modality feature representation. Simultaneously, it utilizes a cross-temporal phase difference strategy to construct an enhanced change representation and further introduces consistency constraints between optical and SAR change predictions at the output. Through joint optimization of the JS divergence term and the contrastive divergence term, it ensures that the prediction distributions of the two modalities tend to be consistent, thereby obtaining stable and accurate change detection results in both multimodal fusion and modality-missing scenarios.
[0048] Example 1 This invention provides a multimodal remote sensing change detection method based on modal debiasing fine-tuning mechanism and consistency constraints. The specific implementation process and steps of this method are as follows. This invention uses optical remote sensing images and SAR images of the same area acquired at different observation times as input. Through multimodal data preprocessing, patch partitioning and token construction based on the ViT-L structure, shared Transformer backbone feature extraction, modal debiasing LoRA fine-tuning, cross-modal debiasing attention fusion, and consistency constraints and multi-head prediction at the output end, it achieves high-precision identification of surface change areas.
[0049] Step 1: Multimodal Data Acquisition and Preprocessing like Figure 2 As shown, in the implementation of this invention, it is first necessary to acquire optical remote sensing images and SAR remote sensing images covering the same geographical area with time intervals to construct a multimodal dual-temporal dataset for change detection. Alternatively, publicly released optical-SAR registration datasets can be directly used, such as typical open-source data resources that have completed geometric correction and multimodal registration processing, thereby ensuring a good correspondence between different modal images and reducing the interference of registration errors on model training results.
[0050] Specifically, the study area was selected at the first moment. With the second moment The optical satellite images are denoted as follows: and
[0051] in, and They are the number of rows and columns in the space, This represents the number of spectral bands in the optical image.
[0052] Simultaneously, acquire SAR remote sensing images that have the same spatial coverage and are as close as possible in time to the aforementioned optical images, denoted as follows: and
[0053] in, and The number of rows and columns in the SAR image. This represents the number of SAR polarization channels.
[0054] To ensure the comparability of optical and SAR images in terms of radiometric and geometric significance, high-quality multimodal remote sensing data were selected for preprocessing operations such as radiometric calibration, atmospheric correction, and geometric fine correction.
[0055] To convert continuous, large-scale remote sensing imagery into small sample blocks suitable for deep learning model training, the registered multimodal imagery is cropped and sample constructed by dividing it into blocks according to a fixed window size (e.g., 256×256 pixels). For each cropped region, corresponding optical first and second temporal phase images and SAR first and second temporal phase images are obtained, and these are combined to form a multimodal dual-temporal sample.
[0056] In supervised change detection tasks, a corresponding change label map is constructed for each sample block. Change label image Defined as a two-dimensional grid with dimensions consistent with the sample block space. Each pixel element The values are 0 or 1, where 0 indicates no change and 1 indicates a change. i and j represent the row and column coordinates of the labeled image pixels, respectively. Through the above steps, a multimodal remote sensing change detection dataset is constructed:
[0057] Step 2: Patch Partitioning and Model Input Token Construction To adapt to the Transformer architecture's input requirements for sequential data, multimodal dual-temporal remote sensing images are divided into multiple fixed-size patches. Each patch undergoes linear projection and embedding encoding, transforming it into a token sequence that can be input into the ViT-L model. This operation enables preliminary alignment of optical and SAR modal features in a unified feature space while preserving local spatial structure information.
[0058] Use a size of [size] for each image. The sliding window is divided into sections, where To preset the patch size, in this model Set to 16. For input size... The image can be divided into: The first patch, then flatten it, the second... The flattened vector representation of each patch is as follows: , Indicates optical or SAR mode, Indicates phase and .
[0059] To map features to a unified Transformer embedding space, each patch vector is transformed into... 3D embedding vector , The input feature dimension is provided to the Transformer. Furthermore, to maintain the distinction between positional and modal attributes in the Transformer model, spatial location encoding is added to the embedded features of each patch. Modal coding Phase coding Finally, the initial token is constructed:
[0060] Step 3: Construction of a multimodal remote sensing change detection model based on ViT-L like Figure 3 As shown, this invention employs a multimodal remote sensing change detection model with the ViT-Large backbone, achieving unified representation learning of features from both optical and SAR remote sensing images through a shared coding structure. The ViT-L backbone comprises 24 stacked Transformer encoder layers, each including a multi-head self-attention module and a feedforward neural network module. This represents the dimension of the word vectors in the model. Self-attention operations dynamically adjust the weights of the input data by calculating the relationships between queries, keys, and values. Specifically, the calculation method is as follows:
[0061] in, It represents the dimensions of the key and query vectors, used to scale the dot product attention mechanism to maintain numerical stability.
[0062] To achieve fine-grained adaptation between optical and SAR modes while keeping the ViT-L backbone parameters essentially frozen, a mode-independent low-rank adaptation branch is introduced into the projection matrix of the attention module. Taking the query matrix as an example, for the mode... definition:
[0063] in, For the shared projection matrix of the pre-trained model backbone, , For modality The LoRA low-rank parameter, ; This is the modal debiasing scaling factor, used to control the degree of influence of LoRA branches.
[0064] To measure the imbalance between the gradient contributions of optical and SAR modes during training, this invention defines the Modality Bias Score (MBS) as follows. Let... and Let these represent the sets of parameters related to the optical mode and the SAR mode, respectively, with corresponding gradient norms as follows: and Furthermore, to prevent the denominator from being zero, a small constant is introduced. have to
[0065] based on Constructing Modal Deviation Regularization Terms , The weighting coefficient is used to control the influence of bias constraints on the total loss, thereby achieving modal debiasing learning.
[0066] Step 4: Construction of Temporal Variation Features and Cross-Modal Bias-Free Attention Fusion After adapting the ViT-L backbone to LoRA, high-level feature representations of the optical and SAR modes in two time phases can be obtained: ,in =256 is the number of patches. = 1024 is the feature dimension.
[0067] By calculating the differences in characteristics of the same modality at different times, a feature of change is constructed:
[0068] To preserve the semantics and trend of change before and after the change, the above features are concatenated to obtain a temporal augmentation representation:
[0069] The optical and SAR dual-modal enhancement features are stacked along the token dimension to form a cross-modal fusion sequence. And Dimensionality is reduced to the unified latent space dimension of Transformer.
[0070] A self-attention mechanism is applied to the dimensionality-reduced token sequence. To suppress excessive concentration of attention within the dominant mode, a bias correction matrix is constructed based on the above optical and SAR methods, where all tokens participate in the attention calculation together. . The intramodal attention component is positive and iteratively learnable, while the intermodal attention component is assigned zero, and a debiasing adjustment factor is introduced. To regulate the intensity of bias reduction. The attention weight after bias reduction is... .
[0071] The final formula for calculating the fused attention output is as follows:
[0072] The upsampling network restores the features to be similar to the original. Figure 1 Feature maps with high resolution And prediction heads for optical, SAR, and fused perspectives are constructed respectively, as follows: , For the corresponding convolution kernel, For the corresponding bias term, These represent the probability distributions of changes in the optical viewpoint, SAR viewpoint, and fusion result, respectively.
[0073] Step 5: Model Training and Evaluation This invention constructs a total loss function by integrating task loss, consistency constraint loss, and modal deviation regularization term, and performs efficient fine-tuning training of non-frozen parameters.
[0074] The loss function of the change detection task utilizes the change probability map obtained from the fusion feature decoding. Compared with the actual change annotation map The difference between them is used to construct the change discrimination loss. A joint form of binary cross-entropy (BCE) loss and Dice loss is adopted: .
[0075] in, ,
[0076] To ensure consistency between optical and SAR change prediction results, a consistency constraint loss is proposed, where the optical and SAR predicted change maps are respectively... The constraints consist of Jensen-Shannon (JS) divergence terms and contrastive divergence (CD) terms, resulting in the consistency constraint loss as follows: Define the average distribution of the two. ( Then there is
[0077]
[0078] Taking into account the above loss terms, the final optimization objective function is defined as:
[0079] Training employs the AdamW optimizer and utilizes the LoRA branch weight update method of the ViT-L backbone, freezing most backbone parameters and training only the low-rank adapter parameters. This strategy significantly reduces training costs, enabling the model to be efficiently transferred to different scenarios and multiple data sources.
[0080] The model performance was comprehensively evaluated using metrics such as F1-score, IoU, Precision, and Recall. Modal robustness evaluation was also introduced to verify that the model still has stable detection capabilities under degradation perception conditions, even with the absence of a single modality.
[0081] Step Six: Remote Sensing Image Change Detection During the inference phase, two temporal optical images and SAR images are input. Through the aforementioned multimodal feature extraction, cross-temporal differential enhancement, and cross-modal debiasing attention fusion processes, a fused spatial semantic change feature map is obtained. Subsequently, the decoding module restores the image to its original resolution, generating a fused viewpoint change probability map, denoted as... .
[0082] To obtain the final change detection result, the fused probability map is converted into a binary change mask using Otsu's automatic thresholding method. When the predicted probability is greater than or equal to the threshold, Take 1 if the value is less than 1, and 0 if the value is less than 1.
[0083] In real-world remote sensing systems, there are limitations in observation conditions, and only one mode may be available. The cross-modal debiasing mechanism and end-to-end consistency constraint learning proposed in this invention enable the model to have the ability to reason about missing modes. When one mode of image is missing, only the dual-temporal features of the other mode are input, and the missing mode is input in the form of a mask.
[0084] To further suppress isolated noise and scattered mislabeling, a post-optimization strategy using maximum connectivity screening and Conditional Random Field (CRF) is employed. This achieves continuity of the boundaries of the changed area and consistency of the terrain features.
[0085] The present invention also provides a farmland surface disturbance and desertification monitoring system based on multimodal remote sensing. The farmland surface disturbance and desertification monitoring system based on multimodal remote sensing can be implemented by executing the process steps of the farmland surface disturbance and desertification monitoring method based on multimodal remote sensing. That is, those skilled in the art can understand the farmland surface disturbance and desertification monitoring method based on multimodal remote sensing as a preferred embodiment of the farmland surface disturbance and desertification monitoring system based on multimodal remote sensing.
[0086] Specifically, a multimodal remote sensing-based monitoring system for farmland surface disturbance and desertification includes: Module M1: Acquires and preprocesses remote sensing images to obtain multimodal dual-temporal remote sensing images; The remote sensing images include optical remote sensing images and SAR remote sensing images; Module M2: Divide the multimodal dual-temporal remote sensing image and construct a token sequence containing modality coding and temporal coding. Input the sequence into a Transformer backbone network with shared weights for feature extraction to obtain optical modality features and SAR modality features. Module M3: Introduces a mode-independent low-rank adapter into the attention module of the Transformer backbone network, and adjusts the gradient contribution based on the mode bias score to perform mode debiasing on optical mode features and SAR mode features; Module M4: Constructs and fuses time-phase differential variation features of the depolarized optical modal features and SAR modal features respectively to obtain fused features; Module M5: Generates a change probability map based on fused features and applies consistency constraints between optical and SAR prediction results, outputting the final change detection results.
[0087] The Transformer backbone network includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer; the model input is multimodal dual-temporal samples. and change label image ; Where t1 and t2 represent the first and second time points, respectively; and Representing optical and SAR modes respectively; The optical modal input images representing times t1 and t2, SAR modal input images representing times t1 and t2; Multimodal dual-temporal remote sensing images are mapped into an embedding space through patch partitioning and token construction. Features are extracted and fused through a multi-layer network to obtain the final change detection result.
[0088] Based on the VIT-L backbone network, a mode bias score (MBS) is defined according to the modal gradient norm difference ratio, and this index is used as a mode bias regularization term to weight the total loss. A cross-modal debiasing attention module is constructed, which attenuates intra-modal attention and enhances cross-modal attention through a bias correction matrix. MBS is defined as:
[0089] in, and Let these represent the sets of parameters related to the optical mode and the SAR mode, respectively, with corresponding gradient norms as follows: and Furthermore, to prevent the denominator from being zero, a small constant is introduced. .
[0090] The optical prediction change map and the SAR prediction change map are respectively The consistency constraint loss is composed of the JS divergence term and the CD divergence term, and its formalization is as follows:
[0091] in, and These are the height and width of the input image, respectively. and Representing optical and SAR modes respectively. The prediction category is M. M represents the average distribution of the optical prediction change map and the SAR prediction change map; u and v are spatial location indices, where u represents the pixel index of the image in the horizontal direction and v represents the pixel index of the image in the vertical direction.
[0092] During model training, the total loss function is used. As an optimization objective, the AdamW optimizer was used for training, and the LoRA branch weight update method of the ViT-L backbone was used to freeze some parameters of the backbone and train only the low-rank adapter parameters. Total loss function for:
[0093]
[0094]
[0095] in It is a joint form of binary cross-entropy (BCE) loss and Dice loss; The defined consistency constraint loss; MBS is the defined modal deviation fraction; , This is a hyperparameter used to control the regularization term.
[0096] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0097] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing, characterized in that, include: Step S1: Acquire remote sensing images and preprocess them to obtain multimodal dual-temporal remote sensing images; The remote sensing images include optical remote sensing images and SAR remote sensing images; Step S2: Divide the multimodal dual-temporal remote sensing image and construct a token sequence containing modality coding and temporal coding. Input the sequence into a Transformer backbone network with shared weights for feature extraction to obtain optical modality features and SAR modality features. Step S3: Introduce a mode-independent low-rank adapter into the attention module of the Transformer backbone network, and adjust the gradient contribution based on the mode bias score to perform mode debiasing on optical mode features and SAR mode features; Step S4: Construct time-series phase difference variation features for the depolarized optical mode features and SAR mode features respectively, and fuse them to obtain fused features; Step S5: Generate a change probability map based on the fused features and apply consistency constraints between optical and SAR prediction results, then output the final change detection results.
2. The method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing according to claim 1, characterized in that, The Transformer backbone network includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer; the model input is multimodal dual-temporal samples. and change label image ; Where t1 and t2 represent the first and second time points, respectively; and Representing optical and SAR modes respectively; The optical modal input images representing times t1 and t2, SAR modal input images representing times t1 and t2; Multimodal dual-temporal remote sensing images are mapped into an embedding space through patch partitioning and token construction. Features are extracted and fused through a multi-layer network to obtain the final change detection result.
3. The method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing according to claim 1, characterized in that, Based on the VIT-L backbone network, a mode bias score (MBS) is defined according to the mode gradient norm difference ratio, and this index is used as a mode bias regularization term to weight the total loss; a cross-modal debiasing attention module is constructed, which attenuates intramodal attention and enhances cross-modal attention through a bias correction matrix; MBS is defined as: in, and Let these represent the sets of parameters related to the optical mode and the SAR mode, respectively, with corresponding gradient norms as follows: and Furthermore, to prevent the denominator from being zero, a small constant is introduced. .
4. The method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing according to claim 1, characterized in that, The optical prediction change map and the SAR prediction change map are respectively The consistency constraint loss is composed of the JS divergence term and the CD divergence term, and its formalization is as follows: in, and These are the height and width of the input image, respectively. and Representing optical and SAR modes respectively. represents the prediction category; M represents the average distribution of the optical prediction change map and the SAR prediction change map; u and v are spatial location indices, where u represents the pixel index of the image in the horizontal direction and v represents the pixel index of the image in the vertical direction.
5. The method for monitoring farmland surface disturbance and desertification based on multimodal remote sensing according to claim 1, characterized in that, During model training, the total loss function is used. As an optimization objective, the AdamW optimizer was used for training, and the LoRA branch weight update method of the ViT-L backbone was used to freeze some parameters of the backbone and train only the low-rank adapter parameters. Total loss function for: in The loss for the transformation detection task is a joint form of binary cross-entropy loss and Dice loss; Loss due to consistency constraints; It is the modal deviation canonical loss. MBS stands for Modal Deviation Fraction; This is a probability map of change prediction, where Y is the actual change label map; JS and CD are the Jensen-Shannon divergence and contrastive divergence, respectively. , This is a hyperparameter used to control the regularization term.
6. A monitoring system for farmland surface disturbance and desertification based on multimodal remote sensing, characterized in that, include: Module M1: Acquires and preprocesses remote sensing images to obtain multimodal dual-temporal remote sensing images; The remote sensing images include optical remote sensing images and SAR remote sensing images; Module M2: Divide the multimodal dual-temporal remote sensing image and construct a token sequence containing modality coding and temporal coding. Input the sequence into a Transformer backbone network with shared weights for feature extraction to obtain optical modality features and SAR modality features. Module M3: Introduces a mode-independent low-rank adapter into the attention module of the Transformer backbone network, and adjusts the gradient contribution based on the mode bias score to perform mode debiasing on optical mode features and SAR mode features; Module M4: Constructs and fuses time-phase differential variation features of the depolarized optical modal features and SAR modal features respectively to obtain fused features; Module M5: Generates a change probability map based on fused features and applies consistency constraints between optical and SAR prediction results, outputting the final change detection results.
7. The farmland surface disturbance and desertification monitoring system based on multimodal remote sensing according to claim 6, characterized in that, The Transformer backbone network includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer; the model input is multimodal dual-temporal samples. and change label image ; Where t1 and t2 represent the first and second time points, respectively; and Representing optical and SAR modes respectively; The optical modal input images representing times t1 and t2, SAR modal input images representing times t1 and t2; Multimodal dual-temporal remote sensing images are mapped into an embedding space through patch partitioning and token construction. Features are extracted and fused through a multi-layer network to obtain the final change detection result.
8. The farmland surface disturbance and desertification monitoring system based on multimodal remote sensing according to claim 6, characterized in that, Based on the VIT-L backbone network, a mode bias score (MBS) is defined according to the mode gradient norm difference ratio, and this index is used as a mode bias regularization term to weight the total loss; a cross-modal debiasing attention module is constructed, which attenuates intramodal attention and enhances cross-modal attention through a bias correction matrix; MBS is defined as: in, and Let these represent the sets of parameters related to the optical mode and the SAR mode, respectively, with corresponding gradient norms as follows: and Furthermore, to prevent the denominator from being zero, a small constant is introduced. .
9. The farmland surface disturbance and desertification monitoring system based on multimodal remote sensing according to claim 6, characterized in that, The optical prediction change map and the SAR prediction change map are respectively The consistency constraint loss is composed of the JS divergence term and the CD divergence term, and its formalization is as follows: in, and These are the height and width of the input image, respectively. and Representing optical and SAR modes respectively. represents the prediction category; M represents the average distribution of the optical prediction change map and the SAR prediction change map; u and v are spatial location indices, where u represents the pixel index of the image in the horizontal direction and v represents the pixel index of the image in the vertical direction.
10. The farmland surface disturbance and desertification monitoring system based on multimodal remote sensing according to claim 6, characterized in that, During model training, the total loss function is used. As an optimization objective, the AdamW optimizer was used for training, and the LoRA branch weight update method of the ViT-L backbone was used to freeze some parameters of the backbone and train only the low-rank adapter parameters. Total loss function for: in The loss for the transformation detection task is a joint form of binary cross-entropy loss and Dice loss; Loss due to consistency constraints; It is the modal deviation canonical loss. MBS stands for Modal Deviation Fraction; This is a probability map of change prediction, where Y is the actual change label map; JS and CD are the Jensen-Shannon divergence and contrastive divergence, respectively; where... , This is a hyperparameter used to control the regularization term.