An end-to-end single-phase remote sensing image anomaly detection method based on mask reconstruction

Abstract

The application belongs to the technical field of remote sensing image intelligent analysis, and specifically discloses an end-to-end single-time-phase remote sensing image anomaly detection method based on mask reconstruction, which comprises the following steps: acquiring and pre-processing normal ground object sample images, and constructing a training set; using the training set, performing parameter efficient fine-tuning on a pre-trained mask reconstruction visual basic model to obtain a fine-tuning model special for normal ground object reconstruction; inputting single-time-phase remote sensing images to be detected into the fine-tuning model, generating an abnormal heat map through multiple mask reconstructions and difference aggregations, and segmenting an abnormal area from the abnormal heat map; further comprising the steps of: based on the segmented abnormal area, acquiring multi-source features and constructing prompt words, and using a multi-modal large model to generate semantic explanations containing abnormal type inference. The end-to-end and automatic processing of single-time-phase remote sensing images from pixel-level detection to semantic-level explanation is realized, and the method has the advantages of low data dependence, high robustness and high generalization.

Description

Technical Field

[0001] This invention belongs to the field of intelligent analysis technology of remote sensing images, specifically relating to an end-to-end single-temporal remote sensing image anomaly detection method based on mask reconstruction. Background Technology

[0002] Rapid identification of surface anomalies (such as landslides, collapses, building collapses, ground fissures, debris flows, typhoon damage, forest fires, urban fires, and earthquake damage) is a crucial aspect of intelligent interpretation of remote sensing imagery and emergency monitoring. Remote sensing technology extracts information by utilizing the reflection characteristics of ground objects to electromagnetic waves of different frequencies through imaging techniques. It is characterized by minimal impact from terrain, wide coverage, and low acquisition costs. Remote sensing imagery, as a significant product of remote sensing technology, is a vital means of rapidly locating surface anomalies at a macroscopic scale and achieving a comprehensive understanding of the situation. Among these technologies, multispectral and hyperspectral imaging remote sensing, capable of capturing the morphology and spectral characteristics of ground objects at high spectral resolution, is a key technology for acquiring ground feature information.

[0003] In the field of remote sensing image anomaly detection, existing research and applications mainly employ three technical approaches. The first is change-based detection methods. These methods require acquiring paired images before and after the anomaly event, identifying anomalous regions through difference analysis. Common techniques include interpolation methods, transformation methods, and deep learning-based bi-temporal change detection. However, in emergency response scenarios, it is often difficult to acquire high-quality pre-disaster images that are temporally consistent with post-disaster images, have matching spatial resolution, and are free from cloud cover. This multi-temporal dependence significantly impacts the practicality of these methods. Furthermore, registration errors and differences in illumination conditions between images from different sources can easily lead to misjudgments. The second approach is mechanistic analysis methods based on prior knowledge. These methods often utilize remote sensing parameters such as the NDVI (ND vegetation index), NBR (Non-Burning index), spectral angle, or spectral curvature as criteria. However, these methods are highly dependent on specific spectral changes triggered by the anomaly event (such as vegetation absorption peaks or fire spectral characteristics). For surface disturbances with indistinct spectral differences (such as building collapses, landslides, and urban road damage), these methods often fail to effectively identify them and are easily affected by land cover type, imaging conditions, and regional differences, making it difficult for the model to maintain stable performance across different regions. The third category is supervised learning-based methods. These methods rely on a large number of anomalous samples and auxiliary annotation files, using deep neural networks or machine learning methods to learn the texture or spatial patterns of anomalous surfaces for disaster classification or anomalous area segmentation. Due to the suddenness and scarcity of anomalous events, large-scale, high-quality anomalous data is difficult to obtain; and the significant differences between different disaster types lead to highly specialized tasks, making it difficult to maintain good generalization ability across regions and disaster types. Furthermore, the high cost of constructing high-quality training sets limits their widespread application in emergency scenarios.

[0004] In the fields of industrial and medical vision, a class of reconstruction-based unsupervised anomaly detection methods has emerged in recent years. Typical examples include autoencoders (AE), variational autoencoders (VAE), reconstruction networks based on GANs (Generative Adversarial Networks), and reconstruction-based methods using masked autoencoders (MAE). These methods are typically trained only on "normal samples," allowing the model to learn patterns of normal product appearance or texture, thus utilizing reconstruction errors to locate anomalous regions during inference. While these methods have achieved good performance in industrial inspection, their applications are mostly limited to close-up, fixed-lighting, single-channel or RGB (three-channel image) images, where the images typically have low noise, low geometric distortion, and controllable background changes. The models are often trained from scratch on specific datasets.

[0005] However, directly applying the aforementioned industrial reconstruction-based anomaly detection methods to remote sensing imagery has significant limitations. Firstly, remote sensing images are mostly multispectral or hyperspectral data, with numerous bands and complex spectral correlations. Existing industrial methods are largely based on RGB band design, failing to effectively utilize the rich band information in multispectral / hyperspectral images. Secondly, remote sensing images are affected by factors such as imaging conditions, solar elevation angle, atmospheric influence, and mixed ground feature pixels, resulting in issues like "different spectra for the same object" or "different objects for the same spectrum." Their spatial scale and texture complexity are far greater than industrial images, making reconstruction errors susceptible to interference from non-anomaly factors. This makes reconstruction methods relying on single textures or simple spectral patterns unable to effectively distinguish normal ground features from real anomalies.

[0006] Existing remote sensing anomaly detection methods based on reconstruction are limited, typically relying on RGB or simple spectral reconstruction. Their reconstruction loss functions are often simplistic and fail to fully utilize multispectral or hyperspectral features. Furthermore, these methods suffer from several problems: first, the models are often low-parameter models trained from scratch, resulting in limited spatial generalization; second, they lack fine-tuning strategies and difference enhancement strategies for high-parameter remote sensing pre-trained models. More importantly, existing automated methods often stop at outputting "anomaly area masks" or "difference maps," lacking the ability to automatically interpret anomalies by integrating spectral information, spatial location, and surface type. They also lack an end-to-end approach from image input to semantic report generation for anomaly events. Therefore, in practical applications, disaster emergency response departments still require significant manual effort in combining spectral features for analysis, background information assessment, event type inference, area estimation, and report writing. This results in a time-consuming and low-automation process, failing to meet the rapid response needs after a disaster.

[0007] In summary, existing technologies generally suffer from the following shortcomings: (1) Multi-temporal image dependence: Traditional change detection methods rely on high-quality pre-disaster and post-disaster paired images, which are difficult to meet the effectiveness requirements of emergency response; (2) Dependence on specific spectral priors or a large amount of labeled data: Spectral feature-based methods have limited ability to identify abnormal event types and are insufficient in addressing regional differences, while supervised deep learning methods often require a large amount of high-quality labeled data; (3) Insufficient model generalization ability: Existing research usually adopts a validation strategy that does not rely on general pre-trained models but trains from scratch on its own dataset. Low-parameter models are difficult to handle complex ground object reconstruction tasks, while training high-parameter models from scratch is costly, which limits the development of related technologies. Moreover, the strategy of training from scratch has an inherent limitation in model generalization compared to the basic model learned from massive amounts of data. (4) Insufficient utilization of multispectral information and feature space: Single-temporal unsupervised reconstruction methods for remote sensing images are not adaptable enough. Most existing methods are directly transferred from the industrial field, focusing on the texture features of the image itself, and failing to make full use of the spectral dimension advantages of multi / hyperspectral data and the differences between normal and abnormal ground objects at the feature space level; (5) Lack of end-to-end methods for automated interpretation of multi-source data: The automation process of existing methods often stops at anomaly detection. It lacks the ability to use multi-source data and the general knowledge of existing multimodal large models and Internet information to assist in retrieval, and cannot automatically generate key auxiliary information such as event type and area, resulting in a significant gap between detection and interpretation.

[0008] Therefore, it is necessary to provide an end-to-end remote sensing image analysis method that can achieve automatic anomaly detection under single-temporal image conditions and further generate semantic interpretations of events, so as to improve the automation, robustness and practicality of emergency monitoring. Summary of the Invention

[0009] To achieve the above objectives, the present invention provides an end-to-end single-temporal remote sensing image anomaly detection method based on mask reconstruction, comprising the following steps: Acquire and preprocess normal ground cover sample images to construct a training set; Using the training set, the parameters of the pre-trained mask reconstruction visual base model are efficiently fine-tuned to obtain a fine-tuned model specifically for normal ground feature reconstruction. The single-temporal remote sensing image to be detected is input into the fine-tuning model. Anomaly heatmaps are generated through multiple mask reconstructions and difference aggregations, and anomalous regions are segmented from them.

[0010] Optionally, the steps also include: Based on the segmented abnormal regions, multi-source features are obtained and prompt words are constructed. A semantic interpretation containing anomaly type inference is generated using a multimodal large model.

[0011] Optionally, the efficient fine-tuning of parameters in the pre-trained mask reconstruction visual baseline model specifically includes: A two-stage, gradual strategy, including a freezing phase and a thawing phase, is adopted for fine-tuning; During the fine-tuning process, a joint loss function is used for constraints, which is used to simultaneously optimize the spectral reconstruction fidelity, spatial structure reconstruction quality, and the clustering of normal ground feature representations of the model.

[0012] Optionally, the joint loss function includes a feature dynamic center loss, which is used to cause the normal ground features extracted by the encoder to cluster towards dynamically updated cluster centers in the feature space.

[0013] Optionally, the two-stage progressive strategy includes: In the first stage, part of the underlying network structure of the model is frozen and fine-tuned using a first loss function, which includes at least pixel-level reconstruction loss and feature dynamic center loss. In the second stage, all or most of the network structure of the model is unfrozen and fine-tuned using a second loss function, which includes at least pixel-level reconstruction loss and multi-scale spatial structure similarity loss.

[0014] Optionally, generating an anomaly heatmap through multiple mask reconstructions and difference aggregation specifically includes: Perform N independent random masking operations on the image to be detected, and obtain N reconstruction results through the fine-tuning model; For each reconstruction, at least two measures of difference between the original image and the reconstructed image are calculated. After standardizing and spatially filtering each difference measure, pixel-by-pixel multiplication and spectral aggregation of all bands are performed to obtain a single-channel enhanced difference map. The anomaly heatmap is generated by averaging the single-channel enhancement difference maps from all N reconstructions pixel by pixel.

[0015] Optionally, the at least two difference measures include the squared error that measures the difference in pixel values, and the structural dissimilarity that measures the similarity of local structures.

[0016] Optionally, abnormal regions are segmented from the abnormal heatmap, specifically by using an adaptive thresholding method based on image statistics to determine the segmentation threshold.

[0017] Optionally, the step of acquiring multi-source features and constructing prompt words specifically includes: The segmented abnormal region binary mask is converted into a vector polygon; Calculate the geometric properties and geographic information of each vector polygon; Extract the spectral features and visualization of the original image region corresponding to each vector polygon; The geometric attributes, geographic information, spectral features, and visualization images are organized into structured multimodal cue words.

[0018] Optionally, the geometric attributes and geographic information include at least morphological features, spatial positioning features, and geographic environment features; the spectral features include multi-band reflectance curves.

[0019] Optionally, the morphological features include at least one of area, perimeter, major axis length, minor axis length, shape index, rectangularity, and boundary complexity; the spatial positioning features include at least one of geometric center coordinates, minimum bounding rectangle range, and azimuth distance from a preset geographic reference point; the geographic environment features include at least one of background surface cover type, administrative division, average elevation, and average slope; and the spectral features further include one or more spectral indices calculated based on the multi-band reflectance curve.

[0020] Optionally, the multimodal prompt words are input into a visual language model, which fuses and analyzes image and text information to automatically generate a natural language briefing that includes the type of abnormal event, the scope of its impact, and background analysis.

[0021] This invention proposes an interpretable reconstruction-based unsupervised anomaly detection method guided by dynamic feature center constraints. Compared with existing methods, it introduces an efficient progressive fine-tuning strategy, utilizing pre-training results to reduce training overhead and enhance generalization. Simultaneously, it introduces an automated interpretation method based on a multimodal large model. This achieves end-to-end automated processing of single-temporal remote sensing images from pixel-level detection to semantic-level interpretation, exhibiting advantages such as low data dependence, high robustness, and high generalization, making it suitable for fields such as emergency management and anomaly monitoring. Attached Figure Description

[0022] The specific details of this disclosure are described below with reference to the accompanying drawings, which will facilitate a more readily understanding of the above and other objects, features, and advantages of this disclosure. The drawings are merely for illustrating the principles of this disclosure. The dimensions and relative positions of the elements are not necessarily drawn to scale in the drawings.

[0023] Figure 1 : A schematic diagram of the process of this invention; Figure 2 : A schematic diagram of the mask and reconstruction result of the single-call mask reconstruction model of this invention; Figure 3The present invention presents the original image before reconstruction and the average error heatmap obtained from the original image after N reconstructions; Figure 4 : A schematic diagram of the main processing flow of the reconstruction difference calculation and multidimensional difference aggregation of input images in this invention; Figure 5 The present invention provides abnormal images and corresponding output images showing the separability of abnormal extraction. Detailed Implementation

[0024] Exemplary aspects of this disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features implementing this disclosure are described in the specification. However, it should be understood that many disclosure-specific decisions can be made in developing any such implementation of this disclosure to achieve the developer's specific goals, and these decisions may vary depending on the specific implementation of this disclosure.

[0025] It should also be noted that, in order to avoid obscuring the content of this disclosure with unnecessary details, only the core structures closely related to the scheme according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0026] It should be understood that this disclosure is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, features may be substituted or borrowed between different embodiments where feasible, and one or more features may be omitted in one embodiment.

[0027] like Figure 1 As shown, training and validation: Start → Image input: Multi-source dataset without surface anomalies → Automated acquisition and preprocessing of normal ground cover sample images. For the input multi-source data block without anomalies, perform automated screening, cropping and other preprocessing operations on the sample images → Two-stage efficient fine-tuning of the basic visual model based on mask reconstruction: A two-stage strategy is adopted to fine-tune the parameters of the basic visual model based on mask reconstruction. Feature Discrimination Performance Evaluation: Input data, image input: images containing surface anomalies and images without surface anomalies. Used to evaluate the feature discrimination ability of the fine-tuned model: If "discrimination performance is poor", return to the two-stage efficient fine-tuning of the basic visual model based on mask reconstruction, and re-optimize the parameters. If "discrimination performance is good", proceed to the next step → Fine-tuning model inference call: Call the fine-tuned model and perform inference on the image set containing anomalies → Reconstruction difference calculation and multi-dimensional difference aggregation: Calculate the difference between the reconstruction result and the original image, and aggregate the multi-dimensional differences → Adaptive threshold calculation and segmentation for the difference map: Based on the difference map, automatically calculate the segmentation threshold and complete the segmentation of the anomaly region → Extraction accuracy evaluation and iterative optimization: Evaluate the accuracy and segmentation effect of anomaly region extraction: If "accuracy is low and segmentation is poor", return to step 3 (model fine-tuning) for iterative optimization; if "accuracy is high and segmentation is good", determine it as the "optimal model input" and complete training and validation.

[0028] Application-based reasoning stage: Input data includes image input: image to be detected and attribute data input: land cover type → Surface anomaly event generalization reasoning detection, using the optimal model, perform surface anomaly event generalization reasoning detection on the image to be detected, multimodal prompt input → Integrate detection results with land cover type to construct multimodal prompt → Common sense semantic interpretation based on visual language model: call the visual language model, combine with multimodal prompt to complete the common sense semantic interpretation of anomaly events → End.

[0029] This invention proposes an end-to-end single-temporal remote sensing image anomaly detection method based on mask reconstruction. The overall implementation steps are divided into two core stages: training and validation, and application inference, comprising nine key operational steps, specifically including: Step 1: Acquire and preprocess normal ground feature sample images to construct a training set; Step 2: Using the training set, perform efficient parameter fine-tuning on the pre-trained mask reconstruction visual base model to obtain a fine-tuned model specifically for normal ground feature reconstruction. Step 3: Evaluation of the feature discrimination effect of the fine-tuned model; Step 4: Abnormal image input and fine-tuning followed by model inference; Step 5: Reconstructing difference calculations and multidimensional difference aggregation; Step Six: Automatic threshold calculation and segmentation of the difference map; Step 7: Estimation and iterative optimization of model extraction accuracy based on labeled data; Step 8: Input the single-temporal remote sensing image to be detected into the fine-tuning model, generate an anomaly heatmap through multiple mask reconstructions and difference aggregations, and segment out the anomaly regions from it; Step 9: Based on the segmented abnormal regions, obtain multi-source features and construct prompt words, and use a multimodal large model to generate a semantic interpretation that includes anomaly type inference.

[0030] like Figure 1 The illustrated process and steps outlined above aim to enable the detection of anomalous event locations and comprehensive judgment of anomalous types within single temporal remote sensing images in emergency management and intelligent remote sensing interpretation. This method first inputs images using a large-scale dataset of images unaffected by surface anomalies. After automated acquisition and preprocessing of normal ground feature sample images, a progressively efficient fine-tuning of a basic visual model based on mask reconstruction is performed. This involves iteratively learning and progressively fine-tuning spectral and multi-scale spatial features under feature space constraints on existing high-parameter mask reconstruction basic visual models (such as Prithvi_EO2_300MTL, a second-generation Earth observation (EO) basic model jointly developed by IBM, NASA, and the Jülich Supercomputing Center). Subsequently, through multiple random independent mask inferences and by aggregating multidimensional differences (including structural and numerical differences) between the reconstructed and original images, a difference heatmap is generated, thereby robustly identifying the geographical locations of anomalous events. Finally, based on the multimodal information carried by the detected abnormal regions (patches), and combined with the massive prior knowledge and multi-source extracted information contained in visual language models, such as ChatGPT (visual version GPT-4V) and LLaVA (open source lightweight multimodal visual language model), the type of abnormal event is automatically determined and a professional abnormal event briefing is automatically generated, realizing end-to-end connection from pixel-level detection to semantic-level interpretation.

[0031] This method utilizes a pre-trained high-parameter mask reconstruction visual foundation model as a base model, combined with a two-stage progressive parameter fine-tuning strategy. Under feature space constraints, it uses a loss function combining spectral and spatial texture to learn large-scale normal ground cover images, efficiently encoding the spectral and multi-scale spatial features of the region. In the anomaly detection stage, multiple random independent mask reconstructions are performed on the input single-temporal multispectral images. The results of pixel and structural similarity differences between the original and reconstructed images are standardized, filtered, enhanced by pixel-by-pixel multiplication, and aggregated by band-by-band differences to generate a robust average difference heatmap. An adaptive thresholding method is used to extract anomaly region masks. In the intelligent interpretation stage, the segmented anomaly regions are vectorized, and multi-source information such as patch area, center latitude and longitude, spectral curve, and land cover is extracted. This information is then input into the multimodal large model in the form of structured multimodal cue words, automatically generating a professional event briefing containing anomaly type inference, coverage area, and background information.

[0032] Step 1: Acquire and preprocess normal ground feature sample images to construct a training set; First, a suitable visual baseline model (e.g., Prithvi_EO2_300MTL) for multispectral / hyperspectral imagery is determined, and a high-quality remote sensing image dataset that does not contain surface anomalies is selected as the information source for normal ground cover samples. Based on the image capture time and the latitude and longitude of the four corners, multispectral / hyperspectral imagery compatible with the model input is automatically acquired. The specific processing strategy is as follows: 1. Target Region and Buffer Construction: The target region is constructed based on the four corner coordinates of the sample image, and a buffer is established by spatial dilation of three times the maximum length and width of the sample image. This ensures that the features of interest and their background are completely included, and that the retrieved image completely covers the target area. 2. Time-consistent image retrieval: Automatically retrieves images that spatially cover the buffer within a 3-day time window before and after the image capture date; 3. Image quality control and cloud pixel removal: Use cloud / cloud shadow quality masks to remove cloud pixels or set them to invalid values; 4. Standardized Image Patch Cropping: While maintaining the original image resolution, the image is cropped according to the input image size compatible with the visual base model, with a 20% overlap between adjacent windows. Edge regions or regions with insufficient area that cannot meet the input image size requirements of the visual base model are not included in the sample set. 5. Cloud cover statistics and filtering: Perform cloud cover statistics independently on the cropped image blocks, remove image blocks with cloud cover exceeding 10%, and download image blocks that meet the criteria. 6. Dataset partitioning: All qualified multispectral image patches are randomly divided into training and validation sets at a ratio of 80%:20%.

[0033] Step 2: Using the training set, perform efficient parameter fine-tuning on the pre-trained mask reconstruction visual base model to obtain a fine-tuned model specifically for normal ground feature reconstruction. A visual baseline model, obtained through self-supervised pre-training on massive datasets, is used as the base model. Combined with the training set from the normal ground cover imagery dataset constructed in step one, a segmented, progressive fine-tuning strategy involving freeze-thaw and low-rank adaptation (such as LoRA) is designed. This aims to enhance the model's ability to extract and reconstruct normal features, thereby improving the distinguishability between abnormal and non-abnormal regions in anomaly detection tasks. The segmented two-stage fine-tuning strategy is as follows: Phase 1 (Lower-level Structure Freezing and Initial Adaptation): Freeze the encoder's lower-level layer, part of the mid-level MLP, and the decoder's lower-level layer. Using efficient fine-tuning techniques, only the high-level semantic structure is trained. The optimization objective is to reduce the MSE (Mean Squared Error) between the reconstructed and original images, along with the joint loss consisting of the spectral cosine similarity index and the dynamic feature center loss (FDCL). The model is trained with a robust learning rate to ensure reconstruction quality while initially learning the spectral composition of normal ground features.

[0034] Phase Two (Full Network Unfreezing and Refined Modeling): Unfreeze all network layers and fine-tune them with a lower or equivalent learning rate. Use a multi-scale joint loss consisting of MSE, MSGMS and Dynamic Feature Center Loss (FDCL) to guide the model to learn more detailed spatial structural features while retaining basic reconstruction capabilities.

[0035] The training process employs a dynamic learning rate adjustment strategy of "warm-up - cosine annealing," with the first 20% of the total epochs (training rounds) designated as warm-up rounds. The joint loss on the validation set is evaluated every 5 epochs, and the model with the best performance loss is saved locally.

[0036] The training loss employs a combination of multiple loss functions. Based on the introduction of feature dynamic center loss, it cleverly combines pixel-level reconstruction error (MSE), spectral cosine similarity loss (SCS), and multi-scale gradient magnitude similarity (MSGMS) to enable the model to gradually learn the spectral structure and spatial texture features of normal ground objects and aggregate normal features in the feature space, which facilitates the subsequent differentiation from abnormal features.

[0037] Step 3: Evaluation of the feature discrimination effect of the fine-tuned model.

[0038] Multispectral remote sensing images containing typical anomalous events are selected as test samples. These samples must have high-quality ground truth masks for anomalous events and be mixed with an equal number of images of the same data specification where no anomalous events occur. These samples are then input into a finely tuned model for forward inference. Feature reduction techniques, such as t-SNE (t-distributed random neighborhood embedding) and UMAP (uniform manifold approximation and projection), are used to perform dimensionality reduction and visualization analysis on the high-dimensional feature vectors extracted from the last layer of the encoder. This is to examine the model's ability to separate normal and anomalous features in the representation space. The model's feature discrimination ability is evaluated by observing the distribution of normal samples (expected to be compactly clustered) and anomalous samples (expected to be far from normal clusters) in two-dimensional space. If normal and anomalous samples are significantly mixed, the training strategy parameters (such as learning rate and loss function weights) in step two are adjusted based on the analysis results, and step two is repeated to optimize the model training direction.

[0039] Step 4: Input of Anomaly Images and Inference Using the Fine-tuned Model. Load the fine-tuned base model, saved after Step 3 (preliminary verification), into the inference environment. Input the multispectral / hyperspectral image containing the anomaly event mask used in Step 3. For each input regular image, perform N random independent masking operations under a fixed masking ratio (ensuring the random seed is fixed but different in each operation). Store the binary map of the mask position and the corresponding reconstructed map generated from each masking operation in a separate directory using a standardized naming convention for subsequent difference analysis and reconstruction consistency verification.

[0040] The preferred method also includes step five: calculation of differences between pre-reconstruction and post-reconstruction images and multi-dimensional difference aggregation. For each image, the reconstruction results of N times are compared with the original image band by band-by-band difference calculation. The calculation metrics include structural dissimilarity (1-SSIM) and squared error (SE). The calculated difference map is then subjected to z-score normalization and Gaussian filtering to eliminate the influence of noise and isolated pixels. Subsequently, by multiplying the N multi-band difference maps generated for each image, which are composed of filtered structural dissimilarity (1-SSIM) and squared error (SE) maps, pixel by pixel in the corresponding bands, and summing the multiplication results of each image in each band pixel by pixel, an enhancement difference metric for each image in the nth reconstruction is constructed. Finally, all enhancement difference metric results for each image are averaged pixel by pixel to generate the final average difference heatmap, which is saved locally in raster form.

[0041] The preferred method also includes step six: automatic threshold calculation and threshold segmentation for the difference map. An automatic thresholding method is used to extract anomalies from the average difference heatmap generated in step five. After the threshold is determined, regions with pixel values ​​higher than the threshold are marked as anomalous event regions (marked as 1), and regions with values ​​lower than the threshold are marked as normal background (marked as 0), forming a binary prediction mask for the anomalous regions, which is then saved as a raster file.

[0042] The preferred method also includes step seven: model extraction accuracy estimation and iterative optimization based on labeled data. Based on the ground truth mask of the anomalous events corresponding to the images containing surface anomalies provided by the user in step four, the prediction mask obtained in step six is ​​automatically and quantitatively evaluated. A series of remote sensing binary classification accuracy indicators are calculated and output as log files, providing objective evidence for model performance. Specific reports include indicators such as the Confusion Matrix, Kappa Coefficient, IoU (Intersection over Union), and F1-Score. The F1-Score is a single indicator that comprehensively measures the model's precision and recall; it is the harmonic mean of the two. Its value ranges from 0 to 1; a higher score indicates a more balanced and better performance in avoiding false positives and false negatives. In addition, pixel-level comparison maps of TP / FP / TN / FN are generated to visually demonstrate the spatial reliability of the model segmentation results.

[0043] Simultaneously, the abnormal and non-abnormal regions in the average difference heatmap obtained in step six are separated using a truth mask, and a histogram of the difference scores in the normal and abnormal regions is plotted in one image to analyze the separability of the differences in the model reconstruction.

[0044] The current sampling accuracy is used as the convergence standard for model performance. If the sampling accuracy does not meet the preset standard, the user is guided to adjust the model fine-tuning parameters in step two based on the visualization analysis results. Steps two through seven are repeated until the sampling accuracy meets the standard, and the final optimized model is saved.

[0045] Step 8: Input the single-temporal remote sensing image to be detected into the fine-tuning model. Generate an anomaly heatmap through multiple mask reconstructions and difference aggregations, and segment out the anomaly regions from it. After completing the preceding steps, load the fine-tuning base model with optimal performance convergence, which was finally verified and saved in Step 7. Input any new single-temporal multispectral remote sensing image to be detected, with the same bands (ideally the resolution information should also be consistent, but inconsistencies are acceptable, though the effect may be slightly worse) as the fine-tuning image. The single-temporal multispectral remote sensing image to be detected needs to undergo the preprocessing procedures described in Step 1 (cloud pixel removal, standardized framing and cropping, 20% overlap setting) to ensure that the input format is consistent with the model training samples. Subsequently, following the data loading and processing procedures described in Steps 4 and 5 (only the inference model is replaced with the fine-tuning base model with optimal performance convergence, which was finally verified and saved in Step 7), perform N random independent mask reconstruction operations on the input image and generate an average difference heatmap of the region to be identified. Following the method in step six, the automatic segmentation thresholding method is used for binarization segmentation, ultimately forming a binary mask of abnormal and non-abnormal regions of the input image, which is then saved in raster file format.

[0046] Step Nine: Based on the segmented anomaly regions, acquire multi-source features and construct prompt words. Utilize a multimodal large model to generate a semantic interpretation including anomaly type inference. Vectorize the final anomaly region binary mask (raster file) obtained in Step Eight, converting it to GeoJSON (a format used for data exchange and network representation in modern geographic information systems) or Shapefile (vector geographic data format). Automatically calculate and add information such as anomaly patch area (unit: square kilometers), center latitude and longitude, and land cover / land use status for each vectorized patch. Based on this, extract the enhanced true-color visualization map corresponding to the vector region. Simultaneously, extract the average multi-band reflectance of pixels near the center pixel of each patch, automatically plotting a spectral curve with the horizontal axis representing the center wavelength (or band number) and the vertical axis representing the average multi-band reflectance of the neighboring pixels of the patch center. Finally, structurally organize the true-color image map, spectral curve, and patch attribute information (area, coordinates, land cover type) into a Prompt. For each vector block, the constructed multimodal cue words are input into the selected multimodal large model, such as ChatGPT (specifically its visual version GPT-4V), LLaVA (Large Language and Vision Assistant), InternVL (Shusheng·Puyu Visual Language Large Model), or a multimodal model fine-tuned in the remote sensing field; the VLM visual language model is one such example. The model automatically determines the most likely type of surface anomaly event for suspected anomalous patches (e.g., flooding, forest fire, mudslide, war destruction, etc.) and generates an anomaly event brief of about 100 words, including possible event types, coverage area, and background information, thus achieving automated connection between anomaly detection and semantic interpretation.

[0047] In a preferred embodiment, the overall process of the method is as follows: Figure 1 As shown, the specific implementation steps are as follows: Step 1: Acquire and preprocess normal ground feature sample images to construct a training set.

[0048] In this embodiment, it specifically includes: Step 101: Determining the Base Model and Selecting the Data Source for Normal Ground Cover Sample Images. This embodiment first determines the Prithvi_EO2_300MTL visual base model, compatible with multispectral remote sensing imagery, as the mask reconstruction base model. Then, it selects an image set obtained by 10% stratified sampling and filtering by normal ground cover samples from a high-quality remote sensing image dataset (SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation) as the spatial index source for normal ground cover samples. This dataset provides information on land cover types and their corresponding patch locations, enabling the extraction of necessary spatial reference information such as the capture time and four-corner latitude and longitude of image samples consistent with the input requirements of the base model.

[0049] Step 102: Acquisition of multispectral remote sensing image data for model-compatible modalities. This embodiment uses the Google Earth Engine platform (a cloud-based geospatial data processing and analysis platform) as the interface for acquiring remote sensing data. The Harmonized Landsat and Sentinel-2 product (a combined Landsat and Sentinel-2 surface reflectance product) fused from Landsat-8 / 9 (a pair of sister Earth observation satellites) and Sentinel-2 is selected as the multispectral image source. This product has undergone unified radiometric calibration and atmospheric correction processing and has been resampled to a consistent 30 m spatial resolution, directly meeting the consistency requirements of the base model for input data.

[0050] Based on the input data requirements of the selected visual foundation model Prithvi_EO2_300MTL (a large-scale, multi-task pre-trained visual foundation model for Earth sciences), the number of image input channels was determined to be 6 bands: blue light (B2), green light (B3), red light (B4), near-infrared (B8A), shortwave infrared 1 (B11), and shortwave infrared 2 (B12). Furthermore, the standardized image patch size required for model training and inference was determined to be 224×224 pixels to ensure consistency between the sample input format and the foundation model structure.

[0051] Step 103: Construction of the target area and automatic generation of the buffer. Using GEE (Geospatial Data Processing and Analysis Platform) scripting, based on the four corner coordinates of each image obtained in Step 101, the target area is constructed for each sample image using its four corner latitude and longitude coordinates. Let the spatial range of the sample image be R, and the image's length, width, and number of pixels be H and W, then the buffer R is constructed. ' Spatial range.

[0052]

[0053] in This is a spatial dilation operation used to construct an external expansion region around the original patch, so that the subsequently retrieved imagery can fully cover the target feature and its surrounding background area. max is the maximum value.

[0054] Step 104, Automatic Image Retrieval with Temporal Consistency. Based on the image retrieval capabilities provided by the GEE platform, the target region R constructed in step 103 is automatically retrieved. ' Within the scope, according to the date the sample images were taken. Implement strict time window filtering. Specifically, set the time window as follows:

[0055] Where days represents days, and all spatially complete coverage buffers R are automatically retrieved within the above time range. ' The HLS multispectral imagery was used. A script was used to filter and retain only images with 100% coverage, ensuring high consistency of the input data in both spatial and temporal dimensions, providing a highly comparable sample base for subsequent mask reconstruction learning.

[0056] Step 105: Image quality control and cloud pixel removal based on cloud / cloud shadow masks. After obtaining candidate images through temporal screening, image quality control is performed using the cloud and cloud shadow quality masks provided with the multispectral imaging product. Specifically, for HLS products, their built-in cloud / shadow mask layer is used. Based on the cloud and shadow positions marked in the mask, the corresponding pixels are removed from the candidate images or set to invalid values ​​in the preprocessing workflow.

[0057] Step 106: Standardized image patch cropping. While maintaining the original multispectral image's 30 m spatial resolution, the image processed in Step 105 is standardized and cropped according to the input requirements of the basic model. Specifically, a fixed window of 224×224 pixels is used to slide and divide the image into patches. To reduce edge artifacts that may occur during mask reconstruction, a 20% overlap is set between adjacent windows. Let the window size be S=224 and the overlap rate be α=0.2, then the sliding step size w is expressed as:

[0058] In practice, a step size of w = 179 pixels is used to cover the entire image and ensure sufficient window overlap. Edge regions that cannot meet the 224×224 pixel size requirement or target regions whose area is insufficient to generate a complete window are not included in the sample set to ensure strict consistency of input size during subsequent model training and inference.

[0059] Step 107: Cloud Coverage Statistics and Filtering for Cropped Image Blocks. After completing the image cropping in Step 106, cloud coverage statistics are performed independently for each 224×224 pixel image block. Specifically, based on the cloud / cloud shadow mask generated in Step 105, a mask sub-image of the same size is cropped at the corresponding position, and the proportion of cloud and cloud shadow pixels to the total number of pixels in the image block is calculated, denoted as the Cloud Coverage Ratio (CCR).

[0060] in, This indicates the number of pixels containing clouds and cloud shadows within the image block.

[0061] Image patches with a cloud cover percentage exceeding 10% were directly removed, and only multispectral image patches meeting the cloud cover threshold were retained. Proceed to the next processing stage and download to a local storage location.

[0062] Step 108: Dataset partitioning of normal ground cover sample images. After completing the cloud cover filtering in Step 107, the remaining multispectral image patches are partitioned into a dataset. Specifically, all qualified normal ground cover multispectral image patches are randomly divided into a training set and a validation set in an 80%:20% ratio. The training set is used for subsequent model fine-tuning, while the validation set is used to monitor the convergence of the joint loss, evaluate the model's feature learning performance, and adjust hyperparameters.

[0063] Step 2: Using the training set, perform efficient parameter fine-tuning on the pre-trained mask reconstruction visual base model to obtain a fine-tuned model specifically for normal ground feature reconstruction.

[0064] After completing the normal ground cover sample construction in step one, the pre-trained high-parameter mask reconstruction basic visual model (Prithvi_EO2_300MTL) is progressively fine-tuned using the acquired large-scale normal ground cover multispectral image patches. This step aims to further enhance the model's ability to extract and reconstruct normal features for different land surface types, texture morphologies, and spectral combinations. This allows the model to accurately capture the spectral consistency patterns and spatial structural features of normal ground covers. By bringing normal ground cover features closer to the feature center, this improves the distinguishability between anomalous and non-anomalous regions and increases the reconstruction differences of anomalous regions in anomaly detection tasks.

[0065] This embodiment employs a freeze-thaw fine-tuning strategy combined with low-rank adaptation (LoRA) to reduce the computational overhead of training large models and improve fine-tuning stability. Specifically, in the initial stage of fine-tuning, all backbone parameters of the model are frozen first. The newly added low-rank adaptation module is then used to update the high-level weights of the model, enabling the model to quickly adapt to the normal ground cover distribution constructed in this embodiment while maintaining its original semantic, spatial, and spectral representation capabilities. Subsequently, the remaining layers are unfrozen to achieve fine-grained learning of normal ground cover patterns in a deeper feature space.

[0066] The fine-tuning process centers on the mask reconstruction task, randomly masking portions of the input image patch to enable the model to learn the inference ability of spectral-spatial joint features under conditions of missing information. The training loss employs a combination of multiple loss functions, including Pixel-level Reconstruction Error (MSE), Spectral Cosine Similarity Loss (SCS), and Feature Dynamic Center Loss (FDCL), constrained by feature clustering within the feature space, while simultaneously enhancing the model's spectral reconstruction ability, spatial structure preservation ability, and cross-regional feature consistency. The overall training adopts a dynamic learning rate adjustment strategy of "warmup-cosine annealing," with the first 20% of the total epochs designated as warm-up rounds. Using the validation set data partitioned in step one, the joint loss of the model on the validation set is evaluated every 5 epochs. This ensures stable convergence during training, and the model with the best performance and loss in the validation set is saved locally for subsequent inference.

[0067] In this embodiment, the specific steps include: Step 201, freezing the underlying structure and initial data adaptation. The goal of this stage is to adapt the model to the spectral characteristics of remote sensing images that do not contain anomalous features, and to bring the features of normal features closer together to facilitate differentiation from anomalous features in subsequent steps. Taking the Prithvi_EO2_300MTL model used as an example, this model consists of an encoder, a connection layer MLP, and a decoder. The freezing strategy here involves freezing the bottom layer of the encoder (the first 6 layers of the Transformer Block model core computing unit), part of the middle layer MLP (the middle 2 layers of the MLP structure), and the bottom layer of the decoder (the first 6 layers of the Transformer Block). The MLP is a multilayer perceptron.

[0068] Using the LoRA technique, only the high-level semantic structure is trained, and the joint loss is composed of the spectral composition difference between the reconstructed image and the original image, the overall reconstruction error, and the degree of clustering of extracted features in the feature space. To optimize the objective.

[0069] For a batch of input training set images, each reconstructed image is used. Compared to the original image The joint loss, composed of mean squared error (MSE), spectral cosine similarity (SCS), and feature dynamic center loss (FDCL), is used as an evaluation index for the reconstruction effect of a single image. For a batch of images, the mean loss is calculated.

[0070]

[0071] in It is a pixel-level reconstruction error. It is spectral cosine similarity. It is a dynamic feature center loss, while , , These are the weights corresponding to each loss. MSE is calculated for each pixel in each band. The calculation of SCS is performed on a multi-band vector for each pixel. The FDCL calculation is performed on a per-feature vector basis. and The calculation method is as follows.

[0072]

[0073] in, and These represent the original pixel value and the reconstructed pixel value in the i-th pixel and the c-th band, respectively. This indicates the number of rows and columns of pixels in the cropped image block (here are...). Number of internal pixels, and Let represent the vectors composed of the pixel values ​​of the i-th pixel in c bands, as shown in the following formula.

[0074]

[0075] The aim is to make the characteristics of normal samples Close to its dynamic cluster center in the representation space .in, It is the p-th patch image block (16) of the input image x. 16) Feature vector output by the encoder; It is the j-th normal feature center (multiple centers are allowed, j>1, where j will be determined by the feature type characteristics of the input image). At the start of training, It is initially set to any feature vector and is dynamically updated in each subsequent iteration.

[0076]

[0077] In this embodiment, the feature center is determined using the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, a density-based clustering algorithm. It divides sufficiently high-density regions into clusters, capable of discovering clusters of arbitrary shapes in noisy spatial data, without requiring a pre-defined number of categories (j). In this section, the neighborhood radius is set. The minimum number of points (MinPts) required to form a high-density region is set to 10, with a value of 0.4. In each iteration, the average vector (centroid) of each cluster automatically identified by DBSCAN is used as the current dynamic feature center. .

[0078] In this step, a robust learning rate is used (e.g., starting from 0 and going up to 5 × 10). -4 Then, the cosine annealing was gradually reduced to 0, and 100 epochs of training were performed with a batch size of 128 to ensure the reconstruction effect while initially learning the spectral composition of normal ground features.

[0079] Step 202, Unfreezing the entire network and refining the model: The goal of this stage is to learn more detailed spatial structural features while preserving basic reconstruction capabilities. All network layers are unfrozen, allowing the entire network parameters to be updated. For spatial texture features, a multi-scale joint loss is employed. This guides the model to learn more detailed spatial structural features. Using reconstructed images... Compared to the original image The joint loss consists of mean square error (MSE) between bands, multi-scale gradient magnitude similarity (MSGMS), and dynamic center loss (FDCL). MSE focuses on pixel-level reconstruction, while MSGMS focuses on capturing texture and detail information to improve the spatial fidelity of the reconstruction. FDCL is used to further ensure the clustering of normal features in the feature space.

[0080]

[0081] in, , , For corresponding losses , , The weight.

[0082] Multi-Scale Gradient Magnitude Similarity (MSS) is a multi-scale image quality assessment metric based on gradient magnitude, which focuses more on capturing details of image edges and textures. We use it in the loss function. The calculation method is as follows. First, calculate the original image. and reconstructed images gradient magnitude and Then, gradient magnitude similarity is calculated at S different scales (e.g., S=1,2,4). For scale S, gradient magnitude similarity The calculation formula is:

[0083] in, This is the stability constant, which is taken as 0.0005 here. Represents the original image At scale s, the first The intensity of the edge or texture of a pixel. Represents the reconstructed image At scale s, the first The intensity of the edge or texture of each pixel. Final It is for all scales S Weighted average:

[0084] Where M represents the number of image pixels at scale s. It is a scale weight, and Here we choose =0.3, =0.4, =0.3.

[0085] In this step, a lower learning rate is used (e.g., starting from 0 and going up to 5×10). -5 Then gradually cosine anneal to 0), or use a learning rate equivalent to that of stage one, with a batch size of 64, and perform fine-tuning for 100 epochs. The batch size is reduced from 128 to 64 because after the entire network is unfrozen, it is necessary to focus on fine-grained spatial structure learning. A smaller batch size can improve the flexibility of gradient updates, help the model capture more detailed texture features, and at the same time avoid parameter oscillations when combined with a low learning rate.

[0086] Preferably, it also includes step three: evaluating the feature discrimination effect of the model after fine-tuning.

[0087] After completing the model fine-tuning in step two, the model has basically achieved high-quality reconstruction of normal images. To verify whether the model's ability to distinguish between images of non-abnormal features and images of features with surface anomalies has met expectations, a systematic evaluation of the reconstructed images and their potential features output by the model is needed. This evaluation will verify whether the encoder can effectively distinguish between normal and anomalous features, providing a clear basis for subsequent parameter adjustments.

[0088] Step 301: Acquisition and Input of Anomalous Image Samples. To verify the feature extractor's ability to distinguish features after fine-tuning, this step selects multispectral remote sensing images containing typical anomalous events as test samples (approximately 100 images containing various anomalous events) and inputs them into the fine-tuned model for forward inference. The sources of anomalous images may include areas with significant spectral or spatial structural deviations, such as sudden fires, flooding, building damage, and road interruptions, to ensure effective testing of the model's sensitivity to anomalous patterns.

[0089] In this step, no training is required on the anomalous images; only standard format unification and spectral normalization processing according to model input requirements are needed before feeding them into the model. The data format, number of bands (6 bands), resolution (30 meters), and size (224) of this sample set are specified. The 224-pixel block must be processed in the same way as the normal ground feature sample image (only the image content includes the occurrence of surface anomalies).

[0090] Simultaneously, this sample set needs to possess high-quality, ground truth masks for anomalous events that closely reflect the actual surface conditions. These masks are accumulated in the early stages based on Sentinel-2 or Landsat imagery through manual interpretation or semi-automatic methods. If the user does not yet possess such anomalous samples containing ground truth masks, they must first obtain some typical anomalous event images and their corresponding precise masks through remote sensing image interpretation or on-site verification for use in this step.

[0091] Step 302: Encoder Feature Extraction and Dimensionality Reduction Visualization Analysis. After completing the anomaly image input in Step 301, to further verify the fine-tuned model's ability to distinguish between normal and anomalous ground features, this step extracts features from the last layer of the model's encoder and performs visualization analysis using dimensionality reduction methods. Specifically, 100 images containing surface anomalies and 100 normal images without surface anomalies are input into the fine-tuned model. High-dimensional feature vectors from the encoder output are extracted patch by patch (one feature vector per patch) from the input images. ), forming the high-dimensional feature vector set at the encoder output { These feature vectors contain a comprehensive representation of the spectral-spatial patterns of land features by the model, and can reflect the model's internal ability to distinguish different land surface conditions.

[0092] To visually represent the distribution relationship between normal and abnormal samples in the feature space, mainstream dimensionality reduction methods, such as t-distributed stochastic neighbor embedding (t-SNE), are applied to the high-dimensional features to reduce the high-dimensional feature vectors to a two-dimensional space for easier visualization and analysis. The specific parameters for t-SNE are set as follows: perplexity is set to 30, the maximum number of iterations is set to 1000, and the rest are left as default. The dimensionality-reduced two-dimensional feature points are visualized on a scatter plot, using different colors to distinguish between the feature points of normal and abnormal samples, and their distribution in the representation space is analyzed: normal samples typically exhibit compact clustering in the feature space, while abnormal samples, due to their disruption of the normal pattern, are often distributed at locations far from normal clusters or form independent clusters.

[0093] Step 303: Performance Evaluation and Parameter Iteration. Based on the dimensionality reduction results from Step 302, observe the clustering and separation of normal and abnormal samples in the feature space. If normal samples form stable and compact clusters, and abnormal samples are far from the normal sample clusters or form independent clusters, it indicates that the model has good feature discrimination ability. If normal and abnormal samples are significantly mixed, the training strategy in Step 2 needs further optimization. Specific strategies include, but are not limited to: Maximum learning rate, warm-up epochs, and maximum number of iterations (Epochs) for Phase 1 / 2; weights of each term in the joint loss function (e.g., ... , , , (etc.); neighborhood radius in feature clustering Etc.; freezing / unfreezing network layers, microcall matrix dimensions, scaling factors, optimizer and other fine-tuning parameters.

[0094] By re-executing the fine-tuning training in step two and repeating the verification process in steps 301–302, a closed-loop optimization mechanism is achieved. When the model achieves stable performance in terms of reconstruction differences, feature separation, and normal class clustering, the fine-tuning phase is considered complete, and the model can be used in the subsequent abnormal region extraction and intelligent interpretation of abnormal events stages.

[0095] The preferred method also includes step four: abnormal image input and fine-tuned model inference.

[0096] Step 401, Anomaly Image Input and Model Invocation. The final, fine-tuned Prithvi_EO2_300MTL pedestal model, guided by normal ground features, is loaded into the inference environment of a high-performance computing device (such as a GPU) for batch inference. The input data consists of 224 images containing potential surface anomalies used for feature discrimination verification in Step 301. 224 single-temporal multispectral remote sensing images.

[0097] Step 402, Multiple Random Independent Masking and Reconstruction. To fully utilize the characteristics of Masked Image Modeling (MIM), namely the difference between normal features being easy to reconstruct and abnormal features being difficult to reconstruct, this step performs multiple independent forward inferences on each input 224×224 pixel block. A high masking ratio V (e.g., 75%) is set to ensure that V pixels in the input image block are randomly masked during each reconstruction inference.

[0098] To ensure that all pixels are reconstructed at least once and to enhance the statistical stability and robustness of anomaly detection, N (N=32) random independent masking operations are performed on each image block. Each mask is generated using a different fixed random seed to ensure the variability and traceability of occlusion locations.

[0099] Each time a masking operation is performed (n=1 to N), a corresponding binary mask position is generated. Figure 2 Value mask location diagram (Occluded pixels are represented by 1, unoccluded pixels by 0). The image after the nth masking is input into the fine-tuned model for forward inference. The model only reconstructs occluded pixels, generating N different reconstructed images. (n=1 to N), the single mask and reconstruction results are shown in the figure. Figure 2 As shown.

[0100] Step 403: Normalized storage of multiple reconstruction results and mask images, storing the N reconstructed images generated from each reconstruction. And the corresponding N mask position diagrams They are stored together in a separate directory and named using a standardized naming convention (e.g., [Image ID]_[Reconstruction Count]_reconstructed.tif and [Image ID]_[Reconstruction Count]_mask.tif) to facilitate subsequent batch difference analysis and reconstruction consistency verification.

[0101] The preferred method also includes step five: reconstruction difference calculation and multidimensional difference aggregation.

[0102] Step 501, Band-by-band difference calculation. For each reconstruction in the N independent reconstruction results, two difference indices are selected to calculate the original input image x and its corresponding n reconstructed images. The band-by-band differences between them.

[0103] Structural dissimilarity (1-SSIM) measures the difference between the reconstructed and original images in terms of brightness, contrast, and structure. Since SSIM values ​​range from [0,1], where 1 represents perfect similarity, we use [1-SSIM] in the loss function. This is used to represent structural differences. The computation is performed based on local windows (e.g., a 5x5 window), and a global traversal of the input image is completed when overlapping sliding windows exist. For images... and The SSIM calculation formula for a specific local window within a certain band is as follows:

[0104] in and It is the average pixel value within the local window. and It is the variance within a local window. It is the covariance within a local window. and It is a stability constant used to avoid the denominator being zero, where This refers to the dynamic range of pixel values ​​(here, because the initial data was standardized by z-score, the data range was truncated to -3 to 3, so L is set to 6). and It is a small constant (e.g.) , ).

[0105] Squared error (SE) is used to measure the reconstruction error pixel by pixel. That is, the squared error can be calculated directly for each band of the image without the need for windowing. The specific calculation method is as follows:

[0106] To eliminate the influence of absolute values ​​and make the two difference measures numerically comparable, the calculated values ​​were... Figures and The graphs were subjected to z-score standardization.

[0107] Subsequently, Gaussian filtering (e.g., using a Gaussian kernel of σ=1.4, k=7) is performed on the standardized difference map band by band. This operation aims to eliminate high-frequency noise and the effects of isolated pixels generated during the reconstruction process, forming a difference raster map.

[0108] Step 502, Multidimensional Difference Enhancement and Convergence. This step involves fusing numerical error (SE) and structural error. The data is aggregated in terms of band and reconstruction count to generate a highly reliable heatmap of anomalies.

[0109] Specifically, such as Figure 4 As shown, for a single input image, the band-by-band 1-SSIM difference map obtained from the normalization and filtering in step 501 is multiplied pixel-by-pixel with the SE difference map to construct an enhanced difference measure. The multiplication operation leverages the complementarity of the two difference measures, highlighting regions with significant reconstruction biases in both spectral values ​​and spatial structure, thereby greatly suppressing false positives under a single indicator.

[0110]

[0111] in, Indicating the nth reconstruction and band The increased differences Indicating the nth reconstruction and band The squared error, Indicating the nth reconstruction and band Structural dissimilarity, This represents a pixel-by-pixel multiplication operation between two matrices.

[0112] By analyzing the results of a single reconstruction Sum of corresponding pixels is performed across all bands C=6 to achieve comprehensive accumulation of errors across bands. Then, the N cumulative difference maps obtained from N independent reconstructions (N=32) are averaged pixel-by-pixel. Finally, a result is generated from the original input image. and its Nth reconstruction image The final average difference heatmap obtained by difference fusion ,like Figure 3 As shown.

[0113]

[0114] The final average difference heatmap Save as a raster file. The pixel values ​​in this heatmap directly represent the probability or degree of an anomaly of that pixel, serving as the sole basis for subsequent automated segmentation. A visual flowchart of the main processing steps is shown below. Figure 3 As shown.

[0115] The preferred method also includes step six: automatic threshold calculation and segmentation of the difference map.

[0116] Complete the final average difference heatmap After generation, this step uses a statistical method to automatically determine the optimal segmentation threshold for the abnormal region and finally generates a binary mask for the abnormal region. In this embodiment, the cumulative distribution function (CDF) combined with elbow identification is preferred for threshold determination. This method can effectively utilize the statistical characteristic that normal background dominates in remote sensing images. At the same time, other adaptive thresholding methods such as Otsu's method (maximum inter-class variance principle) can also be selected according to the data characteristics.

[0117] Step 601: Construction of the global cumulative distribution function (CDF) for difference scores. Extract all average difference heatmaps generated in Step 5. All pixel values ​​(i.e., difference scores) are considered. The cumulative distribution function of these difference scores is then calculated. CDF curve This indicates that the difference score is less than or equal to a certain value. The proportion of pixels.

[0118]

[0119] Where P is the probability. These are the pixel differences on the heatmap. The difference value is a variable. Since normal ground features occupy the vast majority of the area in the remotely sensed image, the CDF curve shows a rapid upward trend at lower difference scores, which corresponds to a large area of ​​normal background.

[0120] Step 602: Adaptive threshold determination based on the "elbow point". In step 601, the inflection point where the rapidly rising portion (normal features) of the CDF curve is about to transition to the slowly rising region (abnormal features) is defined as the "elbow point". This point statistically represents the statistical distribution boundary between normal and anomalous difference values. The KneeLocator method is used to identify inflection points on the upward trend curve, with a sensitivity parameter set to 0.7. If this method fails to identify the elbow point, the system will degenerate to using Otsu's method based on the maximum inter-class variance principle for alternative threshold determination. The difference score corresponding to the identified "elbow point" is plotted on the difference value axis (x-axis). This is determined as the final automatic segmentation threshold.

[0121] Step 603, Binary mask segmentation and result output, using a predetermined automatic threshold. For all average difference heatmaps Perform binarization segmentation. Pixel values ​​greater than or equal to... The area marked as an abnormal event area (marked as 1) is defined as a pixel value lower than [a certain value]. The regions are marked as normal background regions (marked as 0). This forms a series of binary masks for the anomalous regions of the input image. Finally, a binary prediction mask for the anomalous regions of the input image is formed.

[0122]

[0123] The mask Save in a standard raster file format (such as GeoTIFF).

[0124] The preferred method also includes step seven: estimation and iterative optimization of model extraction accuracy based on labeled data.

[0125] This step will utilize a ground truth mask of anomalous events from a portion of the imagery provided by the user to annotate the range. The anomaly region prediction mask obtained in step six Perform quantitative evaluation and provide visualization evidence to guide and optimize the model fine-tuning parameters in step two, ensuring that the model converges to optimal performance.

[0126] Step 701, pixel-level binary classification quantitative evaluation. The ground truth mask prepared by the user in step three is used... Compared with the predicted mask obtained in step six Precise pixel-level spatial alignment is performed within the same coordinate system. Based on pixel-level comparison, a confusion matrix for binary classification is calculated, including true positives (TP), true negatives (TN), false positives (FP, false detections), and false negatives (FN, false misses).

[0127] Based on the confusion matrix, it automatically calculates and outputs log files, providing objective evidence for model performance. Specific evaluation metrics include: Intersection over Union (IoU): TP / (TP+FP+FN), is an important indicator for measuring the degree of overlap between segmented regions.

[0128] F1-Score: The harmonic mean of precision and recall, balancing false positives and false negatives.

[0129]

[0130] Kappa coefficient: measures the consistency between predicted results and random classification results. The formula for calculating the Kappa coefficient is as follows: where, For observation consistency (i.e., overall accuracy). This is coincidental consistency (i.e., expected consistency). This refers to the number of pixels within the input image range size.

[0131]

[0132]

[0133]

[0134] Step 702: Reconstruction residual separability analysis and pixel-level segmentation result visualization. Generate TP / FP / TN / FN pixel-level contrast maps, labeled using four different colors (e.g., green-TP, red-FP, yellow-FN, blue-TN). Figure 5 As shown in the figure, this diagram visually illustrates the spatial reliability of the model segmentation results and the specific locations and patterns of false positives and false negatives.

[0135] Utilize each input image Corresponding binary mask The average difference heatmap obtained in step 502 The pixel values ​​were separated into two groups: "normal region difference score" and "abnormal region difference score," and histograms of these two difference scores were plotted on the same graph. The separability of the model reconstruction differences was evaluated by analyzing the distribution and overlap of the two histograms.

[0136] The closer the distribution of variance values ​​in non-abnormal regions is to the low residual region (error is 0), and the closer the distribution of variance values ​​in abnormal regions is to the high residual region, and the smaller the overlap between the two distributions, the stronger the model's ability to distinguish between normal and abnormal conditions. Step 703: Model convergence determination and parameter iterative adjustment. The evaluation result (i.e., sampling accuracy) of the current image set containing the ground truth mask in step 702 is used as the convergence criterion for model performance. If the sampling accuracy does not meet the preset standard (e.g., IoU in outlier regions is below 0.5), the pixel-level contrast map and difference histogram analysis results from step 702 are used to guide the user in adjusting the model fine-tuning parameters in step two.

[0137] After adjusting the parameters, repeat steps two through seven until the sampling accuracy index meets the target. At this point, the fine-tuning is considered complete, and the final optimized model is saved.

[0138] Step 8: Input the single-temporal remote sensing image to be detected into the fine-tuning model, generate an anomaly heatmap through multiple mask reconstructions and difference aggregations, and segment out the anomaly regions from it.

[0139] This step describes how to use the final convergent model, which has been verified and iteratively optimized in step seven, to perform fully automated anomaly detection and precise location on any new, unlabeled single-temporal remote sensing image, thereby enabling the generalization and deployment of the method.

[0140] Step 801: Image preparation and final convergence model loading. Load the fine-tuned base model (such as Prithvi_EO2_300MTL) that has been finally verified and saved in Step 7 and has achieved optimal performance convergence. This model now has the ability to accurately reconstruct normal ground feature patterns and effectively distinguish abnormal patterns in the feature space.

[0141] Then, any new single-temporal multispectral remote sensing image containing potential surface anomalies that has the same band and resolution information as the micro-telegraph image is input into the system (e.g., using a (HLS) fused image containing 6 bands (B2, B3, B4, B8A, B11, B12) with a resolution of 30 meters).

[0142] Since the basic model inference stage already includes cropping images larger than the input size, standardizing the input data, and finally automatically stitching the reconstructed image, only high-quality remote sensing images need to be input in the next stage.

[0143] Step 802: Multiple masking inference and final difference heatmap generation. The model automatically invokes the distributed GPU inference function to crop the prepared input image into blocks and performs parallel inference strictly according to the multiple random independent masking strategy described in steps 402 and 403. Each image undergoes N (e.g., N=32) random independent masking reconstruction operations, generating N reconstructed images. .

[0144] Then, following the multidimensional difference enhancement and aggregation process described in step five, the N reconstruction results are enhanced by SE and 1-SSIM at the band level and aggregated across bands and across reconstruction times to finally generate an average difference heatmap of the region to be identified.

[0145] Step 803, Unsupervised Adaptive Segmentation and Binary Mask Output. Following the method in Step 6, the segmentation threshold is automatically determined for the generated average difference heatmap using the "elbow point" method (or Otsu's method) based on the cumulative distribution function (CDF). .

[0146] Use automatically determined thresholds The heatmap is binarized and segmented. Regions with pixel values ​​greater than or equal to a threshold are marked as anomalous event regions (1), and regions with pixel values ​​less than the threshold are marked as anomalous event regions. The area is marked as normal background (0).

[0147] The final result is a binary mask of the abnormal and non-abnormal regions of the input image, which is saved in raster file formats such as GeoTIFF, enabling rapid, accurate, and fully automatic segmentation of the locations where surface anomalies occur.

[0148] Step 9: Based on the segmented abnormal regions, obtain multi-source features and construct prompt words, and use a multimodal large model to generate a semantic interpretation that includes anomaly type inference.

[0149] This section describes how, after the abnormal region is accurately segmented, multi-source and multi-modal information is aggregated, and a visual language model (VLM) is used for high-level semantic understanding and automatic briefing generation, thereby achieving automated connection between anomaly detection results and semantic interpretation.

[0150] Step 901: Vectorization, attribute extraction, and multi-source information fusion of anomalous patches. The final binary mask (raster file) of the anomalous area obtained in Step 8 is vectorized and converted into a standard GIS vector data format (such as Shapefile or GeoJSON). Each isolated anomalous area after conversion corresponds to an independent polygonal patch feature.

[0151] For each vectorized patch, the following geometric attributes are automatically calculated and added to its attribute table, including: (1) Patch area: accurately calculate the actual coverage area of ​​the anomalous patch on the ground, in square kilometers; (2) Center latitude and longitude: calculate the latitude and longitude coordinates of the geometric centroid (center point) of the patch; (3) Land cover / land use map overlay: overlay the anomalous patch vector with the latest or corresponding year's land cover type map (e.g., using a 10-meter resolution land cover product supported by Sentinel-2). Extract the land cover category (e.g., cultivated land, forest, water body, artificial surface, etc.) corresponding to the center pixel of each patch, and write this category information into the patch attribute table.

[0152] Step 902: Visual modal cue word construction and multimodal cue word combination. In this step, the program extracts the RGB channels from the corresponding image input in step eight based on the boundaries of each extracted vector anomalous patch, performs true-color image visualization under 1% linear stretching, and enhances the overlay area using the vector image as the boundary to obtain a highlighted image that clearly displays the visual area of ​​the anomaly identification region.

[0153] In addition, the average reflectance values ​​of 24 pixels in the central pixel and its surrounding square neighborhood of each patch are extracted across 6 bands (B2, B3, B4, B8A, B11, B12). Based on these multi-band reflectance values, a spectral curve of the anomalous patch with horizontal and vertical axes (where the vertical axis represents reflectance and the horizontal axis represents the center wavelength of the band) is automatically plotted and saved as an image file.

[0154] The Prompt is structured by combining true-color images and spectra of the anomalous area, as well as information on patch attributes (area, coordinates, land cover type, etc.) with task requirements (e.g., "Analyze the remote sensing images of this area, and determine the possible types of surface anomalies based on the shape, spectral characteristics, and land cover type of the patches").

[0155] Specifically, the image input includes a true-color image of the suspected anomalous event area {the true-color image of the anomaly identification area obtained in step 902 (fig1)}, and a spectral curve of the center pixel of the suspected anomalous event area {the spectral curve obtained in step 902 (fig2)}. The text input provides attribute information and instructions in structured text form, such as: "A suspected surface anomaly has been detected. Its center latitude and longitude are: [XXX, YYY]; the coverage area is approximately: [A] square kilometers; the background surface cover type is: [Z]. Please analyze the remote sensing image of this area, and based on the shape of the patches, spectral characteristics, and surface cover type, determine the possible type of surface anomaly event, and generate an anomalous event brief of approximately 100 words." The data in [] is provided by step 901.

[0156] Step 903, VLM intelligent interpretation and automatic briefing generation. For each vector block, the multimodal cue words constructed in step 902 are input in a formatted form into the selected visual language model (e.g., chatgpt, LLaVA, InternVL, or a VLM fine-tuned for the remote sensing domain).

[0157] VLM, leveraging its cross-modal understanding capabilities, performs deep fusion analysis of visual content and textual information to automatically determine the most likely semantic type of anomalous patches (e.g., flooding, forest fires, illegal land excavation, mudslides, etc.) and generate an anomalous event brief. The brief includes at least: the possible event type (e.g., small-scale mudslides, post-fire traces, seasonal flooding, illegal building demolition, etc.), the coverage area and impact level, and inferences based on background information. This achieves automated, end-to-end integration from pixel-level anomaly detection in remote sensing imagery to high-level, understandable semantic interpretation.

[0158] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, such as the combination of technical features between embodiments, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for anomaly detection in single-temporal remote sensing images based on mask reconstruction, characterized in that, Includes the following steps: Acquire and preprocess normal ground cover sample images to construct a training set; Using the training set, the parameters of the pre-trained mask reconstruction visual base model are efficiently fine-tuned to obtain a fine-tuned model specifically for normal ground feature reconstruction. The single-temporal remote sensing image to be detected is input into the fine-tuning model. Anomaly heatmaps are generated through multiple mask reconstructions and difference aggregations, and anomalous regions are segmented from them.

2. The method according to claim 1, characterized in that, It also includes the following steps: Based on the segmented abnormal regions, multi-source features are obtained and prompt words are constructed. A semantic interpretation containing anomaly type inference is generated using a multimodal large model.

3. The method according to claim 1, characterized in that, Efficiently fine-tuning the parameters of the pre-trained mask reconstruction visual baseline model includes: A two-stage, gradual strategy, including a freezing phase and a thawing phase, is adopted for fine-tuning; During the fine-tuning process, a joint loss function is used for constraints, which is used to simultaneously optimize the spectral reconstruction fidelity, spatial structure reconstruction quality, and the clustering of normal ground feature representations of the model.

4. The method according to claim 3, characterized in that, The joint loss function includes a feature dynamic center loss, which is used to cause the normal ground features extracted by the encoder to cluster towards dynamically updated cluster centers in the feature space.

5. The method according to claim 3, characterized in that, The two-stage progressive strategy includes: In the first stage, part of the underlying network structure of the model is frozen and fine-tuned using a first loss function, which includes at least pixel-level reconstruction loss and feature dynamic center loss. In the second stage, all or most of the network structure of the model is unfrozen and fine-tuned using a second loss function, which includes at least pixel-level reconstruction loss and multi-scale spatial structure similarity loss.

6. The method according to claim 1, characterized in that, The generation of anomaly heatmaps through multiple mask reconstructions and difference aggregation specifically includes: Perform N independent random masking operations on the image to be detected, and obtain N reconstruction results through the fine-tuning model; For each reconstruction, at least two measures of difference between the original image and the reconstructed image are calculated. After standardizing and spatially filtering each difference measure, pixel-by-pixel multiplication and spectral aggregation of all bands are performed to obtain a single-channel enhanced difference map. The anomaly heatmap is generated by averaging the single-channel enhancement difference maps from all N reconstructions pixel by pixel.

7. The method according to claim 6, characterized in that, The at least two difference measures include the squared error, which measures the difference in pixel values, and the structural dissimilarity, which measures the similarity of local structures.

8. The method according to claim 1, characterized in that, Abnormal regions are segmented from the abnormal heatmap, and the segmentation threshold is determined by an adaptive thresholding method based on image statistics.

9. The method according to claim 2, characterized in that, The process of acquiring multi-source features and constructing prompt words specifically includes: The segmented abnormal region binary mask is converted into a vector polygon; Calculate the geometric properties and geographic information of each vector polygon; Extract the spectral features and visualization of the original image region corresponding to each vector polygon; The geometric attributes, geographic information, spectral features, and visualization images are organized into structured multimodal cue words.

10. The method according to claim 9, characterized in that, The geometric attributes and geographic information include at least morphological features, spatial positioning features, and geographic environment features; the spectral features include multi-band reflectance curves.

11. The method according to claim 10, characterized in that, The morphological features include at least one of area, perimeter, major axis length, minor axis length, shape index, rectangularity, and boundary complexity; the spatial positioning features include at least one of geometric center coordinates, minimum bounding rectangle range, and azimuth distance from a preset geographic reference point; the geographic environment features include at least one of background surface cover type, administrative division, average elevation, and average slope; the spectral features also include one or more spectral indices calculated based on the multi-band reflectance curve.

12. The method according to claim 9, characterized in that, The multimodal cue words are input into a visual language model, which fuses and analyzes image and text information to automatically generate a natural language briefing that includes the type of abnormal event, the scope of its impact, and background analysis.

Images