Robust disaster detection method based on similarity and semantic information

Abstract

The invention discloses a robust disaster detection method based on similarity and semantic information, and the method employs the following innovative scheme: introducing an inter-frame similarity calculation mechanism based on a perceptual hash algorithm (PHash), carrying out the real-time comparison of continuous frames in a monitoring video, and effectively perceiving whether a scene changes remarkably or not. The module can automatically filter static or redundant frames and screen out key frames possibly containing events, the logic process of a monitoring system in actual operation is met, the calculation load of follow-up processing is remarkably reduced, and the overall processing efficiency of the system is improved. By means of the pre-trained CLIP model, high-level semantic analysis of the monitored image is achieved, and preliminary anomaly judgment can be carried out on the image content from the perspective of vision-language combined representation. The module utilizes the strong cross-modal alignment capability of the CLIP to carry out robust evaluation on whether an image belongs to a normal scene or not, so that the adaptability of a system to an unknown scene and an unseen sample is effectively improved, and the overall recognition robustness is enhanced.

Description

Technical Field

[0001] This invention relates to the field of disaster detection technology, specifically a robust disaster detection method based on similarity and semantic information. Background Technology

[0002] Current research on image-based disaster detection tasks often employs methods based on a single technical approach, such as image classification, object detection, or semantic segmentation, with typical examples including ResNet and YOLO. Although such methods can achieve high recognition accuracy on carefully constructed experimental datasets, the diverse forms, complex backgrounds, and varying behaviors of disaster events in real-world scenarios mean that methods relying solely on limited manually labeled data are insufficient to fully meet the complex needs of practical applications.

[0003] Current technology addresses this problem in the following ways: Image similarity filtering: By calculating the similarity between consecutive frames in a surveillance video using the perceptual hashing algorithm PHash, significant changes in the scene can be effectively determined, thereby filtering out irrelevant frames and selecting key frames. This method not only meets the logical requirements of surveillance systems in actual operation but also significantly reduces the system's computational load and improves processing efficiency.

[0004] Scene understanding at the semantic level: With the help of the pre-trained CLIP model, the system can perform high-level semantic parsing of monitoring images to preliminarily determine whether there are abnormal or disaster events. The CLIP model, with its powerful vision-language alignment capability, can robustly evaluate whether an image belongs to a normal scene, thereby significantly enhancing the system's adaptability to unseen samples and its overall robustness.

[0005] Fine-grained disaster event classification: By pre-training the Masked Autoencoder (MAE) model on ImageNet and the Incident1M disaster dataset, and integrating a custom classification head to predict disaster categories, the system can achieve more refined event identification. Simultaneously, leveraging GradCAM visualization technology, the specific areas where disasters occur can be further located, enhancing the model's interpretability and practicality, and providing more detailed evidence for disaster assessment. Summary of the Invention

[0006] This invention proposes a robust disaster detection method based on similarity and semantic information, employing the following innovative approach: A frame-to-frame similarity calculation mechanism based on the Perceptual Hash (PHash) algorithm is used to compare consecutive frames in surveillance video in real time, effectively detecting whether significant changes have occurred in the scene. This module can automatically filter static or redundant frames, selecting key frames that may contain events. This not only conforms to the logical flow of the monitoring system in actual operation but also significantly reduces the computational load of subsequent processing, improving the overall processing efficiency of the system. A pre-trained CLIP model is used to achieve high-level semantic parsing of the surveillance images, enabling preliminary anomaly detection of the image content from the perspective of joint visual-linguistic representation. This module utilizes CLIP's powerful cross-modal alignment capability to robustly evaluate whether an image belongs to a normal scene, thereby effectively improving the system's adaptability to unknown scenes and unseen samples, enhancing overall recognition robustness. A masked autoencoder (MAE) is used for self-supervised pre-training on image data to obtain a general representation capability for disaster-related features. A classification head is then introduced and fine-tuned on labeled samples to achieve fine-grained disaster event classification. Meanwhile, by combining GradCAM visualization technology, the model can accurately locate the disaster-stricken area, which not only improves the interpretability of the model but also provides more detailed evidence for disaster analysis, assisting in subsequent judgment and decision-making.

[0007] Routine Disaster Monitoring: This system leverages existing disaster prevention and monitoring infrastructure (such as surveillance cameras in mountainous areas, rivers, bridges, and geological disaster sites) to automatically analyze real-time video and image data. The system can detect various disaster phenomena in real time, including sudden geological disasters, and immediately trigger multi-level alarm mechanisms upon detecting potential hazards. This significantly shortens the interval between disaster occurrence and emergency response, improving the intelligence level of the disaster monitoring system and the efficiency of disaster relief. A robust disaster detection method based on similarity and semantic information is proposed. The specific steps of this robust disaster detection method based on similarity and semantic information are as follows: Step 1: Pre-training the Masked Autoencoder (MAE) model. The MAE model is built for reconstruction training. Its basic Transformer modules consist of the following structure: norm1 (LayerNorm), attn (Attention), norm2 (LayerNorm), and mlp (MLP). The encoder is composed of 12 stacked Transformer layers, and the decoder consists of 8 Transformer layers. The model is pre-trained on the ImageNet and Incident1M datasets. The input is the original image and its random masked version. During training, the reconstruction of the masked image region is achieved by minimizing the following reconstruction loss function: ; Step 2: Train the Vision Transformer (ViT) fine-grained classifier to obtain a pre-trained masked autoencoder (MAE). Keep the encoder part and discard the decoder. Use the encoder weights as the backbone network for the downstream disaster detection task. According to the needs of the specific disaster dataset, add one or more randomly initialized task-specific output heads after the pre-trained encoder and set its output dimension to the number of disaster categories. Step 3: Image similarity screening. Monitored images are acquired at fixed time intervals t. Then, the perceptual hash algorithm (Phash) is used to calculate the hash values ​​of the two consecutive frames of images. The offset value p is obtained by subtraction to quantify the visual content difference between the two frames of images. If p is greater than the set threshold, it is marked as an image to be processed. Step 4: Semantic scene understanding. A classification module is built based on the CLIP model and pre-trained weights are loaded. Although CLIP has excellent zero-shot classification performance, there is still a risk of misjudgment in complex scenes, and its performance is highly dependent on the construction of input prompt words. The text prompt words are set to three options: "normal scene", "abnormal scene" and "disaster scene". The image to be processed after being filtered in Step 3 and the above text prompts are input into the CLIP model to obtain its matching results. After Softmax normalization, the probabilities corresponding to the three categories are obtained, and the category with the highest probability is output as the preliminary evaluation result. Step 5: Fine-grained classifier. Using the fine-grained classifier trained in Step 2, perform fine-grained classification of disaster events on the scene images that were determined to be abnormal or disaster categories in Step 4, and output the final classification results. Step 6: GradCAM Visualization. The three-level cascaded disaster detection process can effectively identify disaster phenomena in the scene, but it does not yet have the ability to locate the disaster location. The GradCAM method is introduced to analyze network features to identify key regions that affect classification decisions. The target layer of GradCAM is set as the norm1 layer of the last Transformer module in the pre-trained encoder. After completing the classification in Step 5, the obtained classification results are fed back into the network. GradCAM is used to generate important feature regions corresponding to the category. Finally, the reshape operation is used to map them back to the original image size to achieve visualization of the localization results.

[0008] Preferably, in step 1, This represents the set of indices for all masked image patches in the batch. This represents the size of the set, i.e., the total number of masked blocks. and Representing the first The reconstructed pixel values ​​and real pixel values ​​at each location can be well adapted to image data by the pre-trained model MAE through self-supervised learning. It can form a good feature distribution for images, which greatly facilitates fine-tuning for downstream tasks.

[0009] Preferably, in step 2, the entire network is trained in a supervised manner in an end-to-end manner, during which the encoder weights are frozen and gradient updates and optimizations are performed only on the parameters of the task-specific output head.

[0010] Compared with the prior art, the beneficial effects of the present invention are: 1. Computational efficiency and real-time performance: The system adopts a three-level cascaded processing architecture. Through pre-processing perceptual hash filtering and CLIP zero-shot classification, redundant frames and simple negative samples are greatly filtered out. The most time-consuming fine-grained ViT classifier is only used for a small number of suspicious images, realizing on-demand allocation of computing resources and significantly improving the overall efficiency of the system, making it more suitable for actual deployment.

[0011] 2. Model performance and data dependence: An innovative approach employs a large-scale self-supervised pre-trained MAE encoder as the feature backbone, freezing its weights and fine-tuning only the lightweight classification head. This strategy powerfully transfers general visual representations to disaster domains, significantly reducing dependence on labeled data while improving the model's generalization ability and convergence speed.

[0012] 3. Regarding the credibility and interpretability of the results, the system combines multimodal cue words to optimize the semantic guidance capabilities of CLIP and integrates GradCAM visualization technology. It not only provides classification results but also generates intuitive heatmaps to locate key disaster areas in images, greatly enhancing the credibility of the results and the interpretability of the system, providing intuitive visual evidence for decision-making. Attached Figure Description

[0013] Figure 1 This is a diagram of the three-level cascaded architecture of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0015] Example 1

[0016] Image similarity filtering module Innovation: The paper proposes to use the perceptual hash (PHash) algorithm as a front-end filtering module. By calculating the structural similarity between video frames, it can efficiently identify scene changes, thereby accurately filtering out "key frames" that may contain events from massive video streams, and abandoning the redundant calculation mode of performing complex model reasoning on each frame.

[0017] Semantic-level scene understanding module Innovation: A CLIP-based evaluation mechanism is innovatively introduced. Leveraging powerful visual language semantic alignment capabilities, image content is matched with semantic information such as "normal" and "disaster," achieving zero-sample detection of whether an anomaly exists in the image.

[0018] Fine-grained classification and localization module Innovation: A progressive training paradigm of "self-supervised pre-training + supervised fine-tuning" was constructed, and GradCAM visualization technology was integrated. MAE's self-supervised learning enables the model to learn general representations from a large number of unlabeled images, and then fine-tunes it to adapt to downstream tasks through a small number of labeled samples. Finally, GradCAM provides pixel-level heatmaps for disaster area localization.

[0019] A robust disaster detection method based on similarity and semantic information is proposed. The specific steps of this robust disaster detection method based on similarity and semantic information are as follows: Step 1: Pre-training the Masked Autoencoder (MAE) model. The MAE model is built for reconstruction training. Its basic Transformer modules consist of the following structure: norm1 (LayerNorm), attn (Attention), norm2 (LayerNorm), and mlp (MLP). The encoder is composed of 12 stacked Transformer layers, and the decoder consists of 8 Transformer layers. The model is pre-trained on the ImageNet and Incident1M datasets. The input is the original image and its random masked version. During training, the reconstruction of the masked image region is achieved by minimizing the following reconstruction loss function: ; This represents the set of indices for all masked image patches in the batch. This represents the size of the set, i.e., the total number of masked blocks. and Representing the first The reconstructed pixel values ​​and real pixel values ​​at each position can be well adapted to image data through the self-supervised learning pre-trained model MAE, which can form a good feature distribution for images, greatly facilitating fine-tuning for downstream tasks. Step 2: Train the Vision Transformer (ViT) fine-grained classifier to obtain a pre-trained masked autoencoder (MAE). Keep the encoder part and discard the decoder. Use the encoder weights as the backbone network for the downstream disaster detection task. According to the needs of the specific disaster dataset, add one or more randomly initialized task-specific output heads after the pre-trained encoder and set their output dimension to the number of disaster categories. Perform supervised training on the entire network in an end-to-end manner. During this process, freeze the encoder weights and only update and optimize the gradients of the parameters of the task-specific output heads.

[0020] Step 3: Image similarity screening. Monitored images are acquired at fixed time intervals t. Then, the perceptual hash algorithm (Phash) is used to calculate the hash values ​​of the two consecutive frames of images. The offset value p is obtained by subtraction to quantify the visual content difference between the two frames of images. If p is greater than the set threshold, it is marked as an image to be processed. Step 4: Semantic scene understanding. A classification module is built based on the CLIP model and pre-trained weights are loaded. Although CLIP has excellent zero-shot classification performance, there is still a risk of misjudgment in complex scenes, and its performance is highly dependent on the construction of input prompt words. The text prompt words are set to three options: "normal scene", "abnormal scene" and "disaster scene". The image to be processed after being filtered in Step 3 and the above text prompts are input into the CLIP model to obtain its matching results. After Softmax normalization, the probabilities corresponding to the three categories are obtained, and the category with the highest probability is output as the preliminary evaluation result. Step 5: Fine-grained classifier. Using the fine-grained classifier trained in Step 2, perform fine-grained classification of disaster events on the scene images that were determined to be abnormal or disaster categories in Step 4, and output the final classification results. Step 6: GradCAM Visualization. The three-level cascaded disaster detection process can effectively identify disaster phenomena in the scene, but it does not yet have the ability to locate the disaster location. The GradCAM method is introduced to analyze network features to identify key regions that affect classification decisions. The target layer of GradCAM is set as the norm1 layer of the last Transformer module in the pre-trained encoder. After completing the classification in Step 5, the obtained classification results are fed back into the network. GradCAM is used to generate important feature regions corresponding to the category. Finally, the reshape operation is used to map them back to the original image size to achieve visualization of the localization results.

[0021] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A robust disaster detection method based on similarity and semantic information, characterized in that: The specific steps of this robust disaster detection method based on similarity and semantic information are as follows: Step 1: Masked Autoencoder (MAE) pre-training. The MAE model is built for reconstruction training. Its basic Transformer modules consist of the following structures: norm1, attn, norm2, and mlp. The encoder is composed of 12 stacked Transformer layers, and the decoder consists of 8 Transformer layers. The model is pre-trained on the ImageNet and Incident1M datasets. The input is the original image and its random masked version. During training, the reconstruction of the masked image region is achieved by minimizing the following reconstruction loss function: ； Step 2: Train the Vision Transformer fine-grained classifier, obtain the pre-trained masked autoencoder, retain the encoder part and discard the decoder, use the encoder weights as the backbone network of the downstream disaster detection task, and add one or more randomly initialized task-specific output heads after the pre-trained encoder according to the needs of the specific disaster dataset, and set its output dimension to the number of disaster categories. Step 3: Image similarity screening. Monitored images are acquired at fixed time intervals t. Then, the perceptual hash algorithm is used to calculate the hash values ​​of the two consecutive frames of images. The offset value p is obtained by the difference operation to quantify the visual content difference between the two frames of images. If p is greater than the set threshold, it is marked as an image to be processed. Step 4: Semantic scene understanding. A classification module is built based on the CLIP model and pre-trained weights are loaded. Although CLIP has excellent zero-shot classification performance, there is still a risk of misjudgment in complex scenes, and its performance is highly dependent on the construction of input prompt words. The text prompt words are set to three options: "normal scene", "abnormal scene" and "disaster scene". The image to be processed after being filtered in Step 3 and the above text prompts are input into the CLIP model to obtain its matching results. After Softmax normalization, the probabilities corresponding to the three categories are obtained, and the category with the highest probability is output as the preliminary evaluation result. Step 5: Fine-grained classifier. Using the fine-grained classifier trained in Step 2, perform fine-grained classification of disaster events on the scene images that were determined to be abnormal or disaster categories in Step 4, and output the final classification results. Step 6: GradCAM Visualization. The three-level cascaded disaster detection process can effectively identify disaster phenomena in the scene, but it does not yet have the ability to locate the disaster location. The GradCAM method is introduced to analyze network features to identify key regions that affect classification decisions. The target layer of GradCAM is set as the norm1 layer of the last Transformer module in the pre-trained encoder. After completing the classification in Step 5, the obtained classification results are fed back into the network. GradCAM is used to generate important feature regions corresponding to the category. Finally, the reshape operation is used to map them back to the original image size to achieve visualization of the localization results.

2. The robust disaster detection method based on similarity and semantic information according to claim 1, characterized in that: In step 1, represents the set of indices of all masked image blocks in the batch. This represents the size of the set, i.e., the total number of masked blocks. and Representing the first The reconstructed pixel values ​​and real pixel values ​​at each location can be well adapted to image data by the pre-trained model MAE through self-supervised learning. It can form a good feature distribution for images, which greatly facilitates fine-tuning for downstream tasks.

3. The robust disaster detection method based on similarity and semantic information according to claim 1, characterized in that: In step 2, the entire network is trained in a supervised manner in an end-to-end manner. During this process, the encoder weights are frozen, and gradient updates and optimizations are performed only on the parameters of the task-specific output head.

Images