Debris flow early warning system and method based on Transform fused sound and image

Abstract

The invention relates to a debris flow early warning system and method based on Transform fusion sound and image, the system comprises a front end acquisition module, an edge calculation module and a remote alarm module, the system can synchronously acquire visible light image and environment sound information of a monitoring area, and uploads the acquired data to a cloud server in real time through a communication module; the cloud model is based on a Transform structure, carries out deep fusion analysis on sound and image features, and combines synthetic data and a data enhancement strategy, thereby effectively improving the recognition precision of the model for weak symptoms before debris flow occurrence and the overall generalization ability of the system. The system can be widely applied to debris flow high-incidence areas, meets the actual requirements of high-frequency monitoring, intelligent identification and rapid early warning of geological disasters in a complex environment, solves the problems of single mode, low identification precision, difficult arrangement and the like of an existing system, and improves the intelligent level and response efficiency of the geological disaster monitoring system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of geological disaster monitoring and intelligent identification, and particularly relates to a debris flow early warning system and method based on fusion of sound and image of Transformer. BACKGROUND

[0002] In the geological disaster monitoring and early warning system, real-time collection and accurate identification of environmental information are crucial to protect the safety of personnel and property. Debris flow, landslide and other sudden and destructive disasters pose higher requirements on traditional monitoring methods. In recent years, video monitoring and sound sensing devices have been gradually applied in high-risk areas to capture physical characteristics during disasters. However, existing systems generally rely on single modal identification of images or sound, which is difficult to fully perceive weak signs before disasters, and the problems of false positives and false negatives are still prominent.

[0003] Traditional methods mainly use physical sensors such as rainfall and water level, which have certain threshold recognition capabilities, but have limited coverage in actual deployment and are difficult to respond to sudden situations in time. At the same time, visual or sound single modal models have their own limitations, the former is difficult to model time series information, and the latter is easily disturbed by the environment, with low fusion accuracy.

[0004] In addition, debris flow field data is scarce and difficult to obtain, especially sound data is often missing due to the suddenness of disasters, which seriously restricts model training. At present, there is a lack of unified and effective multi-modal modeling mechanism, and the fusion of image and sound features is still a difficulty. In terms of transmission, some devices still rely on wired or low-speed wireless protocols, which are difficult to adapt to the remote uploading needs of high-frequency and high-dimensional data in complex environments such as mountains.

[0005] In summary, in order to improve the intelligent level of debris flow early warning, a multi-modal identification system that fuses image and sound information is needed. SUMMARY

[0006] In view of the problems of weak single modal identification capability, data sample scarcity, lagging early warning response and limited remote deployment of existing debris flow monitoring technology, the present application provides a debris flow early warning system and method based on fusion of sound and image of Transformer, which fuses multi-modal image and sound, has time series modeling capability, and supports remote intelligent identification.

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions: A mudslide early warning system based on the fusion of sound and image of the Transformer, the system comprises a front-end acquisition module, an edge computing module and a remote alarm module connected in sequence, the front-end acquisition module comprises a camera assembly and a sound pickup assembly; the edge computing module comprises an image encoding submodule, an audio encoding submodule, a multi-modal fusion submodule, and a risk discrimination submodule; the remote alarm module comprises an abnormal trigger mechanism, an alarm information packaging module, a data compression and security module, and a cloud uploading module.

[0008] The system of the application integrates camera and sound pickup and other acquisition modules, can synchronously acquire visible light images and environmental sound information of a monitoring area, and can upload the acquired data to a cloud server in real time through a 4G / 5G communication module. The cloud model is based on the Transformer structure, performs deep fusion analysis on sound and image features, and combines synthetic data and data enhancement strategies to effectively improve the recognition accuracy of weak signs before mudslides and the overall generalization ability of the system.

[0009] As a preferred, the camera assembly is used for acquiring a sequence of RGB images of the monitoring area, and the sound pickup assembly is used for acquiring on-site environmental sound signals, and the camera assembly and the sound pickup assembly are integrated.

[0010] As a preferred, the image encoding submodule: based on the YOLOv12 structure, performs multi-target detection on continuous image frames, extracts a sequence of image target features containing time information, and captures visual abnormal changes before the occurrence of mudslides; The audio encoding submodule: based on the combination structure of CNN+GRU, processes sound signals; first, local spatial features in an audio spectrum graph are extracted through a convolutional neural network, and then time series modeling is performed through a gated recurrent unit to extract time-domain dynamic features of typical mudslide precursor audio including impact sound, sliding sound and water flow sound; The multi-modal fusion submodule: adopts the Transformer structure, fuses the image feature sequence output by the image encoding submodule and the sound time sequence feature output by the audio encoding submodule, establishes semantic association between modalities through self-attention and cross-attention mechanisms, and generates a unified fusion representation; The risk discrimination submodule: outputs a risk score after receiving the fusion representation, and judges whether the mudslide early warning trigger condition is met.

[0011] As a preferred, the abnormal trigger mechanism: when the risk score output by the fusion recognition module exceeds a set threshold, the remote alarm logic is triggered; Alarm information packaging module: integrate data including current timestamp, device number, geographic location information, risk level with recognition results, while extracting representative key image frames in the current frame, sound segment summary in the corresponding time period and risk score results of the recognition model; Data compression and security module: compress and encode image and audio data, and ensure the security of transmission data through encryption; Cloud uploading module: adopt 4G / 5G wireless communication module, and upload information to the preset cloud platform interface in real time according to the alarm priority.

[0012] The application also provides a debris flow early warning method based on the fusion of sound and image of the Transformer, which adopts the above system and comprises the following steps: S1, multi-modal data acquisition, the device integrating the camera assembly and the sound pickup assembly is deployed in the monitoring area to continuously monitor the environment and collect RGB image sequences and synchronous audio data; S2, image feature extraction, the image sequence is input into the YOLOv12 target detection model, the time sequence features of the image target are extracted, and high confidence target boxes and their feature representations are reserved; S3, audio feature extraction, the collected original audio signal is preprocessed, then the audio spectrum graph is input into the CNN+GRU network, and the time domain and frequency domain joint features in the acoustic sequence are extracted; S4, multi-modal fusion and recognition, the image feature sequence output by the YOLOv12 and the audio feature sequence output by the CNN+GRU are input into the Transformer fusion model, the cross attention mechanism is used to perform deep semantic fusion on the two types of features, the time sequence cooperative change relationship between the image and the audio is captured, and a risk score vector is output; S5, abnormality recognition and remote alarm, when the risk score output by the fusion model exceeds the set threshold, the system judges that it is a suspected debris flow event, and immediately triggers the remote alarm logic; if the score does not exceed the threshold, the system continues to collect and analyze subsequent data, and maintains the monitoring standby state.

[0013] As preferred, step S1 is specifically: S1-1, image acquisition: the image acquisition is completed by the camera assembly fixedly installed in the monitoring area, an industrial-grade wide-angle camera is adopted, and the camera continuously images the scene at a frequency of 1-5 frames / second; S1-2, sound acquisition: the environmental audio is collected by the sound pickup assembly, and the sampling rate is above 16kHz; the sound signal is synchronously collected in real time, and is cached as a sliding audio segment of 10 seconds, and is automatically labeled in cooperation with the image timestamp; S1-3, synchronization and preprocessing mechanism: image frames and audio data are aligned through a unified timestamp mechanism and cached to the edge computing module in a sliding window manner.

[0014] As preferred, step S2 is specifically: S2-1, image preprocessing: input the continuous image frame sequence obtained in step S1 into the edge computing platform, and first perform standardization preprocessing operations on the image, including resolution scaling and brightness equalization; S2-2, model pre-training and deployment: pre-train the YOLOv12 model on the ICL Landslide Dataset, extract multi-class target image features related to geological disasters, and perform multi-scale semantic alignment; after completing the pre-trained model, deploy it to the edge device; S2-3, target detection and semantic extraction: input the image sequence frame by frame into the pre-trained YOLOv12 model to extract potential debris flow related target information in each frame of image; output includes position information, class label, confidence score and intermediate layer semantic feature vector of each target.

[0015] As preferred, step S3 is specifically: S3-1, audio signal preprocessing: denoising and gain control are performed on the collected original audio signal to remove invalid silent segments and background noise; then frame segmentation and window function processing are performed on the audio to extract 1-3 second effective sound segments and convert them into mel spectrum graph representation; S3-2, optimization training based on synthetic data and enhancement strategy: construct an enhanced training set containing synthetic audio segments of rockfall sound, water flow impact sound, and ground collapse sound; at the same time, introduce enhancement strategies including spectral disturbance, time shift, and reverberation simulation; S3-3, CNN+GRU network feature extraction: input the spectrum graph into the audio feature extraction model, first extract local acoustic texture and frequency domain distribution features by convolutional neural network CNN, then model the time sequence by gated recurrent unit GRU to capture the dynamic evolution process of sound events; the output is a time sequence feature sequence of the audio modality; S3-4, pre-training of audio model and deployment: pre-train the CNN+GRU network based on disaster sound samples and enhanced audio set, and after training, deploy it to the edge computing module.

[0016] As preferred, step S4 is specifically: S4-1, uniform encoding and alignment of features: input the image time sequence feature vector sequence obtained in S2 and the audio time sequence feature vector sequence in S3 into the unified feature encoding layer to perform dimension normalization and position encoding respectively; S4-2, cross attention fusion based on Transformer is performed: a multi-modal Transformer fusion model is constructed, a cross attention mechanism is adopted, the model uses image features as a main query, audio as a key-value pair, and dynamically aggregates cross-modal correlation features in disaster precursors; S4-3, fusion representation generation and risk classification: after the output features of the Transformer fusion module are integrated through multi-head attention, the classification discrimination module is input, and the classification results of the fused debris flow risk level are output, and an abnormal confidence score is attached; S4-4, model training and optimization mechanism: the multi-modal Transformer model is trained through a joint loss function, including image recognition loss, audio classification loss and fusion decision loss, supporting supervised learning and joint training of a small amount of semi-supervised samples; the model is pruned and deployed on an edge computing module for real-time early warning.

[0017] As preferred, step S5 is specifically: S5-1, risk score output and threshold judgment: the debris flow risk level output generated in step S4 is converted into a normalized score value, if the score is higher than the set threshold, the system determines that it is a suspected debris flow event; otherwise, it is considered as a normal state, and the system continues to enter the next round of data acquisition and analysis process; S5-2, trigger alarm and data upload: when an abnormal event is determined, the edge device immediately triggers a remote alarm logic, and packs and uploads data including the identification result, key image frame, sound summary segment, score value and time stamp to the cloud platform through 4G / 5G wireless network; S5-3, abnormal cache and local record: the local device caches the image, audio and judgment information of the last triggered time, which is used for abnormal backtracking and model adaptive training; if the network is interrupted, the data is temporarily stored and automatically supplemented after the communication is restored; S5-4, state update and continuous monitoring: after completing the alarm process, the system state is automatically switched to monitoring standby, and the detection process is restarted periodically.

[0018] Compared with the prior art, the present application has the following beneficial effects: (1) The present application aims at the problem that debris flow disaster precursors are difficult to identify in time, fully considers the difficulties in data acquisition in mountainous environment, multi-modal information heterogeneity and real-time response demand, has the characteristics of multi-modal fusion, edge intelligent discrimination and remote early warning linkage; an image-sound dual-modal recognition method based on YOLOv12 and CNN-GRU is adopted as the core means, the joint modeling of ground surface changes and sudden abnormal sound in the video is realized, and the accurate perception of debris flow risk is realized; (2) The application introduces a Transformer structure to carry out mid-term feature fusion, models the deep correlation between images and audio through a cross-attention mechanism, and effectively improves the system's ability to model the temporal evolution of disasters and understand semantics; (3) The application has local lightweight reasoning and remote reporting capabilities, and can automatically upload key images, audio summaries and risk scores after detecting suspected abnormalities, forming a closed-loop warning mechanism of "recognition-judgment-upload", and providing a highly reliable and adaptive intelligent solution for geological disaster warning; (4) The application can be widely used in mudslide-prone areas such as mountainous areas, valleys, reservoir slopes and roadsides, and meets the actual needs of high-frequency monitoring, intelligent identification and rapid warning of geological disasters in complex environments, solves the problems of single mode, low identification accuracy and difficult deployment of existing systems, and improves the intelligent level and response efficiency of the geological disaster monitoring system. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a schematic diagram of the system framework of the application; Figure 2 is a schematic diagram of the device deployment environment of the application; Figure 3 is a model structure diagram of the application. DETAILED DESCRIPTION

[0020] The technical solutions of the application will be further described in detail below through examples, which are illustrative of the application and not limiting. Based on the examples in this application, all other examples obtained by those skilled in the art without creative labor are within the scope of protection of this application.

[0021] Reference Figure 1 , Figure 2 A mudslide warning system based on fusion of sound and image based on Transformer, the system comprises a front-end acquisition module, an edge computing module and a remote alarm module connected in sequence, the front-end acquisition module comprises a camera assembly and a microphone assembly; the edge computing module comprises an image encoding submodule, an audio encoding submodule, a multi-modal fusion submodule, and a risk discrimination submodule; the remote alarm module comprises an abnormal trigger mechanism, an alarm information packaging module, a data compression and security module, and a cloud uploading module. Each part works together to realize multi-modal perception and intelligent warning of mudslide risk.

[0022] The camera assembly is used to acquire a sequence of RGB images of the monitoring area, and the microphone assembly is used to acquire the sound signals of the on-site environment, and the camera assembly and the microphone assembly are integrated.

[0023] Image encoding submodule: Based on the YOLOv12 structure, multiple target detection is performed on consecutive image frames, and image target feature sequences containing time information are extracted to capture visual abnormal changes before the occurrence of debris flow; Audio encoding submodule: Based on the combination structure of CNN+GRU, the sound signal is processed; first, the local spatial features in the audio spectrum graph are extracted through the convolutional neural network, and then the time series modeling is performed through the gated recurrent unit to extract the time domain dynamic features of the typical debris flow precursor audio including impact sound, sliding sound and water flow sound; Multi-modal fusion submodule: Adopting the Transformer structure, the image feature sequences output by the image encoding submodule and the sound time sequence features output by the audio encoding submodule are fused to establish semantic association between modalities through self-attention and cross-attention mechanisms to generate a unified fusion representation; Risk discrimination submodule: After receiving the fusion representation, it outputs a risk score and determines whether the debris flow warning trigger condition is met.

[0024] Abnormal trigger mechanism: When the risk score output by the fusion recognition module exceeds the set threshold, the remote alarm logic is triggered; Alarm information packaging module: The data including the current timestamp, device number, geographic location information, risk level are integrated with the recognition result, and at the same time, the representative key image frames in the current frame, the sound segment summary in the corresponding time period and the risk score result of the recognition model are extracted; Data compression and security module: The image and audio data are compressed and encoded, and the security of the transmission data is guaranteed through encryption; Cloud uploading module: Using 4G / 5G wireless communication module, the information is uploaded to the preset cloud platform interface in real time according to the alarm priority.

[0025] The camera assembly is a high-definition industrial-grade 5 million pixel CMOS camera with night infrared light supplement and automatic focusing function, which is used to continuously capture the image change information of target areas such as slopes, valleys and drainage channels before and after debris flow. The obtained image sequence is uploaded to the edge computing module in real time, and input into the YOLOv12 target detection model pre-trained on the ICL Landslide Dataset for multiple target detection and time sequence feature extraction. At the same time, the data in the typical mountain scene are fine-tuned to adapt to different light and topography environments.

[0026] The pickup assembly has a sensitivity of -38 dB and a sampling rate of 16 kHz, and can continuously collect environmental sound, so as to perceive the acoustic anomalies such as flowing water, falling stones and impacts generated before the debris flow. The collected audio signals are input into the CNN+GRU network after preprocessing, wherein the CNN part extracts the local spatial structure features of the spectrum graph, and the GRU is responsible for capturing the dynamic changes of the impact sound in time sequence, forming a stable time domain and frequency domain joint representation.

[0027] Both the image and audio features are processed locally by the edge computing module Jetson Nano. The edge computing module integrates a unified feature encoding module to normalize the dimensions and align the time of the image and audio features, and further inputs a Transformer fusion model based on the cross-attention mechanism to realize deep semantic interaction of cross-modal features. The joint features output by the fusion are processed by the multi-head attention integration and classification module, and the debris flow risk score and grade label are output, and millisecond-level inference can be realized on the edge to meet the real-time early warning requirements.

[0028] When the risk score output by the fusion model is higher than the preset threshold (such as 0.75), the system regards the recognition result as a “suspected debris flow event” and immediately triggers the remote alarm mechanism. The system automatically packages the current key image frame, audio summary, device number, positioning information and risk score, and uploads them to the cloud platform through the 4G communication module, realizes remote data backhaul, event reporting and map visualization linkage management. The cloud system also supports event log storage, historical playback, risk level statistics and manual review interface, improving the intelligence and explainability of the entire early warning system.

[0029] The system of the present application adopts an integrated structure design, integrates the camera, pickup, computing and communication modules in an IP66 level shell that is dustproof, waterproof and shockproof, is suitable for deployment in complex outdoor environments, supports solar power supply and low-power standby mode, ensures that the device can be stably operated in remote mountainous areas, reservoir dam sites, road slopes and other debris flow-prone areas for a long time, and realizes all-weather, automatic and intelligent monitoring and early warning of geological disasters.

[0030] Reference Figure 3 The specific implementation method of the above system is as follows: S1, multi-modal data acquisition, deploy the device integrating the camera assembly and the pickup assembly in the monitoring area to continuously monitor the environment and collect RGB image sequences and synchronous audio data; specifically including: S1-1, image acquisition: image acquisition is completed by a camera assembly fixedly installed in the monitoring area, a low-illumination industrial wide-angle camera is adopted, and night light supplement and wide dynamic range (WDR) functions are provided; the camera continuously images the scene at a frequency of 2-5 frames / second, which can adapt to complex environmental lighting conditions such as daytime, nighttime, and foggy weather; image acquisition is triggered by the edge computing module at regular time intervals to ensure timing stability; S1-2, sound acquisition: environmental audio is collected by a high-sensitivity capacitive microphone assembly, with a sampling rate of 16 kHz or higher, covering the audible range of human ears and low-frequency impact signals, ensuring the inclusion of potential disaster characteristic sounds such as rolling stones, accelerating water flow, and ground sliding; sound signals are collected in real time and synchronized, and cached as a sliding audio segment of 10 seconds, automatically labeled with image timestamps; S1-3, synchronization and preprocessing mechanism: image frames and audio data are aligned through a unified timestamp mechanism and cached in a sliding window manner to the edge computing module; to ensure data stability, the system performs image quality detection and audio signal-to-noise ratio estimation before each round of data collection to remove interference samples.

[0031] The image and audio data collected through step S1 provide input basis for subsequent multi-modal fusion, which can record the visual dynamic changes and acoustic characteristics before the occurrence of debris flow, and realize collaborative perception at the original signal level.

[0032] S2, image feature extraction, the image sequence is input into the YOLOv12 target detection model to extract the time sequence features of the image target and retain the high-confidence target frame and its feature representation; specifically including: S2-1, image preprocessing: the continuous image frame sequence obtained in step S1 is input into the edge computing platform, first performing standardization preprocessing operations on the image, including resolution scaling, brightness equalization, etc., to adapt to the subsequent model input requirements and improve the image quality in weak light or complex environments; S2-2, YOLOv12 model pre-training and deployment: the YOLOv12 model is pre-trained on the ICL Landslide Dataset to extract multi-class target image features related to geological disasters and perform multi-scale semantic alignment; after completing the pre-trained model, it is deployed to the edge device to realize local fast inference; S2-3, target detection and semantic extraction: the image sequence is input frame by frame into the pre-trained YOLOv12 model to extract potential debris flow-related target information in each image frame; the output includes the position information, class label, confidence score, and intermediate layer semantic feature vector of each target.

[0033] S3, audio feature extraction, pre-processing the collected original audio signal, and then inputting the audio spectrum graph into the CNN+GRU network to extract the joint features of time domain and frequency domain in the acoustic sequence; specifically including: S3-1, audio signal pre-processing: denoising and gain control are performed on the collected original audio signal to remove invalid silent segments and background noise; then frame segmentation and window function processing are performed on the audio to extract 1-3 second effective sound segments, and convert them into mel spectrum graph representation for subsequent network input; S3-2, optimization training based on synthetic data and enhancement strategy: to solve the problem of scarcity of real debris flow sound samples, an enhanced training set of synthetic audio segments including rolling stone sound, water flow impact sound and ground collapse sound is constructed; at the same time, enhancement strategies including spectral perturbation, time shift and reverberation simulation are introduced to improve the robustness and generalization ability of the model to environmental noise, device differences and complex sound sources; S3-3, CNN+GRU network feature extraction: input the spectrum graph into the audio feature extraction model, first extract local acoustic texture and frequency domain distribution features by convolutional neural network CNN, then model the time sequence by gated recurrent unit GRU to capture the dynamic evolution process of sound events; the output is a time sequence feature sequence of audio modal, which has time sensitivity and class discrimination ability; S3-4, pre-training of audio model and deployment: the CNN+GRU network is pre-trained based on public disaster sound samples and enhanced audio set, and after training, it is deployed to the edge computing module to realize fast processing and feature coding of real-time collected audio.

[0034] S4, multi-modal fusion and recognition, input the image feature sequence output by YOLOv12 and the audio feature sequence output by CNN+GRU into the Transformer fusion model, use cross-attention mechanism to perform deep semantic fusion on the two types of features, capture the time sequence cooperative change relationship between image and audio, and output risk score vector; specifically including: S4-1, uniform feature encoding and alignment: input the image time sequence feature vector sequence obtained in S2 and the audio time sequence feature vector sequence in S3 into the uniform feature encoding layer, respectively perform dimension normalization and position encoding, ensure that the two modalities are aligned on the time axis, and compatible with the input requirements of the Transformer network structure; S4-2, cross-attention fusion based on Transformer: a multi-modal Transformer fusion model is constructed to realize deep semantic interaction between images and audio through cross-attention mechanism; the model uses image features as the main query (Query), and audio as the key-value pair (Key / Value) to dynamically aggregate the cross-modal correlation features in disaster precursors; the model supports simultaneous attention to multiple frame images and sound changes under continuous time sequence, effectively modeling the disaster evolution trend; The above formula represents the multi-head attention mechanism in the transformer, wherein, represents the query vector (Query) matrix, represents the key vector (Key) matrix, and V is the value vector (Value) matrix; represents the dot product similarity of the query vector and the key vector, and represents the correlation of the input sequence with other sequences is a scaling factor used to maintain numerical stability; represents the softmax normalization of each row to obtain a probability distribution. After normalization, a weight matrix is obtained, which is multiplied by the value vector matrix to obtain a weighted feature representation.

[0035] matrix is the corresponding learnable projection matrix; after stacking the Transformer layer, the fused joint feature representation is obtained, which can comprehensively reflect the potential landslide evolution trend under multi-modal conditions; S4-3, fusion representation generation and risk classification: the output features of the Transformer fusion module are input into the classification discrimination module after multi-head attention integration, and the fused landslide risk level classification results are output, with an abnormal confidence score; S4-4, model training and optimization mechanism: the multi-modal Transformer model is trained through a joint loss function, including image recognition loss, audio classification loss, and fusion decision loss, supporting supervised learning and joint training with a small amount of semi-supervised samples; the model is pruned and deployed on the edge computing module for real-time early warning, supporting real-time local inference and remote linkage reporting.

[0036] S5, abnormality identification and remote alarm: when the risk score output by the fusion model exceeds the set threshold (such as 0.75), the system judges it as a "suspected landslide event" and triggers the remote alarm logic immediately; if the score does not exceed the threshold, the system continues to collect and analyze subsequent data, maintaining the monitoring standby state; specifically including: S5-1, risk score output and threshold judgment: the debris flow risk level output generated in step S4 is converted into a normalized score value, and a continuous risk score is output, which is divided into three risk levels: low (<0.4), medium (0.4-0.75), and high (>0.75); if the score is higher than the set threshold (such as 0.75), the system determines that it is a "suspected debris flow event"; otherwise, it is considered normal, and the system continues to enter the next round of data collection and analysis process; S5-2, trigger alarm and data upload: when an abnormal event is determined, the edge device immediately triggers a remote alarm logic, and uploads data including identification results, key image frames, sound summary fragments, score values and time stamps to the cloud platform through 4G / 5G wireless network for further decision and linkage response by the back-end system; S5-3, abnormal cache and local recording: the local device caches the image, audio and determination information of the last triggered, which is used for abnormal backtracking and model adaptive training; if the network is interrupted, the data is temporarily stored and automatically supplemented after the communication is restored; S5-4, state update and continuous monitoring: after completing the alarm process, the system state is automatically switched to monitoring standby, and the detection process is restarted periodically to ensure continuous monitoring and multiple determination capability in emergency scenarios, avoiding missed detection or false alarm.

[0037] To verify the effectiveness of the present application, a multi-modal debris flow recognition data set including image + audio is constructed, covering typical mountain landslides, debris flows caused by heavy rainfall and other real scenarios. The model is pre-trained on the image part of the public ICL LandslideDataset, and is jointly fine-tuned on the field collected data. The experiment is completed on the JetsonNano platform, and the recognition accuracy, F1 score and processing delay are evaluated. The recognition performance comparison results under different modalities are shown in Table 1.

[0038] Table 1: Comparison of recognition performance under different modalities As can be seen from Table 1, the accuracy of the present application reaches 92.6%, and the F1 score reaches 92.1%. The accuracy is improved by 6.1% and 13.1% compared with image (86.5%) and audio (79.5%) single modal recognition, respectively, effectively improving the accuracy and stability of debris flow recognition, and having better practicality and reliability.

[0039] The above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features thereof; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A mudslide early warning system based on fusion of sound and image of Transformer, characterized in that, The system comprises a front-end acquisition module, an edge computing module and a remote alarm module connected in sequence, the front-end acquisition module comprises a camera assembly and a sound pickup assembly; the edge computing module comprises an image coding submodule, an audio coding submodule, a multi-modal fusion submodule and a risk discrimination submodule; the remote alarm module comprises an abnormality triggering mechanism, an alarm information packaging module, a data compression and security module and a cloud uploading module.

2. The mudslide early warning system based on fusion of sound and image of the Transformer according to claim 1, characterized in that, The camera assembly is used for acquiring a sequence of RGB images of a monitoring area, and the sound pickup assembly is used for acquiring an on-site environmental sound signal, and the camera assembly and the sound pickup assembly are integrated. 3.The mudslide early warning system based on fusion of sound and image of the Transformer according to claim 1, wherein, The image coding submodule: based on the YOLOv12 structure, multi-target detection is performed on continuous image frames, a sequence of image target features containing time information is extracted, and visual abnormal changes before the occurrence of a debris flow are captured; The audio coding submodule: based on a combination structure of CNN+GRU, sound signals are processed; first, local spatial features in an audio spectrum graph are extracted through a convolutional neural network, and then time sequence modeling is performed through a gated recurrent unit to extract time-domain dynamic features of typical debris flow precursor audio including impact sound, sliding sound and water flow sound; The multi-modal fusion submodule: adopts a Transformer structure, fuses the image feature sequence output by the image coding submodule and the sound time sequence feature output by the audio coding submodule, establishes semantic association between modalities through self-attention and cross-attention mechanisms, and generates a unified fusion representation; The risk discrimination submodule: outputs a risk score after receiving the fusion representation, and judges whether the debris flow early warning triggering condition is met.

4. The mudslide early warning system based on fusion of sound and image of the Transformer according to claim 1, characterized in that, The abnormality triggering mechanism: when the risk score output by the fusion recognition module exceeds a set threshold, the remote alarm logic is triggered; The alarm information packaging module: integrates data including a current timestamp, a device number, geographic location information, a risk level and recognition results, and extracts representative key image frames in the current frame, a sound segment summary in the corresponding time period and a risk score result of the recognition model; The data compression and security module: compresses and encodes image and audio data, and ensures the security of transmission data through encryption; The cloud uploading module: adopts a 4G / 5G wireless communication module, and uploads information to a preset cloud platform interface in real time according to alarm priority.

5. A method for mudslide early warning based on fusion of sound and image of a Transformer, characterized in that, The system of any one of claims 1-4 comprises the following steps: S1, multi-modal data acquisition: deploying a device integrating a camera assembly and a sound pickup assembly in a monitoring area to continuously monitor the environment and acquire a sequence of RGB images and synchronous audio data; S2, image feature extraction: inputting the image sequence into a YOLOv12 target detection model to extract time sequence features of image targets and retaining high-confidence target boxes and their feature representations; S3, audio feature extraction: preprocessing the acquired original audio signal, and then inputting an audio spectrum graph into a CNN+GRU network to extract joint features in the time domain and the frequency domain of an acoustic sequence; S4, multimodal fusion and recognition, input the image feature sequence output by YOLOv12 and the audio feature sequence output by CNN+GRU into a Transformer fusion model, use cross-attention mechanism to perform deep semantic fusion on the two types of features, capture the timing collaborative change relationship between image and audio, and output a risk score vector; S5, abnormality recognition and remote alarm, when the risk score output by the fusion model exceeds the set threshold, the system judges that it is a suspected debris flow event, and immediately triggers the remote alarm logic; If the score does not exceed the threshold, the system continues to collect and analyze subsequent data, and maintains the monitoring standby state.

6. The method according to claim 5, wherein, Step S1 is specifically: S1-1, image acquisition: image acquisition is completed by a camera assembly fixedly installed in the monitoring area, an industrial-grade wide-angle camera is used, and the camera continuously images the scene at a frequency of 1-5 frames / second; S1-2, sound acquisition: environmental audio is collected by a sound pickup assembly, with a sampling rate of 16kHz or higher; sound signals are synchronously collected in real time and cached as sliding audio segments of 10 seconds each, automatically labeled with image timestamps; S1-3, synchronization and preprocessing mechanism: image frames and audio data are aligned through a unified timestamp mechanism, and cached to an edge computing module in a sliding window manner.

7. The method according to claim 6, wherein, Step S2 is specifically: S2-1, image preprocessing: input the continuous image frame sequence obtained in step S1 into the edge computing platform, first perform standardization preprocessing operations on the image, including resolution scaling and brightness equalization; S2-2, model pre-training and deployment: pre-train the YOLOv12 model on the ICL Landslide Dataset, extract multi-class target image features related to geological disasters, and perform multi-scale semantic alignment; after completing the pre-trained model, deploy it to the edge device; S2-3, target detection and semantic extraction: input the image sequence frame by frame into the pre-trained YOLOv12 model, extract potential debris flow related target information in each image frame; output includes the position information, class label, confidence score of each target and intermediate layer semantic feature vector.

8. The method according to claim 7, wherein, Step S3 is specifically: S3-1, audio signal preprocessing: denoise and gain control are performed on the collected original audio signal, and invalid silent segments and background noise are removed; then frame segmentation and window function processing are performed on the audio, and 1-3 second effective sound segments are extracted and converted into mel spectrum graph representation; S3-2, optimization training based on synthetic data and enhancement strategy: construct an enhanced training set containing synthetic audio segments of rolling stone sound, water flow impact sound and ground collapse sound; at the same time, introduce enhancement strategies including spectrum disturbance, time shift and reverberation simulation; S3-3, CNN+GRU network feature extraction: input the spectrum graph into the audio feature extraction model, first extract local acoustic texture and frequency domain distribution features by convolutional neural network CNN, then model the time sequence by gated recurrent unit GRU, and capture the dynamic evolution process of the sound event; the output is a time sequence feature sequence of the audio mode; S3-4, Pre-training of audio model and deployment: The CNN+GRU network is pre-trained based on disaster sound samples and enhanced audio sets, and after training is completed, it is deployed to the edge computing module.

9. The method according to claim 8, wherein, Step S4 is specifically: S4-1, uniform coding and alignment of features: input the image time sequence feature vector sequence obtained in S2 and the audio time sequence feature vector sequence in S3 into the uniform feature coding layer, respectively, for dimension normalization and position coding; S4-2, cross-attention fusion based on Transformer: a multi-modal Transformer fusion model is constructed, using cross-attention mechanism, the model uses image features as the main query and audio as the key-value pair, dynamically aggregating cross-modal correlation features in disaster precursors; S4-3, fusion representation generation and risk classification: after the output features of the Transformer fusion module are integrated through multi-head attention, they are input into the classification and discrimination module, and the integrated debris flow risk level classification results are output, with an abnormal confidence score; S4-4, model training and optimization mechanism: the multi-modal Transformer model is trained through a joint loss function, including image recognition loss, audio classification loss and fusion decision loss, supporting supervised learning and joint training with a small amount of semi-supervised samples; the model is pruned and compressed and deployed on the edge computing module for real-time early warning.

10. The method according to claim 9, wherein, Step S5 is specifically: S5-1, risk score output and threshold judgment: convert the debris flow risk level output generated in step S4 into a normalized score value, if the score is higher than the set threshold, the system determines it as a suspected debris flow event; Otherwise, it is considered as a normal state, and the system continues to enter the next round of data collection and analysis process; S5-2, trigger alarm and data upload: when an abnormal event is determined, the edge device immediately triggers a remote alarm logic, and uploads the data including the recognition result, key image frame, sound summary segment, score value and time stamp to the cloud platform through 4G / 5G wireless network; S5-3, abnormal caching and local recording: the local device caches the images, audio and judgment information of the last triggered event for abnormal backtracking and model adaptive training; if the network is interrupted, the data is temporarily stored and automatically supplemented after the communication is restored; S5-4, state update and continuous monitoring: after completing the alarm process, the system state is automatically switched to monitoring standby, and the detection process is restarted periodically.

Images